Fwd: [greenstone-users] Out of Memory Error

From John Rowe
DateWed, 19 Oct 2005 11:04:20 +1300
Subject Fwd: [greenstone-users] Out of Memory Error
In-Reply-To (00ad01c5d42c$54d28720$6600a8c0-DHRWZP71)
Excuse my last email, the send button is rather close to the
close button in Apple's Mail.app!

Unfortunately the metadata.xml handler was never really designed
with data of this size in mind. If you are still interested in doing
this using GSDL, a custom import plugin that works on the original
documents might be the easiest way. Any XML parser will have trouble
with 310 megs of XML - generally you can multiply the ram usage by
three and then add a little to see how much ram you will need to
parse any specific XML file.

Cheers,
John Rowe

DL Consulting
Greenstone Digital Library and Digitisation Specialists
contact@dlconsulting.co.nz
www.dlconsulting.co.nz


Begin forwarded message:

> From: "Nathan Einwechter" <nathan@inorb.com>
> Date: 19 October 2005 10:38:51 AM
> To: <greenstone-users@list.scms.waikato.ac.nz>
> Subject: RE: [greenstone-users] Out of Memory Error
>
>
> The problem is that this is ALL custom. I'm actually using it for
> hierarchies of security log information (IP addresses, ports,
> protocols). This is all generated by a program I have created.
>
> I'm thinking about just exporting to a self designed XML repository at
> this stage though. Greenstone has been good so far, but we're
> looking at
> scaling up even farther and I don't know how it can be done using
> GSDL.
>
> -- Nathan
>
> -----Original Message-----
> From: greenstone-users-bounces@list.scms.waikato.ac.nz
> [mailto:greenstone-users-bounces@list.scms.waikato.ac.nz] On Behalf Of
> John Rowe
> Sent: October 18, 2005 5:28 PM
> To: greenstone-users@list.scms.waikato.ac.nz
> Subject: Fwd: [greenstone-users] Out of Memory Error
>
> What is contained in your metadata.xml? What filetypes are in
> your collection? Is there any way you could build an equivalent
> collection without using the metadata.xml? Have you experimented with
> a custom import plugin?
>
> Cheers,
> John Rowe
>
> DL Consulting
> Greenstone Digital Library and Digitisation Specialists
> contact@dlconsulting.co.nz
> www.dlconsulting.co.nz
>
>
> Begin forwarded message:
>
>
>> From: "Nathan Einwechter" <nathan@inorb.com>
>> Date: 19 October 2005 10:04:14 AM
>> To: <greenstone-users@list.scms.waikato.ac.nz>
>> Subject: RE: [greenstone-users] Out of Memory Error
>>
>>
>> Does anyone know the maximum size I can have for a metadata.xml
>> file for
>> a given amount of ram etc.?
>>
>> -- Nathan
>>
>> -----Original Message-----
>> From: greenstone-users-bounces@list.scms.waikato.ac.nz
>> [mailto:greenstone-users-bounces@list.scms.waikato.ac.nz] On
>> Behalf Of
>> John Rowe
>> Sent: October 17, 2005 10:03 PM
>> To: greenstone-users@list.scms.waikato.ac.nz
>> Subject: Fwd: [greenstone-users] Out of Memory Error
>>
>> it seems that the Greenstone import process is simply not built
>> to handle a metadata file that large. The easiest solution would be
>> to split your import directory (and metadata.xml file) up into
>> smaller directories under the main import directory. The filesystem
>> should end up looking something like the following:
>> import/
>> import/001/metadata.xml
>> import/001/file1
>> import/001/file2
>> import/001/.....
>> import/001/file9
>> import/002/metadata.xml
>> import/002/file10
>> import/002/file11
>> import/002/...
>> import/002/file19
>> import/003/metadata.xml
>> import/003/file20
>> import/001/file21
>> .............
>>
>> Hope this helps.
>>
>> Cheers,
>> John Rowe
>>
>> DL Consulting
>> Greenstone Digital Library and Digitisation Specialists
>> contact@dlconsulting.co.nz
>> www.dlconsulting.co.nz
>>
>>
>>
>> _______________________________________________
>> greenstone-users mailing list
>> greenstone-users@list.scms.waikato.ac.nz
>> https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users
>>
>>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> https://list.scms.waikato.ac.nz/mailman/private/greenstone-users/
> attachm
> ents/20051019/05ec692b/attachment.html
> _______________________________________________
> greenstone-users mailing list
> greenstone-users@list.scms.waikato.ac.nz
> https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users
>
>
> _______________________________________________
> greenstone-users mailing list
> greenstone-users@list.scms.waikato.ac.nz
> https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://list.scms.waikato.ac.nz/mailman/private/greenstone-users/attachments/20051019/a5da7777/attachment.html