Re: [greenstone-users] Out of Memory Error

From Katherine Don
DateThu, 20 Oct 2005 10:43:36 +1300
Subject Re: [greenstone-users] Out of Memory Error
In-Reply-To (00b001c5d430$3d5eaa20$6600a8c0-DHRWZP71)
Hi Nathan

At the end of your processing, can you output greenstone archive files
instead of metadata.xml? Then you wouldn't need to do an import at all.

Cheers,
katherine

Nathan Einwechter wrote:
> They're log files. BUT - I can't really create an import plugin as the
> files have to go through a whole bunch of different types of processing,
> analysis, etc. by the system before being ready to be pushed to
> Greenstone.. I guess I could push the results into a specific format and
> then create an import plugin for that format though.
>
> Good idea.
>
> -- Nathan
>
> -----Original Message-----
> From: greenstone-users-bounces@list.scms.waikato.ac.nz
> [mailto:greenstone-users-bounces@list.scms.waikato.ac.nz] On Behalf Of
> John Rowe
> Sent: October 18, 2005 5:54 PM
> To: greenstone-users@list.scms.waikato.ac.nz
> Subject: Fwd: [greenstone-users] Out of Memory Error
>
> What are the original source documents? Can you build a custom
> import plugin to import the original files rather than impo
>
> Cheers,
> John Rowe
>
> DL Consulting
> Greenstone Digital Library and Digitisation Specialists
> contact@dlconsulting.co.nz
> www.dlconsulting.co.nz
>
>
> Begin forwarded message:
>
>
>>From: "Nathan Einwechter" <nathan@inorb.com>
>>Date: 19 October 2005 10:38:51 AM
>>To: <greenstone-users@list.scms.waikato.ac.nz>
>>Subject: RE: [greenstone-users] Out of Memory Error
>>
>>
>>The problem is that this is ALL custom. I'm actually using it for
>>hierarchies of security log information (IP addresses, ports,
>>protocols). This is all generated by a program I have created.
>>
>>I'm thinking about just exporting to a self designed XML repository at
>>this stage though. Greenstone has been good so far, but we're
>>looking at
>>scaling up even farther and I don't know how it can be done using
>>GSDL.
>>
>>-- Nathan
>>
>>-----Original Message-----
>>From: greenstone-users-bounces@list.scms.waikato.ac.nz
>>[mailto:greenstone-users-bounces@list.scms.waikato.ac.nz] On Behalf Of
>>John Rowe
>>Sent: October 18, 2005 5:28 PM
>>To: greenstone-users@list.scms.waikato.ac.nz
>>Subject: Fwd: [greenstone-users] Out of Memory Error
>>
>> What is contained in your metadata.xml? What filetypes are in
>>your collection? Is there any way you could build an equivalent
>>collection without using the metadata.xml? Have you experimented with
>>a custom import plugin?
>>
>>Cheers,
>> John Rowe
>>
>>DL Consulting
>>Greenstone Digital Library and Digitisation Specialists
>>contact@dlconsulting.co.nz
>>www.dlconsulting.co.nz
>>
>>
>>Begin forwarded message:
>>
>>
>>
>>>From: "Nathan Einwechter" <nathan@inorb.com>
>>>Date: 19 October 2005 10:04:14 AM
>>>To: <greenstone-users@list.scms.waikato.ac.nz>
>>>Subject: RE: [greenstone-users] Out of Memory Error
>>>
>>>
>>>Does anyone know the maximum size I can have for a metadata.xml
>>>file for
>>>a given amount of ram etc.?
>>>
>>>-- Nathan
>>>
>>>-----Original Message-----
>>>From: greenstone-users-bounces@list.scms.waikato.ac.nz
>>>[mailto:greenstone-users-bounces@list.scms.waikato.ac.nz] On
>>>Behalf Of
>>>John Rowe
>>>Sent: October 17, 2005 10:03 PM
>>>To: greenstone-users@list.scms.waikato.ac.nz
>>>Subject: Fwd: [greenstone-users] Out of Memory Error
>>>
>>> it seems that the Greenstone import process is simply not built
>>>to handle a metadata file that large. The easiest solution would be
>>>to split your import directory (and metadata.xml file) up into
>>>smaller directories under the main import directory. The filesystem
>>>should end up looking something like the following:
>>> import/
>>> import/001/metadata.xml
>>> import/001/file1
>>> import/001/file2
>>> import/001/.....
>>> import/001/file9
>>> import/002/metadata.xml
>>> import/002/file10
>>> import/002/file11
>>> import/002/...
>>> import/002/file19
>>> import/003/metadata.xml
>>> import/003/file20
>>> import/001/file21
>>> .............
>>>
>>> Hope this helps.
>>>
>>>Cheers,
>>> John Rowe
>>>
>>>DL Consulting
>>>Greenstone Digital Library and Digitisation Specialists
>>>contact@dlconsulting.co.nz
>>>www.dlconsulting.co.nz
>>>
>>>
>>>
>>>_______________________________________________
>>>greenstone-users mailing list
>>>greenstone-users@list.scms.waikato.ac.nz
>>>https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users
>>>
>>>
>>
>>-------------- next part --------------
>>An HTML attachment was scrubbed...
>>URL:
>>https://list.scms.waikato.ac.nz/mailman/private/greenstone-users/
>>attachm
>>ents/20051019/05ec692b/attachment.html
>>_______________________________________________
>>greenstone-users mailing list
>>greenstone-users@list.scms.waikato.ac.nz
>>https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users
>>
>>
>>_______________________________________________
>>greenstone-users mailing list
>>greenstone-users@list.scms.waikato.ac.nz
>>https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users
>>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> https://list.scms.waikato.ac.nz/mailman/private/greenstone-users/attachm
> ents/20051019/369fd314/attachment.html
> _______________________________________________
> greenstone-users mailing list
> greenstone-users@list.scms.waikato.ac.nz
> https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users
>
>
> _______________________________________________
> greenstone-users mailing list
> greenstone-users@list.scms.waikato.ac.nz
> https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users
>
>