RE: [greenstone-users] Out of Memory Error

From Nathan Einwechter
DateTue, 18 Oct 2005 18:06:44 -0400
Subject RE: [greenstone-users] Out of Memory Error
In-Reply-To (1598F9AA-6381-45B0-8559-C76C5B3DFB41-dlconsulting-co-nz)
They're log files. BUT - I can't really create an import plugin as the
files have to go through a whole bunch of different types of processing,
analysis, etc. by the system before being ready to be pushed to
Greenstone.. I guess I could push the results into a specific format and
then create an import plugin for that format though.

Good idea.

-- Nathan

-----Original Message-----
From: greenstone-users-bounces@list.scms.waikato.ac.nz
[mailto:greenstone-users-bounces@list.scms.waikato.ac.nz] On Behalf Of
John Rowe
Sent: October 18, 2005 5:54 PM
To: greenstone-users@list.scms.waikato.ac.nz
Subject: Fwd: [greenstone-users] Out of Memory Error

What are the original source documents? Can you build a custom
import plugin to import the original files rather than impo

Cheers,
John Rowe

DL Consulting
Greenstone Digital Library and Digitisation Specialists
contact@dlconsulting.co.nz
www.dlconsulting.co.nz


Begin forwarded message:

> From: "Nathan Einwechter" <nathan@inorb.com>
> Date: 19 October 2005 10:38:51 AM
> To: <greenstone-users@list.scms.waikato.ac.nz>
> Subject: RE: [greenstone-users] Out of Memory Error
>
>
> The problem is that this is ALL custom. I'm actually using it for
> hierarchies of security log information (IP addresses, ports,
> protocols). This is all generated by a program I have created.
>
> I'm thinking about just exporting to a self designed XML repository at
> this stage though. Greenstone has been good so far, but we're
> looking at
> scaling up even farther and I don't know how it can be done using
> GSDL.
>
> -- Nathan
>
> -----Original Message-----
> From: greenstone-users-bounces@list.scms.waikato.ac.nz
> [mailto:greenstone-users-bounces@list.scms.waikato.ac.nz] On Behalf Of
> John Rowe
> Sent: October 18, 2005 5:28 PM
> To: greenstone-users@list.scms.waikato.ac.nz
> Subject: Fwd: [greenstone-users] Out of Memory Error
>
> What is contained in your metadata.xml? What filetypes are in
> your collection? Is there any way you could build an equivalent
> collection without using the metadata.xml? Have you experimented with
> a custom import plugin?
>
> Cheers,
> John Rowe
>
> DL Consulting
> Greenstone Digital Library and Digitisation Specialists
> contact@dlconsulting.co.nz
> www.dlconsulting.co.nz
>
>
> Begin forwarded message:
>
>
>> From: "Nathan Einwechter" <nathan@inorb.com>
>> Date: 19 October 2005 10:04:14 AM
>> To: <greenstone-users@list.scms.waikato.ac.nz>
>> Subject: RE: [greenstone-users] Out of Memory Error
>>
>>
>> Does anyone know the maximum size I can have for a metadata.xml
>> file for
>> a given amount of ram etc.?
>>
>> -- Nathan
>>
>> -----Original Message-----
>> From: greenstone-users-bounces@list.scms.waikato.ac.nz
>> [mailto:greenstone-users-bounces@list.scms.waikato.ac.nz] On
>> Behalf Of
>> John Rowe
>> Sent: October 17, 2005 10:03 PM
>> To: greenstone-users@list.scms.waikato.ac.nz
>> Subject: Fwd: [greenstone-users] Out of Memory Error
>>
>> it seems that the Greenstone import process is simply not built
>> to handle a metadata file that large. The easiest solution would be
>> to split your import directory (and metadata.xml file) up into
>> smaller directories under the main import directory. The filesystem
>> should end up looking something like the following:
>> import/
>> import/001/metadata.xml
>> import/001/file1
>> import/001/file2
>> import/001/.....
>> import/001/file9
>> import/002/metadata.xml
>> import/002/file10
>> import/002/file11
>> import/002/...
>> import/002/file19
>> import/003/metadata.xml
>> import/003/file20
>> import/001/file21
>> .............
>>
>> Hope this helps.
>>
>> Cheers,
>> John Rowe
>>
>> DL Consulting
>> Greenstone Digital Library and Digitisation Specialists
>> contact@dlconsulting.co.nz
>> www.dlconsulting.co.nz
>>
>>
>>
>> _______________________________________________
>> greenstone-users mailing list
>> greenstone-users@list.scms.waikato.ac.nz
>> https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users
>>
>>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> https://list.scms.waikato.ac.nz/mailman/private/greenstone-users/
> attachm
> ents/20051019/05ec692b/attachment.html
> _______________________________________________
> greenstone-users mailing list
> greenstone-users@list.scms.waikato.ac.nz
> https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users
>
>
> _______________________________________________
> greenstone-users mailing list
> greenstone-users@list.scms.waikato.ac.nz
> https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
https://list.scms.waikato.ac.nz/mailman/private/greenstone-users/attachm
ents/20051019/369fd314/attachment.html
_______________________________________________
greenstone-users mailing list
greenstone-users@list.scms.waikato.ac.nz
https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users