Re: [greenstone-users] AZLists in very large collections

From Rene Schrama
DateThu, 04 Dec 2003 16:38:08 +0100
Subject Re: [greenstone-users] AZLists in very large collections
Hi Stefan and Katherine,

Thanks for your quick response. I added 256 MB of memory so now I have
512 MB. I increased the paging file to 4 GB (fixed size). The 6,500
documents took 7 minutes to build (4 times faster). The entire
collection (52,000) took about 12 hours, displayed an out of memory
message and then just stopped without completing the infodb phase. I
then rebuilt the 6,500 documents with only the hierarchy classifier,
without the hierarchy classifier but with all other classifiers, and
with all classifiers. The regular classifiers affect PF usage (100+ MB
is normal) only a little and take it to 200+ MB, but whenever the
hierarchy classifier is present it jumps to 600+ MB, with or without the
other classifiers. Btw the hierarchy file (thesaurus structure) is about
10,000 lines, maybe this could have some influence (?). Anyway, I don't
think it will ever build in the current version so it's probably best to
drop the thesaurus for now and either put the CD on hold or go ahead
without the thesaurus. Tonight I will rebuild the collection without the
hierarchy classifier and let you know what happened.