[greenstone-users] Re: Problem of Slow collection loading in GLI

From Katherine Don
DateWed Jan 27 16:17:16 2010
Subject [greenstone-users] Re: Problem of Slow collection loading in GLI
In-Reply-To (EE16C1684DDF4379B5B63D9DEEA83758-caominhkiem)
Yes, GLI can be slow in loading very large collections.

I suggest you use command line building.

To use GLI to add documents and metadata, create a new empty dummy
collection. Add documents to this collection, but make sure they are in
a subfolder in the collection. Add metadata to the documents as usual.
Then, on the command line, move the folders from the dummy collection's
import folder into your main collections import folder.
They need to be in a subfolder so the metadata.xml file doesn't
overwrite any that is in there already.

Then, you can import and build the collection on the command line.
If you have a later version of greenstone (2.82 or 2.83 eg), you can do
incremental import of the new documents so you don't need to reimport
everything again. And if your collection uses Lucene (instead of
mg/mgpp) you can do incremental indexing too.


Cao Minh Kiem wrote:
> Dear GSDL users,
> We would like to ask for your help. We create a digital document
> collection using GLI. At this time, the size of collection is about 10
> Gb. Every time when we open GLI to add new documents and metadata, it
> take us very long (more than 40 minutes) to load the collection.
> Is it the capability of GLI to load a collection? If not, could you
> help us in solving this issue?
> Best regards
> Cao Minh Kiem
> Deputy Director
> National Centre for S&T Information
> 24 Ly Thuong Kiet, Hanoi, VIETNAM
> Email: kiemcm@vista.gov.vn