Re: [greenstone-users] Database limit

From Stuart A Yeates
DateTue, 14 Oct 2003 11:07:02 +1300
Subject Re: [greenstone-users] Database limit
In-Reply-To (Pine-LNX-4-33-0310140137150-1132-100000-mmsl-serc-iisc-ernet-in)
Anandh Jayaraman wrote:
> I want to build 1.5 TB data (more than 29,000 books and thesis) into
> greenstone. Is it possible? If so, how long will it take to import and
> build.? System details i've given below.
>
> i have tried for 100 MB data (Text files), it took 340 minutes and 19
> seconds to finish.
>
> I have tried it in windows 2000 professional OS, 512MB RAM,and Pentium 4.
> I used gsdl 2.4 greenstone version.
>
> If it is not enough then what all resources i need to build that
> much huge amount of data, into greenstone?

Hello Anandh

First off, a disclaimer: I know nothing about windows 2000.

Building a 1.5 TB collection in greenstone is unlikely to be straight
forword, because there are issues on many operating systems and much
supporting software with files larger than 2-4 GB. Such systems are very
hard to debug and test because of their sheer size and the time it takes
to build or rebuild them.

What I suggest you build several small collections, each in the range of
several hundred MB and use cross-collection searching to search across
them as though they were a single collection. Cross collection searching
is discussed in the Developers Guide (starting on page 25 of the enlish
language version).

I hope this helps.
stuart