[greenstone-users] Disk space

From William T. Mann
DateThu Oct 14 11:08:32 2010
Subject [greenstone-users] Disk space
In-Reply-To (004101cb6aec$c49a8c40$4dcfa4c0$-gov-ar)
Diego,

Thanks for the quick reply! This is a big help to setting up my build machine
properly.

Just one more thing: now that I've gone through the import process and my
archives directory is populated (and I've freed up the necessary space), is it
possible to start the build process where it left off? That is, can the build
process be started with the creating of the indexes (the building folder)
without having to go through another 2 days of processing? I'm using the
PagedImage component (without the cache for space reasons) and as I stated
before there are over 40200 images!

Thanks again!

--
William Mann

On Wednesday 13 October 2010 17:39:05 Diego Spano wrote:
> William,
>
> the way Greenstone stores all objects is the following: If you import 27 GB
> of jpg files, then you will have 27 GB in "import" folder, 27 GB in
> "archive" folder (after import process), and 27 GB in "building" folder
> (after buildcol process). The last step of creating a collection from GLI
> is to rename building folder as index, so now you don□t have building but
> you have index folder (with 27 GB).
>
> The problem you faced is because you create again the collection from GLI,
> so now you have 85 GB in import folder, 85 GB in archives folder (the
> previous content is deleted by default) and finally Greenstone needs
> another 85 GB for "building" folder. But you also have the old index
> folder (27 GB), so Greenstone is asking for 85 (building) plus 27 (index).
>
> You have 2 choices.
>
> Option 1- If you want to create collection from GLI, first delete index
> folder!
>
> Option 2- Run the process from command line (this is what I recommend). You
> can achieve more control on what you need.
> The import folder can reside anywhere. You can even take the files from
> another computer. You can mount a Windows share on a lInux server. You can
> have two different disks, etc etc. You don□t have the need to mount the
> file system as import folder. You can have different filesystems too.
> Assume your collection name is "pictures" and that you have the files on a
> filesystem mounted on /my_docs.
>
> What you need is to redirect the import folder, assigning "importdir"
> option to the import.pl process. (this cannot be done through GLI).
>
> Perl -S import.pl -importdir /my_docs pictures
>
> The archive folder can be redirected too, to any other folder or shared
> resource. In this case, you need to specify the location to both process,
> the import and the build.
>
> Perl -S import.pl -importdir /my_docs -archivedir /archives pictures
> Perl -S buildcol.pl -archivedir /archives pictures
>
> The index folder must reside in the same location where you have GS
> installed. But there is another way to reduce space. You can notice that
> the contents of the archive folder are "almost" the same of "index/assoc"
> folder. So, if you are on Linux, you can make a link like index/assoc -->
> /archives and then you get more space. On Windows this is not possible, as
> far as I know....
>
>
> It is like a puzzle!.
>
> Hope this helps.
>
> Diego
>
> Diego Spano
> Prodigio Consultores
> Bernardo de Irigoyen N□ 1114 2□B
> Capital Federal - Argentina
> Tel: (54 11) 5093-5313
> www.prodigioconsultores.com
>
>
>
> -----Mensaje original-----
> De: greenstone-users-bounces@list.scms.waikato.ac.nz
> [mailto:greenstone-users-bounces@list.scms.waikato.ac.nz] En nombre de
> William Mann
> Enviado el: Mi□rcoles, 13 de Octubre de 2010 06:08 a.m.
> Para: greenstone-users@list.scms.waikato.ac.nz
> Asunto: [greenstone-users] Disk space
>
> Hi,
>
> I've been using Greenstone with a very small (27 GB) test collection and
> now I have to build the final collection that consists of 40200+ images
> (about 85 GB). Since the first attempt to build ended with a disk space
> error, I then added a second disk to my computer and mounted the
> partitions as import and archives, also disabling the cache so the cached
> dir wouldn't fill up. This time all the files got processed but the build
> ended the same with a disk space error. After searching my drive I found
> that there was now a building and index directory: the first with 62GB of
> data and the second with 27GB of data. Since I've found that to process
> the files and start the building of the collection takes a little more
> than 2 days (at least using gli), I was wondering exactly what do I need
> to put on external disks to keep my drive from running out of space? Why
> is it making so many copies of my data in different places?
>
> --
> Rag. William Mann
> Comune di Belluno
> Servizio Sistemi Informativi
> Piazza Castello, 14
> 32100 Belluno
> Tel. 0437-913156
> e-mail: wtmann@comune.belluno.it

--
William T. Mann
Comune di Belluno