[greenstone-users] Problem Building Remote Collection

From Daniel Ishimitsu
DateTue Apr 14 09:10:14 2009
Subject [greenstone-users] Problem Building Remote Collection
In-Reply-To (49B6F9A1-6060408-cs-waikato-ac-nz)
Hi Anupama,

I think I figured it out. Replying back to the list in case anyone has
similar problems. The StackOverflowError i was getting was from trying
to output the user-friendly error message (which listed 1000's of files)
and not the import/build itself. I added an additional flag to the .bat
file -Xss16M which bumped the stack space (instead of heap space) to 16M
from a default of 2M. Once the user-friendly error message was showing
and nothing else went wrong, I found the files which were rejected by
the import, corrected them, and everything is fine again. Thanks for
your help in troubleshooting this problem.

Daniel Ishimitsu

University of Hawaii
Hamilton Library, DNS

Anupama of Greenstone Team wrote:
> Hi again Daniel,
>
> I'm so sorry, but I made a mistake in my last email: you are using the
> remote greenstone server of course, yet, instead of directing you to
> editing the appropriate client-gli.sh/client-gli.bat scripts, I asked
> you to modify the gli scripts that are used with a local greenstone
> server.
>
> 1. Will you please try my suggestion given in the previous email
> again, but this time make the edits to the client-gli script
> appropriate to the operating system your running the remote client on
> (.sh file for Linux, .bat on Windows).
>
>
> > Let me know if there's any other data I could give you to assist in
> > the troubleshooting.
>
> Thanks. The idea was that we'd try to generate test data consisting of
> about 1200 small files and attempt something similar to what you've
> been going through and then debug the code. However, if (1) above does
> not work out for you, and if your remote server is available to the
> public to access, we can try rebuilding your collection from my
> machine using a remote client here and try to debug the client code
> where this stack exception is occurring. This may be a little
> difficult since we can't modify the gli server code on your end with
> debug statements, but it's apparently a GLI problem rather than
> anything to do with the server.
>
> Fingers crossed and sorry again about the error I made,
> Anupama
>
>
> Daniel Ishimitsu wrote:
>> Hi Anupama,
>>
>> Thanks for your response. I changed the 128M to 512M in gli.sh on
>> the server, but the same error appears at the same point. Let me know
>> if there's any other data I could give you to assist in the
>> troubleshooting.
>>
>> Daniel Ishimitsu
>>
>> University of Hawaii
>> Hamilton Library, DNS
>>
>>
>>
>> Anupama of Greenstone Team wrote:
>>> Hi Daniel,
>>>
>>> We'll have to investigate your problem, but in the meantime, could
>>> you try evading it by allocating a larger stacksize to the Java
>>> Virtual Machine when running GLI.
>>>
>>>
>>> 1. Open the file gli.bat (if you're on Windows) or gli.sh (if you're
>>> on Linux) in a very basic text editor. If on Windows, I'd advise you
>>> to open it in Notepad rather than Wordpad.
>>>
>>>
>>> 2 a. If you're on Linux, then in *gli.sh*, find the line:
>>>
>>> basic_command="$javapath -Xmx128M -classpath
>>> classes/:GLI.jar:lib/apache.jar:lib/qfslib.jar
>>> org.greenstone.gatherer.GathererProg"
>>>
>>> b. If you're on Windows, then in *gli.bat* find the TWO occurrences
>>> of the line
>>> - "%JAVA_EXECUTABLE%" -Xmx128M -cp
>>> classes/;GLI.jar;lib/apache.jar;lib/qfslib.jar ....
>>>
>>>
>>> 3. Change the occurrences of "-Xmx128M" to "-Xmx256M". Try running
>>> GLI again and see if the StackOverflow problem is gone.
>>>
>>>
>>> 4. If the above did not work, repeate but by changing the value to
>>> -Xmx512M.
>>> This is now allocating half a Gigabyte of memory to the program,
>>> instead of the original 128 Megabytes.
>>>
>>>
>>> Tell us if the above doesn't help.
>>>
>>> Hopefully, it will allow you to finally get on with your work, while
>>> we look into the real causes behind it.
>>>
>>> Regards,
>>> Anupama
>>>
>>>
>>>
>>> Daniel Ishimitsu wrote:
>>>> Hi Everyone,
>>>>
>>>> Any help would be appreciated! I have a Greenstone collection that
>>>> consists of 1,200 web pages with a total of 16,000 images (gif and
>>>> jpg). I'm trying to build the collection using the remote GLI (I
>>>> tried both the applet and the separate client program) but I get a
>>>> StackOverflowError along the way.
>>>> Loading the collection is fine (it takes about 4 minutes), I click
>>>> on the Create tab, then click on Complete Rebuild, then Build
>>>> Collection. After about 15 minutes I get an import complete
>>>> message, but the program seems stuck in a loop and the console
>>>> displays the stackoverflowerror. Below is a snipped version of the
>>>> logs and also the stacktrace.
>>>> Other added bits of messiness:
>>>> We're hosting the server on a linux machine, but using the remote
>>>> client/applet from a windows machine. The collection was initially
>>>> built on a windows box and moved to the linux machine (I tried
>>>> creating it from scratch remotely but the Gather stage stopped
>>>> responding after a few hundred images were uploaded)
>>>>
>>>>
>>>> Greenstone Message Log:
>>>> ************** Import Started **************
>>>> The file is being processed by IndexPlugin.
>>>> The file ttp_htms/3146.html is being processed by
>>>> StructuredHTMLPlugin.
>>>> The file ttp_htms/2847.html is being processed by
>>>> StructuredHTMLPlugin.
>>>> The file ttp_htms/2482.html is being processed by
>>>> StructuredHTMLPlugin.
>>>> The file ttp_htms/4038.html is being processed by
>>>> StructuredHTMLPlugin.
>>>> ...
>>>> The file ttp_htms/3026.html is being processed by
>>>> StructuredHTMLPlugin.
>>>> The file ttp_htms/3101.html is being processed by
>>>> StructuredHTMLPlugin.
>>>> ************** Import Finished **************
>>>> 1227 documents were considered for processing:
>>>> 1222 documents were processed and included in the collection.
>>>> 5 were rejected.
>>>>
>>>>
>>>> Console Output:
>>>> *********************************************
>>>> Import complete
>>>> *********************************************
>>>> <ImportComplete considered='1227' processed='1222' blocked='0'
>>>> ignored='0' failed='5'>
>>>> * 1227 documents were considered for processing
>>>> * 1222 were processed and included in the collection
>>>> * 5 were rejected
>>>> See /opt/gsdl/collect/trustter/etc/fail.log for a list of
>>>> unrecognised and/or rejected documents
>>>>
>>>> at
>>>> org.greenstone.gatherer.remote.RemoteGreenstoneServer.sendCommandToServer(RemoteGreenstoneServer.java:505)
>>>>
>>>> at
>>>> org.greenstone.gatherer.remote.RemoteGreenstoneServerAction$RunScriptAction.perform(RemoteGreenstoneServerAction.java:537)
>>>>
>>>> at
>>>> org.greenstone.gatherer.remote.ActionQueue.run(ActionQueue.java:134)
>>>> Exception in thread "RemoteGreenstoneServerActionQueue"
>>>> java.lang.StackOverflowError
>>>> at sun.awt.AppContext.get(Unknown Source)
>>>> at
>>>> com.sun.java.swing.SwingUtilities3.getDelegateRepaintManager(Unknown
>>>> Source)
>>>> at javax.swing.RepaintManager.getDelegate(Unknown Source)
>>>> at javax.swing.RepaintManager.addDirtyRegion(Unknown Source)
>>>> at javax.swing.JComponent.repaint(Unknown Source)
>>>> at java.awt.Component.repaint(Unknown Source)
>>>> at javax.swing.JComponent.setFont(Unknown Source)
>>>> at javax.swing.LookAndFeel.installColorsAndFont(Unknown Source)
>>>> at
>>>> javax.swing.plaf.basic.BasicLabelUI.installDefaults(Unknown Source)
>>>> at javax.swing.plaf.basic.BasicLabelUI.installUI(Unknown
>>>> Source)
>>>> at javax.swing.JComponent.setUI(Unknown Source)
>>>> at javax.swing.JLabel.setUI(Unknown Source)
>>>> at javax.swing.JLabel.updateUI(Unknown Source)
>>>> at javax.swing.JLabel.<init>(Unknown Source)
>>>> at javax.swing.JLabel.<init>(Unknown Source)
>>>> at
>>>> javax.swing.plaf.basic.BasicOptionPaneUI.addMessageComponents(Unknown
>>>> Source)
>>>> at
>>>> javax.swing.plaf.basic.BasicOptionPaneUI.addMessageComponents(Unknown
>>>> Source)
>>>> at
>>>> javax.swing.plaf.basic.BasicOptionPaneUI.addMessageComponents(Unknown
>>>> Source)
>>>> at
>>>> javax.swing.plaf.basic.BasicOptionPaneUI.addMessageComponents(Unknown
>>>> Source)
>>>> at
>>>> javax.swing.plaf.basic.BasicOptionPaneUI.addMessageComponents(Unknown
>>>> Source)
>>>> ...(continues for about 1000 times)
>>>>
>>>> Thanks,
>>>>
>>>
>>
>