giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <>
Subject [jira] [Commented] (GIRAPH-488) ArrayOutOfBoundsException in org.apache.giraph.worker.InputSplitPathOrganizer
Date Wed, 23 Jan 2013 04:10:13 GMT


Eli Reisman commented on GIRAPH-488:

Yes I believe so, until Alessandro had to upgrade the split organizer code
for multiple threads, this same solution made everything work correctly
where that  array index error is concerned, at least for the last 6 months.
This patch should fix the problem here too. I ran tests and a few jobs
today and it ran fine. he had me look over his fix, but I didn't realize
this had changed.

I tried to add a line or two of comments to the patch to make clear to
future devs why the base offset in the split organizer must be calculated
for the full split list if the locality is off, and calculated for the
nonlocality list as we are inserting the locality list back into it, using
the nonlocal list size (a bit smaller if we found some local blocks in the
split list we're scanning) thats what it does now (and did before when it

Its the kind of error that only comes up when an unlucky hash gets
generated, so if Sebastian's job sets this off, I'd like to see him patch
this into trunk, and run his job again to see what happens just to be extra
certain, but I am very confident this takes care of the issue.

> ArrayOutOfBoundsException in org.apache.giraph.worker.InputSplitPathOrganizer
> -----------------------------------------------------------------------------
>                 Key: GIRAPH-488
>                 URL:
>             Project: Giraph
>          Issue Type: Bug
>    Affects Versions: 0.2.0
>            Reporter: Sebastian Schelter
>            Assignee: Eli Reisman
>         Attachments: GIRAPH-488-1.patch
> I ran into a strange exception when testing the RandomWalkVertex on a cluster of 26 machines
running Hadoop 1.0.4
> {noformat}
> java.lang.IllegalStateException: run: Caught an unrecoverable exception Index: 225, Size:
> 	at
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(
> 	at
> 	at org.apache.hadoop.mapred.Child$
> 	at Method)
> 	at
> 	at
> 	at org.apache.hadoop.mapred.Child.main(
> Caused by: java.lang.IndexOutOfBoundsException: Index: 225, Size: 205
> 	at java.util.ArrayList.rangeCheckForAdd(
> 	at java.util.ArrayList.addAll(
> 	at org.apache.giraph.worker.InputSplitPathOrganizer.prioritizeLocalInputSplits(
> 	at org.apache.giraph.worker.InputSplitPathOrganizer.<init>(
> 	at org.apache.giraph.worker.InputSplitsCallable.<init>(
> 	at org.apache.giraph.worker.VertexInputSplitsCallable.<init>(
> 	at 
> org.apache.giraph.worker.VertexInputSplitsCallableFactory.newCallable(
> 	at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(
> 	at org.apache.giraph.worker.BspServiceWorker.loadVertices(
> 	at org.apache.giraph.worker.BspServiceWorker.setup(
> 	at
> 	at
> 	... 7 more
> {noformat}

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message