hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Utkarsh Srivastava (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-31) concurrent modification error
Date Wed, 28 Nov 2007 18:17:43 GMT

    [ https://issues.apache.org/jira/browse/PIG-31?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546329

Utkarsh Srivastava commented on PIG-31:

+1 to Ben's patch.

The point of the finalize method is to collect the temp files left around by the big bag.
It should not be calling content.clear().

If the users decide to call databag.clear() while they have an iterator open on the bag, its
only right that they get a CoMod exception.

Internally we don't call clear except at one place, where also, it is not strictly necessary.


> concurrent modification error
> -----------------------------
>                 Key: PIG-31
>                 URL: https://issues.apache.org/jira/browse/PIG-31
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Olga Natkovich
>            Assignee: Olga Natkovich
>            Priority: Critical
>         Attachments: concurrentmod.patch
> Many users encountered this problem. The stack looks like:
> Error:java.lang.RuntimeException
> 	at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> 	at java.util.AbstractList$Itr.next(AbstractList.java:343)
> 	at com.yahoo.pig.builtin.SUM.sum(SUM.java:58)
> 	at com.yahoo.pig.builtin.SUM.exec(SUM.java:25)
> 	at com.yahoo.pig.builtin.SUM.exec(SUM.java:21)
> 	at com.yahoo.pig.impl.eval.FuncEvalSpec$1.add(FuncEvalSpec.java:101)
> 	at com.yahoo.pig.impl.eval.GenerateSpec$CrossProductItem.(GenerateSpec.java:140)
> 	at com.yahoo.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:52)
> 	at com.yahoo.pig.impl.eval.GenerateSpec$CrossProductItem.add(GenerateSpec.java:217)
> 	at com.yahoo.pig.impl.eval.collector.UnflattenCollector.add(UnflattenCollector.java:42)
> 	at com.yahoo.pig.impl.eval.collector.DataCollector.addToSuccessor(DataCollector.java:79)
> 	at com.yahoo.pig.impl.eval.SimpleEvalSpec$1.add(SimpleEvalSpec.java:21)
> 	at com.yahoo.pig.impl.eval.GenerateSpec$CrossProductItem.exec(GenerateSpec.java:248)
> 	at com.yahoo.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:61)
> 	at com.yahoo.pig.impl.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:144)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:355)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1707)
> We traced the issue to the same bag being shared between multiple operations and clearing
it in one place caused problem in another. Will attach Ben's fix shortly. All unit and end-to-end
tests passed.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message