hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-31) concurrent modification error
Date Wed, 28 Nov 2007 18:25:43 GMT

    [ https://issues.apache.org/jira/browse/PIG-31?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546335
] 

Alan Gates commented on PIG-31:
-------------------------------

I am fine with BigDataBag.finalize() not calling DataBag.clear().  The question I still have
is should DataBag.clear() call content.clear().  Shouldn't it instead allocate a new content
object, so that if someone still has a reference to the old content object they aren't left
to die with a comodification error?  If we think that once someone calls DataBag.clear() then
any other instances of content should be declared invalid, we at least need to give a way
to check that, so that we don't encounter this error again and again have no idea how we got
it.

> concurrent modification error
> -----------------------------
>
>                 Key: PIG-31
>                 URL: https://issues.apache.org/jira/browse/PIG-31
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Olga Natkovich
>            Assignee: Olga Natkovich
>            Priority: Critical
>         Attachments: concurrentmod.patch
>
>
> Many users encountered this problem. The stack looks like:
> Error:java.lang.RuntimeException
> 	at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> 	at java.util.AbstractList$Itr.next(AbstractList.java:343)
> 	at com.yahoo.pig.builtin.SUM.sum(SUM.java:58)
> 	at com.yahoo.pig.builtin.SUM.exec(SUM.java:25)
> 	at com.yahoo.pig.builtin.SUM.exec(SUM.java:21)
> 	at com.yahoo.pig.impl.eval.FuncEvalSpec$1.add(FuncEvalSpec.java:101)
> 	at com.yahoo.pig.impl.eval.GenerateSpec$CrossProductItem.(GenerateSpec.java:140)
> 	at com.yahoo.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:52)
> 	at com.yahoo.pig.impl.eval.GenerateSpec$CrossProductItem.add(GenerateSpec.java:217)
> 	at com.yahoo.pig.impl.eval.collector.UnflattenCollector.add(UnflattenCollector.java:42)
> 	at com.yahoo.pig.impl.eval.collector.DataCollector.addToSuccessor(DataCollector.java:79)
> 	at com.yahoo.pig.impl.eval.SimpleEvalSpec$1.add(SimpleEvalSpec.java:21)
> 	at com.yahoo.pig.impl.eval.GenerateSpec$CrossProductItem.exec(GenerateSpec.java:248)
> 	at com.yahoo.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:61)
> 	at com.yahoo.pig.impl.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:144)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:355)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1707)
> We traced the issue to the same bag being shared between multiple operations and clearing
it in one place caused problem in another. Will attach Ben's fix shortly. All unit and end-to-end
tests passed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message