pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3241) ConcurrentModificationException in POPartialAgg
Date Mon, 11 Mar 2013 18:31:14 GMT

    [ https://issues.apache.org/jira/browse/PIG-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599097#comment-13599097
] 

Rohini Palaniswamy commented on PIG-3241:
-----------------------------------------

I was wrong about iterator.remove() of HashMap throwing ConcurrentModificationException. Happens
only while iterating directly over the entrySet()

{code}
for (Entry<String, String> entry : map.entrySet()) {
            map.remove(entry.getKey());
        }
{code}
throws ConcurrentModificationException

{code}
Iterator<Entry<String, String>> spillingIterator = map.entrySet().iterator();
Entry<String, String> entry = spillingIterator.next();
spillingIterator.remove();
{code}
does not throw ConcurrentModificationException

  So theoretically it should be fine if the map was accessed within a single thread. Need
to investigate if there is more than one thread accessing POPartialAgg. Possible that it could
be a 2.0 issue. Simple fix would be to change that to ConcurrentHashMap, but need to find
the underlying cause to assess further impact. I am blocked with few deadlines before going
on vacation and don't have time to spare for the investigation. I am fine if anyone else can
take a look at this. Cheolsoo?

[~lohit],
   Can you change it to ConcurrentHashMap and compile and see that fixes the problem for you?
 
                
> ConcurrentModificationException in POPartialAgg
> -----------------------------------------------
>
>                 Key: PIG-3241
>                 URL: https://issues.apache.org/jira/browse/PIG-3241
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.11
>            Reporter: Lohit Vijayarenu
>            Priority: Blocker
>             Fix For: 0.12, 0.11.1
>
>
> While running few PIG scripts against Hadoop 2.0, I see consistently see ConcurrentModificationException

> {noformat}
> at java.util.HashMap$HashIterator.remove(HashMap.java:811)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.aggregate(POPartialAgg.java:365)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.aggregateSecondLevel(POPartialAgg.java:379)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.getNext(POPartialAgg.java:203)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:263)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:283)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:729)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
> {noformat}
> It looks like there is rawInputMap is being modified while elements are removed from
it. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message