hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bennie Schut (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1671) multithreading on Context.pathToCS
Date Mon, 27 Sep 2010 10:11:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915262#action_12915262
] 

Bennie Schut commented on HIVE-1671:
------------------------------------

Perhaps because we now have multiple sub queries running in hive for the same overall query
we can have concurrent use of this map?
We could simply fix this by using the ConcurrentHashMap


  private Map<String, ContentSummary> pathToCS = new ConcurrentHashMap<String, ContentSummary>();


> multithreading on Context.pathToCS
> ----------------------------------
>
>                 Key: HIVE-1671
>                 URL: https://issues.apache.org/jira/browse/HIVE-1671
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Bennie Schut
>             Fix For: 0.7.0
>
>
> we having 2 threads running at 100%
> With a stacktrace like this:
> "Thread-16725" prio=10 tid=0x00007ff410662000 nid=0x497d runnable [0x00000000442eb000]
>    java.lang.Thread.State: RUNNABLE
>         at java.util.HashMap.get(HashMap.java:303)
>         at org.apache.hadoop.hive.ql.Context.getCS(Context.java:524)
>         at org.apache.hadoop.hive.ql.exec.Utilities.getInputSummary(Utilities.java:1369)
>         at org.apache.hadoop.hive.ql.exec.MapRedTask.estimateNumberOfReducers(MapRedTask.java:329)
>         at org.apache.hadoop.hive.ql.exec.MapRedTask.setNumberOfReducers(MapRedTask.java:297)
>         at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:84)
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:108)
>         at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>         at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message