hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (HIVE-4679) WebHCat can deadlock Hadoop if the number of concurrently running tasks if higher or equal than the number of mappers
Date Thu, 06 Jun 2013 21:24:20 GMT

     [ https://issues.apache.org/jira/browse/HIVE-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eugene Koifman reassigned HIVE-4679:
------------------------------------

    Assignee: Eugene Koifman
    
> WebHCat can deadlock Hadoop if the number of concurrently running tasks if higher or
equal than the number of mappers
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-4679
>                 URL: https://issues.apache.org/jira/browse/HIVE-4679
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>    Affects Versions: 0.10.0
>            Reporter: Eugene Koifman
>            Assignee: Eugene Koifman
>
> o In the current Templeton design, each time a Job is submitted thru the REST API (it
can be Pig/Hive or MR job), it will consume one Hadoop map slot. Given that the number of
map slots is finite in the cluster (16 node cluster will have 32 map slots), in some circumstances,
a user can deadlock the cluster if Templeton job submission pipeline takes over all map slots
(Templeton map tasks will wait for the actual underlying jobs to complete, what will never
happen, given that Hadoop has no free map slots to schedule new tasks).
> o HCat queries use a different mechanism and do not contribute to the deadlock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message