hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmytro Molkov (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HIVE-1349) Do not pull counters of non initialized jobs
Date Tue, 18 May 2010 20:43:42 GMT

     [ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Dmytro Molkov updated HIVE-1349:

    Attachment: HIVE-1349.patch

Hey guys, take a look at the patch please.
When the job is created the status is PREP, it is switched to RUNNING when the setup is finished,
so while the job status is PREP we are skipping the polling of counters.
The poll interval is also made configurable, so hive can hit jobtrackers getCounter which
is not perfect less frequently.

> Do not pull counters of non initialized jobs
> --------------------------------------------
>                 Key: HIVE-1349
>                 URL: https://issues.apache.org/jira/browse/HIVE-1349
>             Project: Hadoop Hive
>          Issue Type: Improvement
>            Reporter: Dmytro Molkov
>         Attachments: HIVE-1349.patch
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters
of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress
lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock.
Since the read of the file can  take a couple (a couple of tens of seconds) the job tracker
is effectively blocked for that period of time.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message