pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4700) Enable progress reporting for Tasks in Tez
Date Mon, 05 Jun 2017 17:40:04 GMT

    [ https://issues.apache.org/jira/browse/PIG-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037262#comment-16037262
] 

Rohini Palaniswamy commented on PIG-4700:
-----------------------------------------

bq. Could you make it 1000? i.e schedule it every second
  Actually it can be optimized better. Progress information is sent as part of task heartbeat.
In setups like ours heartbeat interval is set to 3 seconds so that the AM can handle 10s of
thousands of tasks. So we should set it to a value slightly lower than heartbeat interval.

{code}
progressHelper.scheduleProgressTaskService(100, Math.max(1000, (conf.getInt(TezConfiguration.TEZ_TASK_AM_HEARTBEAT_INTERVAL_MS,
TezConfiguration.TEZ_TASK_AM_HEARTBEAT_INTERVAL_MS_DEFAULT) - 50)));
{code}

> Enable progress reporting for Tasks in Tez
> ------------------------------------------
>
>                 Key: PIG-4700
>                 URL: https://issues.apache.org/jira/browse/PIG-4700
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Satish Subhashrao Saley
>             Fix For: 0.18.0
>
>         Attachments: PIG-4700-1.patch, PIG-4700-2.patch, PIG-4700-3.patch, PIG-4700-4.patch,
PIG-4700-5.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message