hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21785) Add task queue/runtime stats per LLAP daemon to output
Date Fri, 31 May 2019 23:39:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853483#comment-16853483
] 

Hive QA commented on HIVE-21785:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970510/HIVE-21785.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16049 tests executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=156)
	[intersect_all.q,unionDistinct_1.q,table_nonprintable.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,results_cache_diff_fs.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q]
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
(batchId=248)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17378/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17378/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17378/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12970510 - PreCommit-HIVE-Build

> Add task queue/runtime stats per LLAP daemon to output
> ------------------------------------------------------
>
>                 Key: HIVE-21785
>                 URL: https://issues.apache.org/jira/browse/HIVE-21785
>             Project: Hive
>          Issue Type: Improvement
>          Components: llap
>    Affects Versions: 3.1.1
>            Reporter: Oliver Draese
>            Assignee: Oliver Draese
>            Priority: Major
>             Fix For: 3.1.1
>
>         Attachments: HIVE-21785.1.patch, HIVE-21785.patch
>
>
> There are several scenarios, where we want to investigate if a particular LLAP daemon
is performing faster or slower than the others in the cluster. In these scenarios, it is specifically
important to figure out if tasks spent significant time, waiting for an available executor
(queued) vs. on the execution itself. Also, a skew in task-to-daemon assignment is interesting.
> This patch adds these statistics to the TezCounters and therefore to the job output on
a per LLAP daemon base. Here is an example.
> {{INFO : LlapTaskRuntimeAgg by daemon:}}
> {{INFO :    Count-host-1.example.com: 41}}
> {{INFO :    Count-host-2.example.com: 39}}
> {{INFO :    Count-host-3.example.com: 45}}
> {{INFO :    QueueTime-host-1.example.com: 51437776}}
> {{INFO :    QueueTime-host-2.example.com: 35758306}}
> {{INFO :    QueueTime-host-3.example.com: 47168327}}
> {{INFO :    RunTime-host-1.example.com: 165151539295}}
> {{INFO :    RunTime-host-2.example.com: 141729193528}}
> {{INFO :    RunTime-host-3.example.com: 166876988771}}
> The "Count-" are simple task counts for the appended host name (LLAP daemon)
> The "QueueTime-" values tell, how long tasks waited in the TaskExecutorService's queue
before getting actually executed.
> The "RunTime-" values cover the time from execution start to finish (where finish can
either be successful execution or a killed/failed execution).
> For the new counts to appear in the output, both - the preexisting hive.tez.exec.print.summary
and the new hive.llap.task.time.print.summary have to be set to true.
>  
> {{<property>}}
> {{  <name>hive.tez.exec.print.summary</name>}}
> {{  <value>true</value>}}
> {{</property>}}
> {{<property>}}
> {{  <name>hive.llap.task.time.print.summary</name>}}
> {{  <value>true</value>}}
> {{</property>}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message