impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "anujphadke (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] Impala-3342: Adding thread counters to measure time spent during plan fragment execution
Date Mon, 17 Oct 2016 21:35:34 GMT
anujphadke has posted comments on this change.

Change subject: Impala-3342: Adding thread counters to measure time spent during plan fragment
execution
......................................................................


Patch Set 2:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/4633/2//COMMIT_MSG
Commit Message:

PS2, Line 7: Impala-3342 Adding thread counters to measure time spent during plan
           : fragment execution
> Please fix the formatting of the msg.
Done


PS2, Line 11: meausure
> measure
Done


PS2, Line 13: hdfs/kudu scanner and in a blocking join
> Why is this worth calling out? Doesn't this measure all exec nodes?
This change replaces every instance of the total_cpu_timer. Replacing this timer in those
2 places and adding the THREAD_COUNTERS which get aggregated in the plan-fragment-executor.

Adding thread counters in every thread would bulk up the profile.


Line 14: 
> What does the profile look like? Would be helpful to see what the plan frag
Fragment F01:
      Instance c64650f62b67d849:ff790c3600000002 (host=anuj-OptiPlex-9020:22000):(Total: 31.008ms,
non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:29/3.60
GB 
        MemoryUsage(500.000ms): 57.78 MB, 129.77 MB, 137.79 MB, 129.79 MB, 129.79 MB, 121.79
MB, 153.78 MB, 121.79 MB, 129.79 MB, 129.79 MB, 129.79 MB, 129.79 MB, 97.79 MB, 137.79 MB,
137.79 MB, 129.79 MB, 121.79 MB, 105.79 MB, 105.89 MB, 121.79 MB, 129.79 MB, 121.79 MB, 137.79
MB, 113.79 MB, 113.79 MB, 137.79 MB, 121.79 MB, 129.79 MB, 129.79 MB, 113.79 MB, 105.86 MB,
97.76 MB, 113.76 MB, 105.76 MB, 73.72 MB
        ThreadUsage(500.000ms): 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 5, 5, 5, 4
         - AverageThreadTokens: 5.86 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 169.83 MB (178079056)
         - PerHostPeakMemUsage: 1.17 GB (1256325736)
         - PlanFragmentThreadInvoluntaryContextSwitches: 4.32K (4321)
         - PlanFragmentThreadTotalWallClockTime: 1m42s
           - PlanFragmentThreadSysTime: 206.195ms
           - PlanFragmentThreadUserTime: 41s769ms
         - PlanFragmentThreadVoluntaryContextSwitches: 35.57K (35570)
         - PrepareTime: 30.293ms
         - RowsProduced: 30.00M (29999795)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 2s396ms
         - TotalStorageWaitTime: 920.765ms
        CodeGen:(Total: 41.328ms, non-child: 41.328ms, % non-child: 100.00%)
           - CodegenTime: 518.571us
           - CompileTime: 3.161ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.90 MB (1996720)
           - NumFunctions: 9 (9)
           - NumInstructions: 113 (113)
           - OptimizationTime: 7.696ms
           - PrepareTime: 30.095ms
        DataStreamSender (dst_id=5):(Total: 16s230ms, non-child: 16s230ms, % non-child: 100.00%)
           - BytesSent: 428.18 MB (448979535)
           - NetworkThroughput(*): 236.47 MB/sec
           - OverallThroughput: 26.38 MB/sec


http://gerrit.cloudera.org:8080/#/c/4633/2/be/src/runtime/plan-fragment-executor.cc
File be/src/runtime/plan-fragment-executor.cc:

Line 213:       ADD_COUNTER(profile(), PER_HOST_PEAK_MEM_COUNTER, TUnit::BYTES);
> Let's create the new counter here along with the other ones so that the set
Done


http://gerrit.cloudera.org:8080/#/c/4633/2/be/src/runtime/runtime-state.h
File be/src/runtime/runtime-state.h:

Line 241:   ///Fragment thread counters
> The comment isn't too helpful. Maybe:
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/4633
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifa88aa6f3371fa42d11ecc122f43c7d83623c300
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: anujphadke <aphadke@cloudera.com>
Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: Yonghyun Hwang
Gerrit-Reviewer: anujphadke <aphadke@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message