impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "anujphadke (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Date Thu, 07 Jul 2016 23:54:54 GMT
anujphadke has posted comments on this change.

Change subject: IMPALA-3766:  Applying LZ4 compression on buffers before spilling
......................................................................


Patch Set 2:

Before -

[localhost:21000] > profile;
Query Runtime Profile:
Query (id=a348adf5718dd795:e38675e8e7ea1a94):
  Summary:
    Session ID: 624487d98d041692:7ce44c89f323fc8b
    Session Type: BEESWAX
    Start Time: 2016-07-07 15:44:59.063728000
    End Time: 2016-07-07 16:50:19.658784000
    Query Type: QUERY
    Query State: FINISHED
    Query Status: OK
    Impala Version: impalad version 2.7.0-cdh5-INTERNAL DEBUG (build d43a6771b7a892df2bab8d62c25ec8154ac9fab4)
    User: anuj
    Connected User: anuj
    Delegated User: 
    Network Address: ::ffff:127.0.0.1:59531
    Default Db: tpch
    Sql Statement: select * from lineitem order by l_quantity
    Coordinator: anuj-OptiPlex-9020:22000
    Query Options (non default): MEM_LIMIT=1073741824
    Plan: 
----------------
Estimated Per-Host Requirements: Memory=864.00MB VCores=1

02:MERGING-EXCHANGE [UNPARTITIONED]
|  order by: l_quantity ASC
|  hosts=1 per-host-mem=unavailable
|  tuple-ids=1 row-size=263B cardinality=6001215
|
01:SORT
|  order by: l_quantity ASC
|  hosts=1 per-host-mem=160.00MB
|  tuple-ids=1 row-size=263B cardinality=6001215
|
00:SCAN HDFS [tpch.lineitem, RANDOM]
   partitions=1/1 files=1 size=718.94MB
   table stats: 6001215 rows total
   column stats: all
   hosts=1 per-host-mem=704.00MB
   tuple-ids=0 row-size=263B cardinality=6001215
----------------
    Estimated Per-Host Mem: 905969664
    Estimated Per-Host VCores: 1
    Request Pool: default-pool
    Admission result: Admitted immediately
    ExecSummary: 
Operator              #Hosts   Avg Time   Max Time  #Rows  Est. #Rows   Peak Mem  Est. Peak
Mem  Detail        
---------------------------------------------------------------------------------------------------------------
02:MERGING-EXCHANGE        1    1s020ms    1s020ms  6.00M       6.00M          0        -1.00
B  UNPARTITIONED 
01:SORT                    1   37s063ms   37s063ms  6.00M       6.00M  816.20 MB      160.00
MB                
00:SCAN HDFS               1  344.851ms  344.851ms  6.00M       6.00M  201.39 MB      704.00
MB  tpch.lineitem 
    Planner Timeline: 105.297ms
       - Analysis finished: 11.531ms (11.531ms)
       - Equivalence classes computed: 31.690ms (20.159ms)
       - Single node plan created: 45.135ms (13.444ms)
       - Runtime filters computed: 46.123ms (988.831us)
       - Distributed plan created: 51.654ms (5.530ms)
       - Planning finished: 105.297ms (53.643ms)
    Query Timeline: 1h5m
       - Start execution: 40.528us (40.528us)
       - Planning finished: 5s618ms (5s618ms)
       - Submit for admission: 5s620ms (2.164ms)
       - Completed admission: 5s620ms (146.718us)
       - Ready to start 1 remote fragments: 5s621ms (714.555us)
       - All 1 remote fragments started: 5s622ms (1.010ms)
       - Rows available: 31s511ms (25s888ms)
       - First row fetched: 31s680ms (169.455ms)
       - Unregister query: 1h5m (1h4m)
  ImpalaServer:
     - ClientFetchWaitTimer: 1h4m
     - RowMaterializationTimer: 34s415ms
  Execution Profile a348adf5718dd795:e38675e8e7ea1a94:(Total: 27s454ms, non-child: 0.000ns,
% non-child: 0.00%)
    Number of filters: 0
    Filter routing table: 
 ID  Src. Node  Tgt. Node(s)  Targets  Target type  Partition filter  Pending (Expected) 
First arrived   Completed
-------------------------------------------------------------------------------------------------------------------

    Fragment start latencies: Count: 1, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th
%-ile: 0, 95th %-ile: 0, 99.9th %-ile: 0
    Per Node Peak Memory Usage: anuj-OptiPlex-9020:22000(978.74 MB) 
     - FiltersReceived: 0 (0)
     - FinalizationTimer: 0.000ns
    Coordinator Fragment F01:(Total: 26s923ms, non-child: 14.099ms, % non-child: 0.05%)
      MemoryUsage(1m4s): 12.26 MB, 20.12 MB, 22.13 MB, 19.77 MB, 21.58 MB, 20.77 MB, 20.43
MB, 21.78 MB, 19.75 MB, 22.03 MB, 20.41 MB, 20.87 MB, 21.66 MB, 19.83 MB, 22.03 MB, 20.11
MB, 21.33 MB, 21.15 MB, 20.08 MB, 22.14 MB, 19.90 MB, 21.69 MB, 20.84 MB, 20.45 MB, 22.01
MB, 19.73 MB, 22.11 MB, 20.45 MB, 20.92 MB, 21.70 MB, 19.84 MB, 22.17 MB, 20.05 MB, 21.29
MB, 21.27 MB, 20.08 MB, 21.99 MB, 19.89 MB, 21.69 MB, 20.88 MB, 20.44 MB, 22.03 MB, 19.77
MB, 22.00 MB, 20.53 MB, 20.71 MB, 21.70 MB, 19.89 MB, 22.15 MB, 20.20 MB, 21.22 MB, 21.39
MB, 19.93 MB, 22.16 MB, 19.93 MB, 21.56 MB, 20.96 MB, 20.34 MB, 22.05 MB, 19.71 MB, 15.62
MB
       - AverageThreadTokens: 0.00 
       - BloomFilterBytes: 0
       - PeakMemoryUsage: 22.55 MB (23643536)
       - PerHostPeakMemUsage: 0
       - PrepareTime: 43.673us
       - RowsProduced: 6.00M (6001215)
       - TotalCpuTime: 1h5m
       - TotalNetworkReceiveTime: 0.000ns
       - TotalNetworkSendTime: 0.000ns
       - TotalStorageWaitTime: 0.000ns
      BlockMgr:
         - BlockWritesOutstanding: 0 (0)
         - BlocksCreated: 219 (219)
         - BlocksRecycled: 32 (32)
         - BufferedPins: 74 (74)
         - BytesWritten: 944.31 MB (990183996)
         - MaxBlockSize: 8.00 MB (8388608)
         - MemoryLimit: 819.20 MB (858993472)
         - PeakMemoryUsage: 816.00 MB (855638016)
         - TotalBufferWaitTime: 0.000ns
         - TotalEncryptionTime: 0.000ns
         - TotalIntegrityCheckTime: 0.000ns
         - TotalReadBlockTime: 485.948ms
      EXCHANGE_NODE (id=2):(Total: 26s909ms, non-child: 1s020ms, % non-child: 3.79%)
        BytesReceived(1m4s): 8.15 MB, 23.92 MB, 36.90 MB, 48.93 MB, 62.21 MB, 74.37 MB, 87.28
MB, 100.01 MB, 112.26 MB, 125.53 MB, 137.54 MB, 150.62 MB, 163.13 MB, 175.50 MB, 188.63 MB,
200.69 MB, 213.86 MB, 226.13 MB, 238.83 MB, 251.80 MB, 263.83 MB, 277.07 MB, 289.19 MB, 302.03
MB, 314.76 MB, 326.91 MB, 340.17 MB, 352.13 MB, 365.15 MB, 377.61 MB, 389.96 MB, 403.11 MB,
415.06 MB, 428.29 MB, 440.57 MB, 453.15 MB, 466.12 MB, 478.23 MB, 491.48 MB, 503.60 MB, 516.43
MB, 529.18 MB, 541.29 MB, 554.54 MB, 566.55 MB, 579.55 MB, 592.15 MB, 604.48 MB, 617.63 MB,
629.62 MB, 642.76 MB, 655.07 MB, 667.58 MB, 680.65 MB, 692.70 MB, 705.94 MB, 718.11 MB, 730.91
MB, 743.72 MB, 755.81 MB, 766.16 MB
         - BytesReceived: 766.85 MB (804096787)
         - ConvertRowBatchTime: 0.000ns
         - DeserializeRowBatchTimer: 3s974ms
         - FirstBatchArrivalWaitTime: 25s888ms
         - MergeGetNext: 997.745ms
         - MergeGetNextBatch: 64.131ms
         - PeakMemoryUsage: 0
         - RowsReturned: 6.00M (6001215)
         - RowsReturnedRate: 223.02 K/sec
         - SendersBlockedTimer: 1h3m
         - SendersBlockedTotalTimer(*): 1h3m
    Averaged Fragment F00:(Total: 1h4m, non-child: 1h3m, % non-child: 98.77%)
      split sizes:  min: 718.94 MB, max: 718.94 MB, avg: 718.94 MB, stddev: 0
      completion times: min:1h4m  max:1h4m  mean: 1h4m  stddev:0.000ns
      execution rates: min:190.64 KB/sec  max:190.64 KB/sec  mean:190.64 KB/sec  stddev:0.00
/sec
      num instances: 1
       - AverageThreadTokens: 1.03 
       - BloomFilterBytes: 0
       - PeakMemoryUsage: 969.24 MB (1016316584)
       - PerHostPeakMemUsage: 978.74 MB (1026286248)
       - PrepareTime: 143.403ms
       - RowsProduced: 6.00M (6001215)
       - TotalCpuTime: 2m42s
       - TotalNetworkReceiveTime: 0.000ns
       - TotalNetworkSendTime: 1h3m
       - TotalStorageWaitTime: 1s124ms
      CodeGen:(Total: 212.028ms, non-child: 212.028ms, % non-child: 100.00%)
         - CodegenTime: 1.478ms
         - CompileTime: 6.577ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 2.12 MB (2224192)
         - NumFunctions: 29 (29)
         - NumInstructions: 529 (529)
         - OptimizationTime: 74.084ms
         - PrepareTime: 126.830ms
      DataStreamSender (dst_id=2):(Total: 9s711ms, non-child: 9s711ms, % non-child: 100.00%)
         - BytesSent: 766.85 MB (804096787)
         - NetworkThroughput(*): 1.04 GB/sec
         - OverallThroughput: 78.97 MB/sec
         - PeakMemoryUsage: 680.00 B (680)
         - RowsReturned: 6.00M (6001215)
         - SerializeBatchTime: 8s992ms
         - TransmitDataRPCTime: 717.249ms
         - UncompressedRowBatchSize: 1.51 GB (1625021116)
      SORT_NODE (id=1):(Total: 37s408ms, non-child: 37s063ms, % non-child: 99.08%)
         - InMemorySortTime: 6s488ms
         - InitialRunsCreated: 2 (2)
         - MergeGetNext: 11s580ms
         - MergeGetNextBatch: 4s490ms
         - PeakMemoryUsage: 816.20 MB (855842816)
         - RowsReturned: 6.00M (6001215)
         - RowsReturnedRate: 160.42 K/sec
         - SortDataSize: 1.49 GB (1600992812)
         - SpilledRuns: 3 (3)
         - TotalMergesPerformed: 1 (1)
      HDFS_SCAN_NODE (id=0):(Total: 344.851ms, non-child: 344.851ms, % non-child: 100.00%)
         - AverageHdfsReadThreadConcurrency: 0.10 
         - AverageScannerThreadConcurrency: 5.76 
         - BytesRead: 718.94 MB (753867192)
         - BytesReadDataNodeCache: 0
         - BytesReadLocal: 718.94 MB (753867192)
         - BytesReadRemoteUnexpected: 0
         - BytesReadShortCircuit: 718.94 MB (753867192)
         - DecompressionTime: 0.000ns
         - MaxCompressedTextFileLength: 0
         - NumDisksAccessed: 2 (2)
         - NumScannerThreadsStarted: 6 (6)
         - PeakMemoryUsage: 201.39 MB (211169280)
         - PerReadThreadRawHdfsThroughput: 192.58 MB/sec
         - RemoteScanRanges: 0 (0)
         - RowsRead: 6.00M (6001215)
         - RowsReturned: 6.00M (6001215)
         - RowsReturnedRate: 17.40 M/sec
         - ScanRangesComplete: 6 (6)
         - ScannerThreadsInvoluntaryContextSwitches: 371 (371)
         - ScannerThreadsTotalWallClockTime: 1m57s
           - DelimiterParseTime: 3s697ms
           - MaterializeTupleTime(*): 6s048ms
           - ScannerThreadsSysTime: 35.892ms
           - ScannerThreadsUserTime: 9s863ms
         - ScannerThreadsVoluntaryContextSwitches: 5.90K (5903)
         - TotalRawHdfsReadTime(*): 3s733ms
         - TotalReadThroughput: 35.07 MB/sec
    Fragment F00:
      Instance a348adf5718dd795:e38675e8e7ea1a96 (host=anuj-OptiPlex-9020:22000):(Total: 1h4m,
non-child: 1h3m, % non-child: 98.77%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94
MB 
        MemoryUsage(1m4s): 270.90 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.58 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.52 MB, 32.52 MB, 32.51 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 28.77 MB
        ThreadUsage(1m4s): 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.03 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 969.24 MB (1016316584)
         - PerHostPeakMemUsage: 978.74 MB (1026286248)
         - PrepareTime: 143.403ms
         - RowsProduced: 6.00M (6001215)
         - TotalCpuTime: 2m42s
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1h3m
         - TotalStorageWaitTime: 1s124ms
        CodeGen:(Total: 212.028ms, non-child: 212.028ms, % non-child: 100.00%)
           - CodegenTime: 1.478ms
           - CompileTime: 6.577ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 2.12 MB (2224192)
           - NumFunctions: 29 (29)
           - NumInstructions: 529 (529)
           - OptimizationTime: 74.084ms
           - PrepareTime: 126.830ms
        DataStreamSender (dst_id=2):(Total: 9s711ms, non-child: 9s711ms, % non-child: 100.00%)
           - BytesSent: 766.85 MB (804096787)
           - NetworkThroughput(*): 1.04 GB/sec
           - OverallThroughput: 78.97 MB/sec
           - PeakMemoryUsage: 680.00 B (680)
           - RowsReturned: 6.00M (6001215)
           - SerializeBatchTime: 8s992ms
           - TransmitDataRPCTime: 717.249ms
           - UncompressedRowBatchSize: 1.51 GB (1625021116)
        SORT_NODE (id=1):(Total: 37s408ms, non-child: 37s063ms, % non-child: 99.08%)
          ExecOption: Codegen Enabled
           - InMemorySortTime: 6s488ms
           - InitialRunsCreated: 2 (2)
           - MergeGetNext: 11s580ms
           - MergeGetNextBatch: 4s490ms
           - PeakMemoryUsage: 816.20 MB (855842816)
           - RowsReturned: 6.00M (6001215)
           - RowsReturnedRate: 160.42 K/sec
           - SortDataSize: 1.49 GB (1600992812)
           - SpilledRuns: 3 (3)
           - TotalMergesPerformed: 1 (1)
        HDFS_SCAN_NODE (id=0):(Total: 344.851ms, non-child: 344.851ms, % non-child: 100.00%)
          ExecOption: Expr Evaluation Codegen Disabled, Codegen enabled: 0 out of 6
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94
MB 
          Hdfs Read Thread Concurrency Bucket: 0:92.68% 1:4.878% 2:2.439% 3:0% 4:0% 
          File Formats: TEXT/NONE:6 
          BytesRead(500.000ms): 0, 192.00 MB, 192.00 MB, 192.00 MB, 200.00 MB, 240.00 MB,
248.00 MB, 288.00 MB, 296.00 MB, 336.00 MB, 344.00 MB, 384.00 MB, 392.00 MB, 432.00 MB, 448.00
MB, 478.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB,
486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 494.94 MB, 542.94 MB, 550.94
MB, 590.94 MB, 630.94 MB, 638.94 MB, 670.94 MB, 702.94 MB, 710.94 MB, 718.94 MB, 718.94 MB,
718.94 MB
           - AverageHdfsReadThreadConcurrency: 0.10 
           - AverageScannerThreadConcurrency: 5.76 
           - BytesRead: 718.94 MB (753867192)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 718.94 MB (753867192)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 718.94 MB (753867192)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumDisksAccessed: 2 (2)
           - NumScannerThreadsStarted: 6 (6)
           - PeakMemoryUsage: 201.39 MB (211169280)
           - PerReadThreadRawHdfsThroughput: 192.58 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowsRead: 6.00M (6001215)
           - RowsReturned: 6.00M (6001215)
           - RowsReturnedRate: 17.40 M/sec
           - ScanRangesComplete: 6 (6)
           - ScannerThreadsInvoluntaryContextSwitches: 371 (371)
           - ScannerThreadsTotalWallClockTime: 1m57s
             - DelimiterParseTime: 3s697ms
             - MaterializeTupleTime(*): 6s048ms
             - ScannerThreadsSysTime: 35.892ms
             - ScannerThreadsUserTime: 9s863ms
           - ScannerThreadsVoluntaryContextSwitches: 5.90K (5903)
           - TotalRawHdfsReadTime(*): 3s733ms
           - TotalReadThroughput: 35.07 MB/sec

After -
[localhost:21000] > profile;
Query Runtime Profile:
Query (id=394ee7185b41be23:949a047c1e09ec82):
  Summary:
    Session ID: 84ece9f0c0c9e06:f6c333f0e5737cb5
    Session Type: BEESWAX
    Start Time: 2016-07-07 12:43:01.225510000
    End Time: 2016-07-07 13:48:28.978336000
    Query Type: QUERY
    Query State: FINISHED
    Query Status: OK
    Impala Version: impalad version 2.7.0-cdh5-INTERNAL DEBUG (build 34e49668e40386c7e50f87f01e4c5ad926983717)
    User: anuj
    Connected User: anuj
    Delegated User: 
    Network Address: ::ffff:127.0.0.1:57170
    Default Db: tpch
    Sql Statement: select * from lineitem order by l_quantity
    Coordinator: anuj-OptiPlex-9020:22000
    Query Options (non default): MEM_LIMIT=1073741824
    Plan: 
----------------
Estimated Per-Host Requirements: Memory=864.00MB VCores=1

02:MERGING-EXCHANGE [UNPARTITIONED]
|  order by: l_quantity ASC
|  hosts=1 per-host-mem=unavailable
|  tuple-ids=1 row-size=263B cardinality=6001215
|
01:SORT
|  order by: l_quantity ASC
|  hosts=1 per-host-mem=160.00MB
|  tuple-ids=1 row-size=263B cardinality=6001215
|
00:SCAN HDFS [tpch.lineitem, RANDOM]
   partitions=1/1 files=1 size=718.94MB
   table stats: 6001215 rows total
   column stats: all
   hosts=1 per-host-mem=704.00MB
   tuple-ids=0 row-size=263B cardinality=6001215
----------------
    Estimated Per-Host Mem: 905969664
    Estimated Per-Host VCores: 1
    Request Pool: default-pool
    Admission result: Admitted immediately
    ExecSummary: 
Operator              #Hosts   Avg Time   Max Time  #Rows  Est. #Rows   Peak Mem  Est. Peak
Mem  Detail        
---------------------------------------------------------------------------------------------------------------
02:MERGING-EXCHANGE        1    1s021ms    1s021ms  6.00M       6.00M          0        -1.00
B  UNPARTITIONED 
01:SORT                    1   41s010ms   41s010ms  6.00M       6.00M  816.20 MB      160.00
MB                
00:SCAN HDFS               1  412.149ms  412.149ms  6.00M       6.00M  201.76 MB      704.00
MB  tpch.lineitem 
    Planner Timeline: 116.792ms
       - Analysis finished: 11.939ms (11.939ms)
       - Equivalence classes computed: 32.413ms (20.473ms)
       - Single node plan created: 46.386ms (13.973ms)
       - Runtime filters computed: 47.696ms (1.310ms)
       - Distributed plan created: 54.638ms (6.942ms)
       - Planning finished: 116.792ms (62.153ms)
    Query Timeline: 1h5m
       - Start execution: 43.421us (43.421us)
       - Planning finished: 4s826ms (4s826ms)
       - Submit for admission: 4s826ms (261.600us)
       - Completed admission: 4s826ms (149.430us)
       - Ready to start 1 remote fragments: 4s828ms (1.831ms)
       - All 1 remote fragments started: 4s829ms (1.125ms)
       - Rows available: 33s993ms (29s163ms)
       - First row fetched: 34s394ms (401.447ms)
       - Unregister query: 1h5m (1h4m)
  ImpalaServer:
     - ClientFetchWaitTimer: 1h4m
     - RowMaterializationTimer: 34s291ms
  Execution Profile 394ee7185b41be23:949a047c1e09ec82:(Total: 30s738ms, non-child: 0.000ns,
% non-child: 0.00%)
    Number of filters: 0
    Filter routing table: 
 ID  Src. Node  Tgt. Node(s)  Targets  Target type  Partition filter  Pending (Expected) 
First arrived   Completed
-------------------------------------------------------------------------------------------------------------------

    Fragment start latencies: Count: 1, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms,
90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms
    Per Node Peak Memory Usage: anuj-OptiPlex-9020:22000(970.91 MB) 
     - FiltersReceived: 0 (0)
     - FinalizationTimer: 0.000ns
    Coordinator Fragment F01:(Total: 30s200ms, non-child: 15.938ms, % non-child: 0.05%)
      MemoryUsage(1m4s): 10.99 MB, 19.34 MB, 22.10 MB, 19.26 MB, 21.25 MB, 20.63 MB, 19.67
MB, 21.83 MB, 19.01 MB, 21.66 MB, 20.05 MB, 20.10 MB, 21.68 MB, 19.01 MB, 22.03 MB, 19.70
MB, 20.70 MB, 21.08 MB, 19.28 MB, 22.10 MB, 19.27 MB, 21.21 MB, 20.80 MB, 19.59 MB, 21.98
MB, 19.02 MB, 21.66 MB, 20.18 MB, 20.09 MB, 21.72 MB, 18.95 MB, 21.94 MB, 19.71 MB, 20.50
MB, 21.30 MB, 19.15 MB, 22.05 MB, 19.39 MB, 21.13 MB, 20.86 MB, 19.51 MB, 21.99 MB, 19.07
MB, 21.40 MB, 20.29 MB, 19.82 MB, 21.82 MB, 19.00 MB, 21.91 MB, 19.93 MB, 20.42 MB, 21.44
MB, 19.04 MB, 22.03 MB, 19.51 MB, 20.87 MB, 21.00 MB, 19.45 MB, 22.05 MB, 19.22 MB, 17.13
MB
       - AverageThreadTokens: 0.00 
       - BloomFilterBytes: 0
       - PeakMemoryUsage: 22.55 MB (23644896)
       - PerHostPeakMemUsage: 0
       - PrepareTime: 224.969us
       - RowsProduced: 6.00M (6001215)
       - TotalCpuTime: 1h5m
       - TotalNetworkReceiveTime: 0.000ns
       - TotalNetworkSendTime: 0.000ns
       - TotalStorageWaitTime: 0.000ns
      BlockMgr:
         - BlockWritesOutstanding: 0 (0)
         - BlocksCreated: 219 (219)
         - BlocksRecycled: 32 (32)
         - BufferedPins: 74 (74)
         - BytesWritten: 944.32 MB (990186348)
         - CompressedBytesWritten: 437.30 MB (458537208)
         - MaxBlockSize: 8.00 MB (8388608)
         - MemoryLimit: 819.20 MB (858993472)
         - PeakMemoryUsage: 816.00 MB (855638016)
         - TotalBufferWaitTime: 0.000ns
         - TotalEncryptionTime: 0.000ns
         - TotalIntegrityCheckTime: 0.000ns
         - TotalReadBlockTime: 226.005ms
      EXCHANGE_NODE (id=2):(Total: 30s184ms, non-child: 1s021ms, % non-child: 3.39%)
        BytesReceived(1m4s): 7.20 MB, 23.35 MB, 36.77 MB, 48.83 MB, 62.53 MB, 74.82 MB, 87.88
MB, 101.03 MB, 113.27 MB, 126.98 MB, 139.11 MB, 152.48 MB, 165.37 MB, 177.76 MB, 191.43 MB,
203.47 MB, 217.02 MB, 229.56 MB, 242.35 MB, 255.81 MB, 267.86 MB, 281.57 MB, 293.87 MB, 306.84
MB, 320.09 MB, 332.25 MB, 346.00 MB, 358.15 MB, 371.47 MB, 384.39 MB, 396.76 MB, 410.51 MB,
422.55 MB, 436.12 MB, 448.79 MB, 461.47 MB, 475.04 MB, 487.12 MB, 500.78 MB, 513.14 MB, 526.06
MB, 539.35 MB, 551.49 MB, 565.21 MB, 577.44 MB, 590.70 MB, 603.79 MB, 616.08 MB, 629.79 MB,
641.85 MB, 655.32 MB, 668.05 MB, 680.60 MB, 694.24 MB, 706.31 MB, 719.94 MB, 732.40 MB, 745.26
MB, 758.61 MB, 770.73 MB, 782.37 MB
         - BytesReceived: 783.74 MB (821814808)
         - ConvertRowBatchTime: 0.000ns
         - DeserializeRowBatchTimer: 3s948ms
         - FirstBatchArrivalWaitTime: 29s163ms
         - MergeGetNext: 997.881ms
         - MergeGetNextBatch: 65.561ms
         - PeakMemoryUsage: 0
         - RowsReturned: 6.00M (6001215)
         - RowsReturnedRate: 198.81 K/sec
         - SendersBlockedTimer: 1h3m
         - SendersBlockedTotalTimer(*): 1h3m
    Averaged Fragment F00:(Total: 1h4m, non-child: 1h3m, % non-child: 98.68%)
      split sizes:  min: 718.94 MB, max: 718.94 MB, avg: 718.94 MB, stddev: 0
      completion times: min:1h4m  max:1h4m  mean: 1h4m  stddev:0.000ns
      execution rates: min:190.29 KB/sec  max:190.29 KB/sec  mean:190.29 KB/sec  stddev:0.00
/sec
      num instances: 1
       - AverageThreadTokens: 1.03 
       - BloomFilterBytes: 0
       - PeakMemoryUsage: 969.61 MB (1016709800)
       - PerHostPeakMemUsage: 970.91 MB (1018077864)
       - PrepareTime: 167.610ms
       - RowsProduced: 6.00M (6001215)
       - TotalCpuTime: 3m1s
       - TotalNetworkReceiveTime: 0.000ns
       - TotalNetworkSendTime: 1h3m
       - TotalStorageWaitTime: 1s303ms
      CodeGen:(Total: 202.085ms, non-child: 202.085ms, % non-child: 100.00%)
         - CodegenTime: 274.215us
         - CompileTime: 3.591ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 2.12 MB (2224196)
         - NumFunctions: 29 (29)
         - NumInstructions: 529 (529)
         - OptimizationTime: 54.022ms
         - PrepareTime: 143.628ms
      DataStreamSender (dst_id=2):(Total: 9s619ms, non-child: 9s619ms, % non-child: 100.00%)
         - BytesSent: 783.74 MB (821814808)
         - NetworkThroughput(*): 1.03 GB/sec
         - OverallThroughput: 81.48 MB/sec
         - PeakMemoryUsage: 680.00 B (680)
         - RowsReturned: 6.00M (6001215)
         - SerializeBatchTime: 8s876ms
         - TransmitDataRPCTime: 741.034ms
         - UncompressedRowBatchSize: 1.51 GB (1625021116)
      SORT_NODE (id=1):(Total: 41s422ms, non-child: 41s010ms, % non-child: 99.01%)
         - InMemorySortTime: 6s580ms
         - InitialRunsCreated: 2 (2)
         - MergeGetNext: 12s295ms
         - MergeGetNextBatch: 5s318ms
         - PeakMemoryUsage: 816.20 MB (855842816)
         - RowsReturned: 6.00M (6001215)
         - RowsReturnedRate: 144.88 K/sec
         - SortDataSize: 1.49 GB (1600992812)
         - SpilledRuns: 3 (3)
         - TotalMergesPerformed: 1 (1)
      HDFS_SCAN_NODE (id=0):(Total: 412.149ms, non-child: 412.149ms, % non-child: 100.00%)
         - AverageHdfsReadThreadConcurrency: 0.24 
         - AverageScannerThreadConcurrency: 5.74 
         - BytesRead: 718.94 MB (753867192)
         - BytesReadDataNodeCache: 0
         - BytesReadLocal: 718.94 MB (753867192)
         - BytesReadRemoteUnexpected: 0
         - BytesReadShortCircuit: 718.94 MB (753867192)
         - DecompressionTime: 0.000ns
         - MaxCompressedTextFileLength: 0
         - NumDisksAccessed: 2 (2)
         - NumScannerThreadsStarted: 6 (6)
         - PeakMemoryUsage: 201.76 MB (211562496)
         - PerReadThreadRawHdfsThroughput: 195.65 MB/sec
         - RemoteScanRanges: 0 (0)
         - RowsRead: 6.00M (6001215)
         - RowsReturned: 6.00M (6001215)
         - RowsReturnedRate: 14.56 M/sec
         - ScanRangesComplete: 6 (6)
         - ScannerThreadsInvoluntaryContextSwitches: 320 (320)
         - ScannerThreadsTotalWallClockTime: 2m12s
           - DelimiterParseTime: 3s826ms
           - MaterializeTupleTime(*): 6s619ms
           - ScannerThreadsSysTime: 20.468ms
           - ScannerThreadsUserTime: 10s538ms
         - ScannerThreadsVoluntaryContextSwitches: 6.01K (6006)
         - TotalRawHdfsReadTime(*): 3s674ms
         - TotalReadThroughput: 30.59 MB/sec
    Fragment F00:
      Instance 394ee7185b41be23:949a047c1e09ec84 (host=anuj-OptiPlex-9020:22000):(Total: 1h4m,
non-child: 1h3m, % non-child: 98.68%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94
MB 
        MemoryUsage(1m4s): 297.48 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52
MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 30.89 MB
        ThreadUsage(1m4s): 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.03 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 969.61 MB (1016709800)
         - PerHostPeakMemUsage: 970.91 MB (1018077864)
         - PrepareTime: 167.610ms
         - RowsProduced: 6.00M (6001215)
         - TotalCpuTime: 3m1s
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1h3m
         - TotalStorageWaitTime: 1s303ms
        CodeGen:(Total: 202.085ms, non-child: 202.085ms, % non-child: 100.00%)
           - CodegenTime: 274.215us
           - CompileTime: 3.591ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 2.12 MB (2224196)
           - NumFunctions: 29 (29)
           - NumInstructions: 529 (529)
           - OptimizationTime: 54.022ms
           - PrepareTime: 143.628ms
        DataStreamSender (dst_id=2):(Total: 9s619ms, non-child: 9s619ms, % non-child: 100.00%)
           - BytesSent: 783.74 MB (821814808)
           - NetworkThroughput(*): 1.03 GB/sec
           - OverallThroughput: 81.48 MB/sec
           - PeakMemoryUsage: 680.00 B (680)
           - RowsReturned: 6.00M (6001215)
           - SerializeBatchTime: 8s876ms
           - TransmitDataRPCTime: 741.034ms
           - UncompressedRowBatchSize: 1.51 GB (1625021116)
        SORT_NODE (id=1):(Total: 41s422ms, non-child: 41s010ms, % non-child: 99.01%)
          ExecOption: Codegen Enabled
           - InMemorySortTime: 6s580ms
           - InitialRunsCreated: 2 (2)
           - MergeGetNext: 12s295ms
           - MergeGetNextBatch: 5s318ms
           - PeakMemoryUsage: 816.20 MB (855842816)
           - RowsReturned: 6.00M (6001215)
           - RowsReturnedRate: 144.88 K/sec
           - SortDataSize: 1.49 GB (1600992812)
           - SpilledRuns: 3 (3)
           - TotalMergesPerformed: 1 (1)
        HDFS_SCAN_NODE (id=0):(Total: 412.149ms, non-child: 412.149ms, % non-child: 100.00%)
          ExecOption: Expr Evaluation Codegen Disabled, Codegen enabled: 0 out of 6
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94
MB 
          Hdfs Read Thread Concurrency Bucket: 0:91.3% 1:4.348% 2:0% 3:2.174% 4:2.174% 
          File Formats: TEXT/NONE:6 
          BytesRead(500.000ms): 0, 96.00 MB, 192.00 MB, 192.00 MB, 192.00 MB, 200.00 MB, 240.00
MB, 240.00 MB, 288.00 MB, 288.00 MB, 304.00 MB, 336.00 MB, 336.00 MB, 384.00 MB, 384.00 MB,
432.00 MB, 432.00 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94
MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB,
478.94 MB, 486.94 MB, 534.94 MB, 550.94 MB, 582.94 MB, 622.94 MB, 622.94 MB, 662.94 MB, 662.94
MB, 702.94 MB, 702.94 MB, 718.94 MB, 718.94 MB, 718.94 MB, 718.94 MB
           - AverageHdfsReadThreadConcurrency: 0.24 
           - AverageScannerThreadConcurrency: 5.74 
           - BytesRead: 718.94 MB (753867192)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 718.94 MB (753867192)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 718.94 MB (753867192)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumDisksAccessed: 2 (2)
           - NumScannerThreadsStarted: 6 (6)
           - PeakMemoryUsage: 201.76 MB (211562496)
           - PerReadThreadRawHdfsThroughput: 195.65 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowsRead: 6.00M (6001215)
           - RowsReturned: 6.00M (6001215)
           - RowsReturnedRate: 14.56 M/sec
           - ScanRangesComplete: 6 (6)
           - ScannerThreadsInvoluntaryContextSwitches: 320 (320)
           - ScannerThreadsTotalWallClockTime: 2m12s
             - DelimiterParseTime: 3s826ms
             - MaterializeTupleTime(*): 6s619ms
             - ScannerThreadsSysTime: 20.468ms
             - ScannerThreadsUserTime: 10s538ms
           - ScannerThreadsVoluntaryContextSwitches: 6.01K (6006)
           - TotalRawHdfsReadTime(*): 3s674ms
           - TotalReadThroughput: 30.59 MB/sec

[localhost:21000] >

-- 
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <aphadke@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mmokhtar@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: anujphadke <aphadke@cloudera.com>
Gerrit-HasComments: No

Mime
View raw message