impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fawze Abujaber <fawz...@gmail.com>
Subject Re: Estimate peak memory VS used peak memory
Date Thu, 22 Feb 2018 20:13:55 GMT
Hi Mostafa,

It's not a specific query, almost all the query has such differene between
the 2 values.

I can see even queries showing the estimate per node is 8.2 Tib


   - User: psanalytics
   - Database: default
   - Query Type: QUERY
   - Coordinator: slpr-dhc014.lpdomain.com
   <http://slvr-cmc101:7180/cmf/hardware/hosts/22/status>
   - Duration: 6.48s
   - Rows Produced: 708
   - Estimated per Node Peak Memory: 8.2 TiB
   - Per Node Peak Memory Usage: 1.1 GiB
   - Pool: root.impanalytics
   - Threads: CPU Time: 20.1m



How you can explain this behavior, and for sure i don't have 8.2 Tib memory
per node to give neither you.

Can you please explain me how i should treat Estimated per Node Peak Memory
and if it used by impala for the resource pool and admission control and
what is the relation of this value to the default memory per node that i
setup for each resource pool?

Below is part of one of the queries profile which the estimate per node was
~ @GB and the used was 200MB per node.


 Instance 744de1b6228736fa:b54bfaa70000000f
(host=slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child: 1s292ms,
% non-child: 88.82%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 6:1/1.20
MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64 KB
5:2/289.57 KB 3:2/1012.83 KB
        MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 7.17 MB (7521751)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 106.53 MB (111709581)
         - RowsProduced: 32.83K (32826)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s297ms
         - TotalStorageWaitTime: 234.356ms
         - TotalThreadsInvoluntaryContextSwitches: 66 (66)
         - TotalThreadsTotalWallClockTime: 1s715ms
           - TotalThreadsSysTime: 5.998ms
           - TotalThreadsUserTime: 124.975ms
         - TotalThreadsVoluntaryContextSwitches: 303 (303)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s394ms
             - ExecTreeExecTime: 67.115ms
           - OpenTime: 32.795ms
             - ExecTreeOpenTime: 73.243us
           - PrepareTime: 27.602ms
             - ExecTreePrepareTime: 243.141us
        DataStreamSender (dst_id=11):(Total: 38.747ms, non-child: 38.747ms,
% non-child: 100.00%)
           - BytesSent: 39.71 MB (41643000)
           - NetworkThroughput(*): 1.97 GB/sec
           - OverallThroughput: 1.00 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 32.83K (32826)
           - SerializeBatchTime: 16.860ms
           - TransmitDataRPCTime: 19.698ms
           - UncompressedRowBatchSize: 77.58 MB (81350840)
        CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child: 100.00%)
           - CodegenTime: 1.299ms
           - CompileTime: 10.672ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 21.023ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 24.116ms
        SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, %
non-child: 17.85%)
           - PeakMemoryUsage: 627.94 KB (643015)
           - RowsReturned: 32.77K (32768)
           - RowsReturnedRate: 486.81 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child:
25.197ms, % non-child: 74.11%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 32.83K (32826)
             - ProbeTime: 0.000ns
             - RowsReturned: 16.80M (16795311)
             - RowsReturnedRate: 493.99 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 32.83K (32826)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 113.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64
KB 5:2/289.57 KB 3:2/1012.83 KB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11 out of 11
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
          File Formats: PARQUET/SNAPPY:66
          BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB
           - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ; Max:
29.275ms ; Number of samples: 11)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 2.26 MB (2373738)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 2.26 MB (2373738)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 2.26 MB (2373738)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 77 (77)
           - DecompressionTime: 1.543ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 8 (8)
           - NumRowGroups: 11 (11)
           - NumScannerThreadsStarted: 11 (11)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 6.51 MB (6821203)
           - PerReadThreadRawHdfsThroughput: 764.51 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 18.825ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 73.46K (73464)
           - RowsReturned: 32.83K (32826)
           - RowsReturnedRate: 1.54 M/sec
           - ScanRangesComplete: 11 (11)
           - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
           - ScannerThreadsTotalWallClockTime: 287.702ms
             - MaterializeTupleTime(*): 11.921ms
             - ScannerThreadsSysTime: 999.000us
             - ScannerThreadsUserTime: 26.990ms
           - ScannerThreadsVoluntaryContextSwitches: 232 (232)
           - TotalRawHdfsReadTime(*): 2.961ms
           - TotalReadThroughput: 1.51 MB/sec
      Instance 744de1b6228736fa:b54bfaa70000000d
(host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child: 1s254ms,
% non-child: 86.28%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/1.32
MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
        MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 5.86 MB (6145964)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 106.66 MB (111842766)
         - RowsProduced: 30.34K (30340)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s256ms
         - TotalStorageWaitTime: 106.530ms
         - TotalThreadsInvoluntaryContextSwitches: 177 (177)
         - TotalThreadsTotalWallClockTime: 1s557ms
           - TotalThreadsSysTime: 6.998ms
           - TotalThreadsUserTime: 143.976ms
         - TotalThreadsVoluntaryContextSwitches: 136 (136)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s390ms
             - ExecTreeExecTime: 91.500ms
           - OpenTime: 34.085ms
             - ExecTreeOpenTime: 81.436us
           - PrepareTime: 28.524ms
             - ExecTreePrepareTime: 217.949us
        DataStreamSender (dst_id=11):(Total: 48.674ms, non-child: 48.674ms,
% non-child: 100.00%)
           - BytesSent: 36.61 MB (38388760)
           - NetworkThroughput(*): 1.51 GB/sec
           - OverallThroughput: 752.15 MB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 30.34K (30340)
           - SerializeBatchTime: 23.132ms
           - TransmitDataRPCTime: 23.609ms
           - UncompressedRowBatchSize: 71.70 MB (75187520)
        CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child: 100.00%)
           - CodegenTime: 1.382ms
           - CompileTime: 10.435ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 22.325ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 25.459ms
        SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, %
non-child: 22.90%)
           - PeakMemoryUsage: 699.55 KB (716335)
           - RowsReturned: 29.70K (29696)
           - RowsReturnedRate: 323.80 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child:
40.263ms, % non-child: 74.95%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 30.34K (30340)
             - ProbeTime: 0.000ns
             - RowsReturned: 15.43M (15426890)
             - RowsReturnedRate: 287.16 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 30.34K (30340)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 74.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out of 5
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
          File Formats: PARQUET/SNAPPY:30
          BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB
           - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ; Max:
28.687ms ; Number of samples: 5)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.56 MB (1630546)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.56 MB (1630546)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.56 MB (1630546)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 35 (35)
           - DecompressionTime: 1.267ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 4 (4)
           - NumRowGroups: 5 (5)
           - NumScannerThreadsStarted: 5 (5)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.59 MB (5859290)
           - PerReadThreadRawHdfsThroughput: 920.32 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 15.335ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 64.69K (64692)
           - RowsReturned: 30.34K (30340)
           - RowsReturnedRate: 1.79 M/sec
           - ScanRangesComplete: 5 (5)
           - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
           - ScannerThreadsTotalWallClockTime: 132.548ms
             - MaterializeTupleTime(*): 10.585ms
             - ScannerThreadsSysTime: 999.000us
             - ScannerThreadsUserTime: 18.995ms
           - ScannerThreadsVoluntaryContextSwitches: 79 (79)
           - TotalRawHdfsReadTime(*): 1.689ms
           - TotalReadThroughput: 1.04 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000002
(host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child: 1s255ms,
% non-child: 86.41%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
        MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 6.17 MB (6473729)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 107.32 MB (112537640)
         - RowsProduced: 29.97K (29969)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s260ms
         - TotalStorageWaitTime: 130.815ms
         - TotalThreadsInvoluntaryContextSwitches: 15 (15)
         - TotalThreadsTotalWallClockTime: 1s586ms
           - TotalThreadsSysTime: 11.995ms
           - TotalThreadsUserTime: 146.975ms
         - TotalThreadsVoluntaryContextSwitches: 197 (197)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s354ms
             - ExecTreeExecTime: 67.791ms
           - OpenTime: 52.012ms
             - ExecTreeOpenTime: 101.232us
           - PrepareTime: 45.946ms
             - ExecTreePrepareTime: 309.644us
        DataStreamSender (dst_id=11):(Total: 38.995ms, non-child: 38.995ms,
% non-child: 100.00%)
           - BytesSent: 35.95 MB (37697260)
           - NetworkThroughput(*): 2.14 GB/sec
           - OverallThroughput: 921.92 MB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 29.97K (29969)
           - SerializeBatchTime: 18.055ms
           - TransmitDataRPCTime: 16.428ms
           - UncompressedRowBatchSize: 70.83 MB (74269800)
        CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child: 100.00%)
           - CodegenTime: 2.224ms
           - CompileTime: 15.517ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 35.099ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 38.847ms
        SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, %
non-child: 18.53%)
           - PeakMemoryUsage: 636.17 KB (651439)
           - RowsReturned: 29.70K (29696)
           - RowsReturnedRate: 435.89 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child:
27.325ms, % non-child: 77.31%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 29.97K (29969)
             - ProbeTime: 0.000ns
             - RowsReturned: 15.26M (15256601)
             - RowsReturnedRate: 431.62 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 29.97K (29969)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 124.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
          File Formats: PARQUET/SNAPPY:42
          BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB
           - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ; Max:
10.464ms ; Number of samples: 7)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.75 MB (1833560)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.75 MB (1833560)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.75 MB (1833560)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 49 (49)
           - DecompressionTime: 1.758ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 5 (5)
           - NumRowGroups: 7 (7)
           - NumScannerThreadsStarted: 7 (7)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.82 MB (6098991)
           - PerReadThreadRawHdfsThroughput: 621.09 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 17.938ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 65.17K (65166)
           - RowsReturned: 29.97K (29969)
           - RowsReturnedRate: 1.49 M/sec
           - ScanRangesComplete: 7 (7)
           - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
           - ScannerThreadsTotalWallClockTime: 179.698ms
             - MaterializeTupleTime(*): 13.922ms
             - ScannerThreadsSysTime: 2.997ms
             - ScannerThreadsUserTime: 29.993ms
           - ScannerThreadsVoluntaryContextSwitches: 130 (130)
           - TotalRawHdfsReadTime(*): 2.815ms
           - TotalReadThroughput: 1.17 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000008
(host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child: 1s305ms,
% non-child: 90.01%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 7:1/1.08
MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
        MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 6.69 MB (7017157)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 106.39 MB (111556990)
         - RowsProduced: 42.68K (42681)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s315ms
         - TotalStorageWaitTime: 201.411ms
         - TotalThreadsInvoluntaryContextSwitches: 43 (43)
         - TotalThreadsTotalWallClockTime: 1s658ms
           - TotalThreadsSysTime: 6.997ms
           - TotalThreadsUserTime: 108.980ms
         - TotalThreadsVoluntaryContextSwitches: 179 (179)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s411ms
             - ExecTreeExecTime: 68.017ms
           - OpenTime: 19.935ms
             - ExecTreeOpenTime: 48.883us
           - PrepareTime: 19.160ms
             - ExecTreePrepareTime: 134.576us
        DataStreamSender (dst_id=11):(Total: 40.671ms, non-child: 40.671ms,
% non-child: 100.00%)
           - BytesSent: 51.36 MB (53856680)
           - NetworkThroughput(*): 2.44 GB/sec
           - OverallThroughput: 1.23 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 42.68K (42681)
           - SerializeBatchTime: 18.299ms
           - TransmitDataRPCTime: 20.591ms
           - UncompressedRowBatchSize: 100.88 MB (105782860)
        CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child: 100.00%)
           - CodegenTime: 828.865us
           - CompileTime: 6.170ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 12.988ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 16.608ms
        SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, %
non-child: 18.27%)
           - PeakMemoryUsage: 689.95 KB (706507)
           - RowsReturned: 41.98K (41984)
           - RowsReturnedRate: 616.99 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child:
29.422ms, % non-child: 73.86%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 42.68K (42681)
             - ProbeTime: 0.000ns
             - RowsReturned: 21.76M (21760053)
             - RowsReturnedRate: 546.22 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 42.68K (42681)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 96.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
          File Formats: PARQUET/SNAPPY:36
          BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB
           - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ; Max:
42.732ms ; Number of samples: 6)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 2.08 MB (2186159)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 2.08 MB (2186159)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 2.08 MB (2186159)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 42 (42)
           - DecompressionTime: 1.480ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 4 (4)
           - NumRowGroups: 6 (6)
           - NumScannerThreadsStarted: 6 (6)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 6.42 MB (6730467)
           - PerReadThreadRawHdfsThroughput: 77.02 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 13.979ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 89.91K (89910)
           - RowsReturned: 42.68K (42681)
           - RowsReturnedRate: 2.71 M/sec
           - ScanRangesComplete: 6 (6)
           - ScannerThreadsInvoluntaryContextSwitches: 3 (3)
           - ScannerThreadsTotalWallClockTime: 227.825ms
             - MaterializeTupleTime(*): 9.931ms
             - ScannerThreadsSysTime: 1.998ms
             - ScannerThreadsUserTime: 18.994ms
           - ScannerThreadsVoluntaryContextSwitches: 85 (85)
           - TotalRawHdfsReadTime(*): 27.069ms
           - TotalReadThroughput: 1.39 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000005
(host=slpr-dhc001.lpdomain.com:22000):(Total: 1s447ms, non-child: 1s301ms,
% non-child: 89.88%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 6:1/1.20
MB 1:2/2.50 MB 3:1/1.32 MB
        MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 1.25 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 6.32 MB (6623566)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 106.35 MB (111518199)
         - RowsProduced: 33.03K (33027)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s311ms
         - TotalStorageWaitTime: 76.832ms
         - TotalThreadsInvoluntaryContextSwitches: 7 (7)
         - TotalThreadsTotalWallClockTime: 1s518ms
           - TotalThreadsSysTime: 8.997ms
           - TotalThreadsUserTime: 102.983ms
         - TotalThreadsVoluntaryContextSwitches: 128 (128)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s389ms
             - ExecTreeExecTime: 55.767ms
           - OpenTime: 30.577ms
             - ExecTreeOpenTime: 71.274us
           - PrepareTime: 27.610ms
             - ExecTreePrepareTime: 194.662us
        DataStreamSender (dst_id=11):(Total: 36.270ms, non-child: 36.270ms,
% non-child: 100.00%)
           - BytesSent: 39.69 MB (41621640)
           - NetworkThroughput(*): 2.07 GB/sec
           - OverallThroughput: 1.07 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 33.03K (33027)
           - SerializeBatchTime: 15.319ms
           - TransmitDataRPCTime: 18.709ms
           - UncompressedRowBatchSize: 78.06 MB (81851440)
        CodeGen:(Total: 54.301ms, non-child: 54.301ms, % non-child: 100.00%)
           - CodegenTime: 1.269ms
           - CompileTime: 9.193ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 20.433ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 24.055ms
        SUBPLAN_NODE (id=6):(Total: 55.933ms, non-child: 10.883ms, %
non-child: 19.46%)
           - PeakMemoryUsage: 702.06 KB (718909)
           - RowsReturned: 32.77K (32768)
           - RowsReturnedRate: 585.83 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 28.594ms, non-child:
22.150ms, % non-child: 77.46%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 33.03K (33027)
             - ProbeTime: 0.000ns
             - RowsReturned: 16.83M (16827270)
             - RowsReturnedRate: 588.47 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 6.444ms, non-child: 6.444ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 33.03K (33027)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 155.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 16.455ms, non-child: 16.455ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
6:1/1.20 MB 1:2/2.50 MB 3:1/1.32 MB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4 out of 4
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
          File Formats: PARQUET/SNAPPY:24
          BytesRead(500.000ms): 1.54 MB, 1.54 MB, 1.54 MB
           - FooterProcessingTime: (Avg: 6.502ms ; Min: 5.361ms ; Max:
8.670ms ; Number of samples: 4)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.54 MB (1616732)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.54 MB (1616732)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.54 MB (1616732)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 28 (28)
           - DecompressionTime: 1.336ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 3 (3)
           - NumRowGroups: 4 (4)
           - NumScannerThreadsStarted: 4 (4)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 6.04 MB (6336892)
           - PerReadThreadRawHdfsThroughput: 102.07 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 15.237ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 69.25K (69254)
           - RowsReturned: 33.03K (33027)
           - RowsReturnedRate: 2.01 M/sec
           - ScanRangesComplete: 4 (4)
           - ScannerThreadsInvoluntaryContextSwitches: 2 (2)
           - ScannerThreadsTotalWallClockTime: 98.879ms
             - MaterializeTupleTime(*): 9.955ms
             - ScannerThreadsSysTime: 1.998ms
             - ScannerThreadsUserTime: 16.996ms
           - ScannerThreadsVoluntaryContextSwitches: 59 (59)
           - TotalRawHdfsReadTime(*): 15.106ms
           - TotalReadThroughput: 1.03 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000006
(host=slpr-dhc007.lpdomain.com:22000):(Total: 1s444ms, non-child: 1s315ms,
% non-child: 91.12%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 3:1/1.29
MB 2:2/4.14 MB 6:1/444.78 KB
        MemoryUsage(500.000ms): 5.10 MB, 5.01 MB, 455.66 KB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 7.24 MB (7591679)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 105.91 MB (111055430)
         - RowsProduced: 38.66K (38662)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s320ms
         - TotalStorageWaitTime: 69.771ms
         - TotalThreadsInvoluntaryContextSwitches: 94 (94)
         - TotalThreadsTotalWallClockTime: 1s514ms
           - TotalThreadsSysTime: 7.999ms
           - TotalThreadsUserTime: 93.983ms
         - TotalThreadsVoluntaryContextSwitches: 124 (124)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s404ms
             - ExecTreeExecTime: 54.925ms
           - OpenTime: 19.389ms
             - ExecTreeOpenTime: 53.610us
           - PrepareTime: 20.398ms
             - ExecTreePrepareTime: 146.339us
        DataStreamSender (dst_id=11):(Total: 37.498ms, non-child: 37.498ms,
% non-child: 100.00%)
           - BytesSent: 46.98 MB (49262840)
           - NetworkThroughput(*): 2.50 GB/sec
           - OverallThroughput: 1.22 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 38.66K (38662)
           - SerializeBatchTime: 16.134ms
           - TransmitDataRPCTime: 18.317ms
           - UncompressedRowBatchSize: 91.38 MB (95817060)
        CodeGen:(Total: 35.695ms, non-child: 35.695ms, % non-child: 100.00%)
           - CodegenTime: 814.514us
           - CompileTime: 5.998ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 12.677ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 16.588ms
        SUBPLAN_NODE (id=6):(Total: 55.034ms, non-child: 10.012ms, %
non-child: 18.19%)
           - PeakMemoryUsage: 1.27 MB (1332424)
           - RowsReturned: 37.89K (37888)
           - RowsReturnedRate: 688.44 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.183ms, non-child:
24.943ms, % non-child: 75.17%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 38.66K (38662)
             - ProbeTime: 0.000ns
             - RowsReturned: 19.72M (19717525)
             - RowsReturnedRate: 594.20 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 8.240ms, non-child: 8.240ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 38.66K (38662)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 121.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 11.838ms, non-child: 11.838ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
3:1/1.29 MB 2:2/4.14 MB 6:1/444.78 KB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4 out of 4
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
          File Formats: PARQUET/SNAPPY:24
          BytesRead(500.000ms): 1.73 MB, 1.73 MB
           - FooterProcessingTime: (Avg: 12.691ms ; Min: 5.058ms ; Max:
34.003ms ; Number of samples: 4)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.73 MB (1810031)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.73 MB (1810031)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.73 MB (1810031)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 28 (28)
           - DecompressionTime: 1.645ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 3 (3)
           - NumRowGroups: 4 (4)
           - NumScannerThreadsStarted: 4 (4)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 6.97 MB (7305005)
           - PerReadThreadRawHdfsThroughput: 743.47 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 10.650ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 82.44K (82442)
           - RowsReturned: 38.66K (38662)
           - RowsReturnedRate: 3.27 M/sec
           - ScanRangesComplete: 4 (4)
           - ScannerThreadsInvoluntaryContextSwitches: 2 (2)
           - ScannerThreadsTotalWallClockTime: 90.991ms
             - MaterializeTupleTime(*): 10.162ms
             - ScannerThreadsSysTime: 0.000ns
             - ScannerThreadsUserTime: 17.995ms
           - ScannerThreadsVoluntaryContextSwitches: 52 (52)
           - TotalRawHdfsReadTime(*): 2.321ms
           - TotalReadThroughput: 1.73 MB/sec
      Instance 744de1b6228736fa:b54bfaa70000000e
(host=slpr-dhc008.lpdomain.com:22000):(Total: 1s441ms, non-child: 1s228ms,
% non-child: 85.19%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:2/857.88 KB 9:1/683.75 KB 7:1/1.12 MB 8:2/1.83 MB 3:1/392.82 KB
        MemoryUsage(500.000ms): 5.23 MB, 5.23 MB, 1.64 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 5.93 MB (6212879)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 100.53 MB (105413224)
         - RowsProduced: 29.60K (29604)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s203ms
         - TotalStorageWaitTime: 189.481ms
         - TotalThreadsInvoluntaryContextSwitches: 88 (88)
         - TotalThreadsTotalWallClockTime: 1s615ms
           - TotalThreadsSysTime: 7.997ms
           - TotalThreadsUserTime: 138.976ms
         - TotalThreadsVoluntaryContextSwitches: 177 (177)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s330ms
             - ExecTreeExecTime: 71.074ms
           - OpenTime: 46.838ms
             - ExecTreeOpenTime: 104.011us
           - PrepareTime: 63.952ms
             - ExecTreePrepareTime: 330.896us
        DataStreamSender (dst_id=11):(Total: 37.227ms, non-child: 37.227ms,
% non-child: 100.00%)
           - BytesSent: 35.43 MB (37153560)
           - NetworkThroughput(*): 1.90 GB/sec
           - OverallThroughput: 951.78 MB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 29.60K (29604)
           - SerializeBatchTime: 15.769ms
           - TransmitDataRPCTime: 18.254ms
           - UncompressedRowBatchSize: 69.96 MB (73359800)
        CodeGen:(Total: 104.826ms, non-child: 104.826ms, % non-child:
100.00%)
           - CodegenTime: 2.701ms
           - CompileTime: 13.271ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 30.441ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 59.657ms
        SUBPLAN_NODE (id=6):(Total: 71.413ms, non-child: 10.965ms, %
non-child: 15.35%)
           - PeakMemoryUsage: 612.97 KB (627682)
           - RowsReturned: 28.67K (28672)
           - RowsReturnedRate: 401.49 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.170ms, non-child:
27.284ms, % non-child: 77.58%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 29.60K (29604)
             - ProbeTime: 0.000ns
             - RowsReturned: 15.13M (15129178)
             - RowsReturnedRate: 430.16 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 7.885ms, non-child: 7.885ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 29.60K (29604)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 126.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 25.277ms, non-child: 25.277ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:2/857.88 KB 9:1/683.75 KB 7:1/1.12 MB 8:2/1.83 MB 3:1/392.82 KB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
          File Formats: PARQUET/SNAPPY:42
          BytesRead(500.000ms): 1.73 MB, 1.73 MB, 1.73 MB
           - FooterProcessingTime: (Avg: 13.574ms ; Min: 10.108ms ; Max:
28.351ms ; Number of samples: 7)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.73 MB (1812901)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.73 MB (1812901)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.73 MB (1812901)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 49 (49)
           - DecompressionTime: 1.905ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 5 (5)
           - NumRowGroups: 7 (7)
           - NumScannerThreadsStarted: 7 (7)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.65 MB (5926189)
           - PerReadThreadRawHdfsThroughput: 621.87 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 23.273ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 64.74K (64736)
           - RowsReturned: 29.60K (29604)
           - RowsReturnedRate: 1.17 M/sec
           - ScanRangesComplete: 7 (7)
           - ScannerThreadsInvoluntaryContextSwitches: 3 (3)
           - ScannerThreadsTotalWallClockTime: 238.025ms
             - MaterializeTupleTime(*): 17.662ms
             - ScannerThreadsSysTime: 1.998ms
             - ScannerThreadsUserTime: 30.992ms
           - ScannerThreadsVoluntaryContextSwitches: 113 (113)
           - TotalRawHdfsReadTime(*): 2.780ms
           - TotalReadThroughput: 1.15 MB/sec
      Instance 744de1b6228736fa:b54bfaa70000000b
(host=slpr-dhc018.lpdomain.com:22000):(Total: 1s438ms, non-child: 1s311ms,
% non-child: 91.15%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 3:1/1.38
MB 10:3/1.67 MB 5:2/1.48 MB
        MemoryUsage(500.000ms): 4.59 MB, 4.59 MB, 1.53 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 6.33 MB (6640810)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 105.72 MB (110852967)
         - RowsProduced: 28.04K (28043)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s288ms
         - TotalStorageWaitTime: 93.368ms
         - TotalThreadsInvoluntaryContextSwitches: 41 (41)
         - TotalThreadsTotalWallClockTime: 1s536ms
           - TotalThreadsSysTime: 4.999ms
           - TotalThreadsUserTime: 89.983ms
         - TotalThreadsVoluntaryContextSwitches: 173 (173)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s394ms
             - ExecTreeExecTime: 56.183ms
           - OpenTime: 21.596ms
             - ExecTreeOpenTime: 55.153us
           - PrepareTime: 22.675ms
             - ExecTreePrepareTime: 134.513us
        DataStreamSender (dst_id=11):(Total: 29.032ms, non-child: 29.032ms,
% non-child: 100.00%)
           - BytesSent: 33.86 MB (35507960)
           - NetworkThroughput(*): 2.17 GB/sec
           - OverallThroughput: 1.14 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 28.04K (28043)
           - SerializeBatchTime: 12.745ms
           - TransmitDataRPCTime: 15.241ms
           - UncompressedRowBatchSize: 66.28 MB (69502140)
        CodeGen:(Total: 41.989ms, non-child: 41.989ms, % non-child: 100.00%)
           - CodegenTime: 860.034us
           - CompileTime: 6.564ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 13.851ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 21.006ms
        SUBPLAN_NODE (id=6):(Total: 56.265ms, non-child: 9.315ms, %
non-child: 16.56%)
           - PeakMemoryUsage: 688.91 KB (705448)
           - RowsReturned: 27.65K (27648)
           - RowsReturnedRate: 491.38 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 29.399ms, non-child:
22.501ms, % non-child: 76.54%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 28.04K (28043)
             - ProbeTime: 0.000ns
             - RowsReturned: 14.25M (14247810)
             - RowsReturnedRate: 484.62 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 6.898ms, non-child: 6.898ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 28.04K (28043)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 144.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 17.549ms, non-child: 17.549ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
3:1/1.38 MB 10:3/1.67 MB 5:2/1.48 MB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
          File Formats: PARQUET/SNAPPY:36
          BytesRead(500.000ms): 1.58 MB, 1.58 MB, 1.58 MB
           - FooterProcessingTime: (Avg: 5.002ms ; Min: 2.741ms ; Max:
6.341ms ; Number of samples: 6)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.58 MB (1661533)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.58 MB (1661533)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.58 MB (1661533)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 42 (42)
           - DecompressionTime: 1.151ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 3 (3)
           - NumRowGroups: 6 (6)
           - NumScannerThreadsStarted: 6 (6)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 6.06 MB (6354136)
           - PerReadThreadRawHdfsThroughput: 778.82 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 16.292ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 60.93K (60934)
           - RowsReturned: 28.04K (28043)
           - RowsReturnedRate: 1.60 M/sec
           - ScanRangesComplete: 6 (6)
           - ScannerThreadsInvoluntaryContextSwitches: 8 (8)
           - ScannerThreadsTotalWallClockTime: 120.346ms
             - MaterializeTupleTime(*): 9.580ms
             - ScannerThreadsSysTime: 999.000us
             - ScannerThreadsUserTime: 15.994ms
           - ScannerThreadsVoluntaryContextSwitches: 109 (109)
           - TotalRawHdfsReadTime(*): 2.034ms
           - TotalReadThroughput: 1.06 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000009
(host=slpr-dhc017.lpdomain.com:22000):(Total: 1s435ms, non-child: 1s320ms,
% non-child: 91.95%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
9:1/340.56 KB 1:1/1.21 MB 5:2/1013.34 KB 10:2/1.48 MB 8:3/1.48 MB
        MemoryUsage(500.000ms): 224.38 KB, 4.98 MB, 4.61 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 6.35 MB (6661176)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 106.11 MB (111260649)
         - RowsProduced: 32.69K (32687)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s296ms
         - TotalStorageWaitTime: 130.081ms
         - TotalThreadsInvoluntaryContextSwitches: 6 (6)
         - TotalThreadsTotalWallClockTime: 1s582ms
           - TotalThreadsSysTime: 7.997ms
           - TotalThreadsUserTime: 89.982ms
         - TotalThreadsVoluntaryContextSwitches: 259 (259)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s393ms
             - ExecTreeExecTime: 48.186ms
           - OpenTime: 21.458ms
             - ExecTreeOpenTime: 51.008us
           - PrepareTime: 20.257ms
             - ExecTreePrepareTime: 181.262us
        DataStreamSender (dst_id=11):(Total: 28.455ms, non-child: 28.455ms,
% non-child: 100.00%)
           - BytesSent: 39.51 MB (41432100)
           - NetworkThroughput(*): 2.76 GB/sec
           - OverallThroughput: 1.36 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 32.69K (32687)
           - SerializeBatchTime: 12.845ms
           - TransmitDataRPCTime: 13.988ms
           - UncompressedRowBatchSize: 77.25 MB (81006300)
        CodeGen:(Total: 38.787ms, non-child: 38.787ms, % non-child: 100.00%)
           - CodegenTime: 850.797us
           - CompileTime: 6.391ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 14.162ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 17.738ms
        SUBPLAN_NODE (id=6):(Total: 48.306ms, non-child: 8.708ms, %
non-child: 18.03%)
           - PeakMemoryUsage: 684.77 KB (701203)
           - RowsReturned: 31.74K (31744)
           - RowsReturnedRate: 657.14 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 26.113ms, non-child:
20.562ms, % non-child: 78.74%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 32.69K (32687)
             - ProbeTime: 0.000ns
             - RowsReturned: 16.71M (16713896)
             - RowsReturnedRate: 640.04 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 5.551ms, non-child: 5.551ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 32.69K (32687)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 180.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 13.483ms, non-child: 13.483ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
9:1/340.56 KB 1:1/1.21 MB 5:2/1013.34 KB 10:2/1.48 MB 8:3/1.48 MB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 9 out of 9
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
          File Formats: PARQUET/SNAPPY:54
          BytesRead(500.000ms): 0, 2.06 MB, 2.06 MB
           - FooterProcessingTime: (Avg: 6.123ms ; Min: 3.754ms ; Max:
8.084ms ; Number of samples: 9)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 2.06 MB (2157072)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 2.06 MB (2157072)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 2.06 MB (2157072)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 63 (63)
           - DecompressionTime: 1.269ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 5 (5)
           - NumRowGroups: 9 (9)
           - NumScannerThreadsStarted: 9 (9)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.88 MB (6167436)
           - PerReadThreadRawHdfsThroughput: 921.10 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 12.190ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 72.05K (72054)
           - RowsReturned: 32.69K (32687)
           - RowsReturnedRate: 2.42 M/sec
           - ScanRangesComplete: 9 (9)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 167.518ms
             - MaterializeTupleTime(*): 9.184ms
             - ScannerThreadsSysTime: 3.997ms
             - ScannerThreadsUserTime: 17.993ms
           - ScannerThreadsVoluntaryContextSwitches: 177 (177)
           - TotalRawHdfsReadTime(*): 2.233ms
           - TotalReadThroughput: 1.37 MB/sec
      Instance 744de1b6228736fa:b54bfaa70000000a
(host=slpr-dhc010.lpdomain.com:22000):(Total: 1s412ms, non-child: 1s287ms,
% non-child: 91.14%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 8:1/1.38
MB 0:2/1.45 MB 4:1/1.16 MB 7:1/226.78 KB 9:1/1.28 MB
        MemoryUsage(500.000ms): 4.21 MB, 5.06 MB, 3.83 MB
        ThreadUsage(500.000ms): 5, 1, 1
         - AverageThreadTokens: 2.33
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 6.21 MB (6507726)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 78.98 MB (82816582)
         - RowsProduced: 34.84K (34842)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s211ms
         - TotalStorageWaitTime: 112.256ms
         - TotalThreadsInvoluntaryContextSwitches: 9 (9)
         - TotalThreadsTotalWallClockTime: 1s532ms
           - TotalThreadsSysTime: 1.999ms
           - TotalThreadsUserTime: 92.982ms
         - TotalThreadsVoluntaryContextSwitches: 185 (185)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s375ms
             - ExecTreeExecTime: 59.267ms
           - OpenTime: 17.728ms
             - ExecTreeOpenTime: 46.848us
           - PrepareTime: 18.874ms
             - ExecTreePrepareTime: 137.282us
        DataStreamSender (dst_id=11):(Total: 32.771ms, non-child: 32.771ms,
% non-child: 100.00%)
           - BytesSent: 42.05 MB (44091120)
           - NetworkThroughput(*): 2.54 GB/sec
           - OverallThroughput: 1.25 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 34.84K (34842)
           - SerializeBatchTime: 13.977ms
           - TransmitDataRPCTime: 16.163ms
           - UncompressedRowBatchSize: 82.35 MB (86350200)
        CodeGen:(Total: 32.930ms, non-child: 32.930ms, % non-child: 100.00%)
           - CodegenTime: 686.488us
           - CompileTime: 5.514ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 11.619ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 15.395ms
        SUBPLAN_NODE (id=6):(Total: 59.389ms, non-child: 10.261ms, %
non-child: 17.28%)
           - PeakMemoryUsage: 683.40 KB (699799)
           - RowsReturned: 34.82K (34816)
           - RowsReturnedRate: 586.23 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 30.704ms, non-child:
24.298ms, % non-child: 79.14%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 34.84K (34842)
             - ProbeTime: 0.000ns
             - RowsReturned: 17.84M (17843551)
             - RowsReturnedRate: 581.14 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 6.406ms, non-child: 6.406ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 34.84K (34842)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 156.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 18.423ms, non-child: 18.423ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
8:1/1.38 MB 0:2/1.45 MB 4:1/1.16 MB 7:1/226.78 KB 9:1/1.28 MB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
          Hdfs Read Thread Concurrency Bucket: 0:66.67% 1:0% 2:33.33% 3:0%
4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
17:0%
          File Formats: PARQUET/SNAPPY:36
          BytesRead(500.000ms): 1.61 MB, 1.82 MB, 1.82 MB
           - FooterProcessingTime: (Avg: 6.872ms ; Min: 5.472ms ; Max:
7.952ms ; Number of samples: 6)
           - AverageHdfsReadThreadConcurrency: 0.67
           - AverageScannerThreadConcurrency: 1.33
           - BytesRead: 1.82 MB (1908230)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.82 MB (1908230)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.82 MB (1908230)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 42 (42)
           - DecompressionTime: 1.046ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 5 (5)
           - NumRowGroups: 6 (6)
           - NumScannerThreadsStarted: 6 (6)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.93 MB (6221052)
           - PerReadThreadRawHdfsThroughput: 68.29 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 17.139ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 74.63K (74626)
           - RowsReturned: 34.84K (34842)
           - RowsReturnedRate: 1.89 M/sec
           - ScanRangesComplete: 6 (6)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 139.205ms
             - MaterializeTupleTime(*): 7.716ms
             - ScannerThreadsSysTime: 999.000us
             - ScannerThreadsUserTime: 14.994ms
           - ScannerThreadsVoluntaryContextSwitches: 98 (98)
           - TotalRawHdfsReadTime(*): 26.646ms
           - TotalReadThroughput: 1.21 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000003
(host=slpr-dhc005.lpdomain.com:22000):(Total: 1s408ms, non-child: 1s277ms,
% non-child: 90.69%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
10:1/519.80 KB 5:2/574.88 KB 2:1/1.39 MB 1:2/1.03 MB 9:1/560.57 KB
6:1/170.23 KB
        MemoryUsage(500.000ms): 4.53 MB, 4.44 MB, 1.01 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 5.28 MB (5534493)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 77.37 MB (81124776)
         - RowsProduced: 24.11K (24114)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s271ms
         - TotalStorageWaitTime: 142.392ms
         - TotalThreadsInvoluntaryContextSwitches: 42 (42)
         - TotalThreadsTotalWallClockTime: 1s555ms
           - TotalThreadsSysTime: 6.998ms
           - TotalThreadsUserTime: 99.979ms
         - TotalThreadsVoluntaryContextSwitches: 194 (194)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s343ms
             - ExecTreeExecTime: 43.864ms
           - OpenTime: 38.771ms
             - ExecTreeOpenTime: 90.450us
           - PrepareTime: 25.964ms
             - ExecTreePrepareTime: 225.481us
        DataStreamSender (dst_id=11):(Total: 25.743ms, non-child: 25.743ms,
% non-child: 100.00%)
           - BytesSent: 28.99 MB (30395020)
           - NetworkThroughput(*): 2.37 GB/sec
           - OverallThroughput: 1.10 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 24.11K (24114)
           - SerializeBatchTime: 12.033ms
           - TransmitDataRPCTime: 11.924ms
           - UncompressedRowBatchSize: 56.99 MB (59755180)
        CodeGen:(Total: 61.321ms, non-child: 61.321ms, % non-child: 100.00%)
           - CodegenTime: 1.102ms
           - CompileTime: 12.945ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 24.479ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 23.334ms
        SUBPLAN_NODE (id=6):(Total: 44.097ms, non-child: 7.911ms, %
non-child: 17.94%)
           - PeakMemoryUsage: 730.97 KB (748516)
           - RowsReturned: 23.55K (23552)
           - RowsReturnedRate: 534.09 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 22.563ms, non-child:
16.978ms, % non-child: 75.25%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 24.11K (24114)
             - ProbeTime: 0.000ns
             - RowsReturned: 12.23M (12228603)
             - RowsReturnedRate: 541.96 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 5.585ms, non-child: 5.585ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 24.11K (24114)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 179.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 13.622ms, non-child: 13.622ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
10:1/519.80 KB 5:2/574.88 KB 2:1/1.39 MB 1:2/1.03 MB 9:1/560.57 KB
6:1/170.23 KB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 8 out of 8
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
          File Formats: PARQUET/SNAPPY:48
          BytesRead(500.000ms): 1.65 MB, 1.65 MB, 1.65 MB
           - FooterProcessingTime: (Avg: 12.545ms ; Min: 6.441ms ; Max:
22.143ms ; Number of samples: 8)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.65 MB (1734402)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.65 MB (1734402)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.65 MB (1734402)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 56 (56)
           - DecompressionTime: 1.145ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 6 (6)
           - NumRowGroups: 8 (8)
           - NumScannerThreadsStarted: 8 (8)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.00 MB (5247819)
           - PerReadThreadRawHdfsThroughput: 736.05 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 11.739ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 54.66K (54658)
           - RowsReturned: 24.11K (24114)
           - RowsReturnedRate: 1.77 M/sec
           - ScanRangesComplete: 8 (8)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 173.037ms
             - MaterializeTupleTime(*): 9.293ms
             - ScannerThreadsSysTime: 999.000us
             - ScannerThreadsUserTime: 19.991ms
           - ScannerThreadsVoluntaryContextSwitches: 136 (136)
           - TotalRawHdfsReadTime(*): 2.247ms
           - TotalReadThroughput: 1.10 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000010
(host=slpr-dhc011.lpdomain.com:22000):(Total: 1s408ms, non-child: 1s272ms,
% non-child: 90.31%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 8:1/1.34
MB 5:1/177.75 KB 9:2/1.51 MB 11:2/1.48 MB 3:1/173.83 KB
        MemoryUsage(500.000ms): 4.41 MB, 4.41 MB, 988.24 KB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 5.24 MB (5494080)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 77.00 MB (80742460)
         - RowsProduced: 28.24K (28236)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s271ms
         - TotalStorageWaitTime: 104.916ms
         - TotalThreadsInvoluntaryContextSwitches: 62 (62)
         - TotalThreadsTotalWallClockTime: 1s514ms
           - TotalThreadsSysTime: 8.996ms
           - TotalThreadsUserTime: 88.982ms
         - TotalThreadsVoluntaryContextSwitches: 172 (172)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s356ms
             - ExecTreeExecTime: 50.194ms
           - OpenTime: 26.941ms
             - ExecTreeOpenTime: 59.915us
           - PrepareTime: 24.743ms
             - ExecTreePrepareTime: 169.120us
        DataStreamSender (dst_id=11):(Total: 39.302ms, non-child: 39.302ms,
% non-child: 100.00%)
           - BytesSent: 34.08 MB (35740540)
           - NetworkThroughput(*): 1.48 GB/sec
           - OverallThroughput: 867.24 MB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 28.24K (28236)
           - SerializeBatchTime: 13.265ms
           - TransmitDataRPCTime: 22.511ms
           - UncompressedRowBatchSize: 66.74 MB (69979140)
        CodeGen:(Total: 46.908ms, non-child: 46.908ms, % non-child: 100.00%)
           - CodegenTime: 999.391us
           - CompileTime: 8.197ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 17.861ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 20.338ms
        SUBPLAN_NODE (id=6):(Total: 50.346ms, non-child: 8.249ms, %
non-child: 16.39%)
           - PeakMemoryUsage: 706.48 KB (723433)
           - RowsReturned: 27.65K (27648)
           - RowsReturnedRate: 549.15 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 27.022ms, non-child:
20.861ms, % non-child: 77.20%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 28.24K (28236)
             - ProbeTime: 0.000ns
             - RowsReturned: 14.34M (14342766)
             - RowsReturnedRate: 530.77 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 6.160ms, non-child: 6.160ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 28.24K (28236)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 162.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 15.074ms, non-child: 15.074ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
8:1/1.34 MB 5:1/177.75 KB 9:2/1.51 MB 11:2/1.48 MB 3:1/173.83 KB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
          File Formats: PARQUET/SNAPPY:42
          BytesRead(500.000ms): 1.69 MB, 1.69 MB, 1.69 MB
           - FooterProcessingTime: (Avg: 6.723ms ; Min: 5.085ms ; Max:
8.167ms ; Number of samples: 7)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.69 MB (1777017)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.69 MB (1777017)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.69 MB (1777017)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 49 (49)
           - DecompressionTime: 1.245ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 5 (5)
           - NumRowGroups: 7 (7)
           - NumScannerThreadsStarted: 7 (7)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.05 MB (5294444)
           - PerReadThreadRawHdfsThroughput: 617.71 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 13.422ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 61.76K (61756)
           - RowsReturned: 28.24K (28236)
           - RowsReturnedRate: 1.87 M/sec
           - ScanRangesComplete: 7 (7)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 131.024ms
             - MaterializeTupleTime(*): 8.897ms
             - ScannerThreadsSysTime: 3.997ms
             - ScannerThreadsUserTime: 12.994ms
           - ScannerThreadsVoluntaryContextSwitches: 108 (108)
           - TotalRawHdfsReadTime(*): 2.743ms
           - TotalReadThroughput: 1.13 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000012
(host=slpr-dhc015.lpdomain.com:22000):(Total: 1s407ms, non-child: 1s287ms,
% non-child: 91.44%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:1/1.01 MB 9:1/187.94 KB 10:1/244.30 KB 1:1/1.26 MB 2:1/222.25 KB
6:1/1.29 MB 8:2/1.26 MB
        MemoryUsage(500.000ms): 4.95 MB, 4.95 MB, 960.14 KB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 5.85 MB (6131090)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 76.05 MB (79742718)
         - RowsProduced: 33.12K (33121)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s287ms
         - TotalStorageWaitTime: 163.998ms
         - TotalThreadsInvoluntaryContextSwitches: 10 (10)
         - TotalThreadsTotalWallClockTime: 1s581ms
           - TotalThreadsSysTime: 5.999ms
           - TotalThreadsUserTime: 91.981ms
         - TotalThreadsVoluntaryContextSwitches: 202 (202)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s369ms
             - ExecTreeExecTime: 55.689ms
           - OpenTime: 18.168ms
             - ExecTreeOpenTime: 46.705us
           - PrepareTime: 19.634ms
             - ExecTreePrepareTime: 170.693us
        DataStreamSender (dst_id=11):(Total: 30.910ms, non-child: 30.910ms,
% non-child: 100.00%)
           - BytesSent: 39.88 MB (41812920)
           - NetworkThroughput(*): 2.75 GB/sec
           - OverallThroughput: 1.26 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 33.12K (33121)
           - SerializeBatchTime: 13.970ms
           - TransmitDataRPCTime: 14.144ms
           - UncompressedRowBatchSize: 78.28 MB (82081040)
        CodeGen:(Total: 33.819ms, non-child: 33.819ms, % non-child: 100.00%)
           - CodegenTime: 794.441us
           - CompileTime: 5.469ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 11.979ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 15.849ms
        SUBPLAN_NODE (id=6):(Total: 55.809ms, non-child: 10.461ms, %
non-child: 18.75%)
           - PeakMemoryUsage: 675.86 KB (692077)
           - RowsReturned: 32.77K (32768)
           - RowsReturnedRate: 587.14 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 30.755ms, non-child:
24.689ms, % non-child: 80.28%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 33.12K (33121)
             - ProbeTime: 0.000ns
             - RowsReturned: 16.86M (16856081)
             - RowsReturnedRate: 548.06 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 6.065ms, non-child: 6.065ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 33.12K (33121)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 164.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 14.592ms, non-child: 14.592ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:1/1.01 MB 9:1/187.94 KB 10:1/244.30 KB 1:1/1.26 MB 2:1/222.25 KB
6:1/1.29 MB 8:2/1.26 MB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 8 out of 8
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
          File Formats: PARQUET/SNAPPY:48
          BytesRead(500.000ms): 1.96 MB, 1.96 MB, 1.96 MB
           - FooterProcessingTime: (Avg: 15.586ms ; Min: 7.605ms ; Max:
29.109ms ; Number of samples: 8)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.96 MB (2056608)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.96 MB (2056608)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.96 MB (2056608)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 56 (56)
           - DecompressionTime: 1.216ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 7 (7)
           - NumRowGroups: 8 (8)
           - NumScannerThreadsStarted: 8 (8)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.57 MB (5844400)
           - PerReadThreadRawHdfsThroughput: 1017.18 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 12.906ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 72.55K (72552)
           - RowsReturned: 33.12K (33121)
           - RowsReturnedRate: 2.27 M/sec
           - ScanRangesComplete: 8 (8)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 193.175ms
             - MaterializeTupleTime(*): 9.160ms
             - ScannerThreadsSysTime: 999.000us
             - ScannerThreadsUserTime: 16.992ms
           - ScannerThreadsVoluntaryContextSwitches: 128 (128)
           - TotalRawHdfsReadTime(*): 1.928ms
           - TotalReadThroughput: 1.31 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000007
(host=slpr-dhc014.lpdomain.com:22000):(Total: 1s405ms, non-child: 1s285ms,
% non-child: 91.46%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 5:1/1.21
MB 1:1/214.31 KB 2:1/1.35 MB 10:2/1.97 MB 11:1/172.29 KB 9:1/179.34 KB
        MemoryUsage(500.000ms): 4.40 MB, 4.40 MB, 1.41 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 6.11 MB (6411936)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 76.12 MB (79813012)
         - RowsProduced: 31.24K (31237)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s285ms
         - TotalStorageWaitTime: 105.765ms
         - TotalThreadsInvoluntaryContextSwitches: 4 (4)
         - TotalThreadsTotalWallClockTime: 1s515ms
           - TotalThreadsSysTime: 4.998ms
           - TotalThreadsUserTime: 87.983ms
         - TotalThreadsVoluntaryContextSwitches: 167 (167)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s364ms
             - ExecTreeExecTime: 53.780ms
           - OpenTime: 20.139ms
             - ExecTreeOpenTime: 50.643us
           - PrepareTime: 20.954ms
             - ExecTreePrepareTime: 151.743us
        DataStreamSender (dst_id=11):(Total: 29.295ms, non-child: 29.295ms,
% non-child: 100.00%)
           - BytesSent: 37.51 MB (39335440)
           - NetworkThroughput(*): 2.73 GB/sec
           - OverallThroughput: 1.25 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 31.24K (31237)
           - SerializeBatchTime: 12.675ms
           - TransmitDataRPCTime: 13.419ms
           - UncompressedRowBatchSize: 73.83 MB (77411840)
        CodeGen:(Total: 36.845ms, non-child: 36.845ms, % non-child: 100.00%)
           - CodegenTime: 695.456us
           - CompileTime: 6.334ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 13.147ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 16.920ms
        SUBPLAN_NODE (id=6):(Total: 53.928ms, non-child: 9.101ms, %
non-child: 16.88%)
           - PeakMemoryUsage: 673.53 KB (689692)
           - RowsReturned: 30.72K (30720)
           - RowsReturnedRate: 569.64 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 27.770ms, non-child:
21.243ms, % non-child: 76.50%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 31.24K (31237)
             - ProbeTime: 0.000ns
             - RowsReturned: 15.88M (15877903)
             - RowsReturnedRate: 571.75 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 6.527ms, non-child: 6.527ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 31.24K (31237)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 153.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 17.055ms, non-child: 17.055ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
5:1/1.21 MB 1:1/214.31 KB 2:1/1.35 MB 10:2/1.97 MB 11:1/172.29 KB
9:1/179.34 KB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
          File Formats: PARQUET/SNAPPY:42
          BytesRead(500.000ms): 1.79 MB, 1.79 MB, 1.79 MB
           - FooterProcessingTime: (Avg: 8.079ms ; Min: 7.484ms ; Max:
9.282ms ; Number of samples: 7)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.79 MB (1881688)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.79 MB (1881688)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.79 MB (1881688)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 49 (49)
           - DecompressionTime: 1.425ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 6 (6)
           - NumRowGroups: 7 (7)
           - NumScannerThreadsStarted: 7 (7)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.84 MB (6125262)
           - PerReadThreadRawHdfsThroughput: 787.42 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 15.401ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 68.03K (68026)
           - RowsReturned: 31.24K (31237)
           - RowsReturnedRate: 1.83 M/sec
           - ScanRangesComplete: 7 (7)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 130.754ms
             - MaterializeTupleTime(*): 7.935ms
             - ScannerThreadsSysTime: 1.998ms
             - ScannerThreadsUserTime: 14.994ms
           - ScannerThreadsVoluntaryContextSwitches: 101 (101)
           - TotalRawHdfsReadTime(*): 2.278ms
           - TotalReadThroughput: 1.20 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000011
(host=slpr-dhc016.lpdomain.com:22000):(Total: 1s402ms, non-child: 1s303ms,
% non-child: 92.94%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
6:1/486.13 KB 2:2/1.23 MB 4:1/838.47 KB 3:1/260.54 KB 9:1/192.11 KB
1:2/1.73 MB
        MemoryUsage(500.000ms): 4.08 MB, 4.08 MB, 2.62 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 5.50 MB (5763668)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 75.78 MB (79461951)
         - RowsProduced: 27.73K (27728)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s301ms
         - TotalStorageWaitTime: 121.015ms
         - TotalThreadsInvoluntaryContextSwitches: 7 (7)
         - TotalThreadsTotalWallClockTime: 1s531ms
           - TotalThreadsSysTime: 6.998ms
           - TotalThreadsUserTime: 65.984ms
         - TotalThreadsVoluntaryContextSwitches: 204 (204)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s363ms
             - ExecTreeExecTime: 39.105ms
           - OpenTime: 19.516ms
             - ExecTreeOpenTime: 47.279us
           - PrepareTime: 19.365ms
             - ExecTreePrepareTime: 133.101us
        DataStreamSender (dst_id=11):(Total: 23.762ms, non-child: 23.762ms,
% non-child: 100.00%)
           - BytesSent: 33.39 MB (35012100)
           - NetworkThroughput(*): 2.65 GB/sec
           - OverallThroughput: 1.37 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 27.73K (27728)
           - SerializeBatchTime: 9.546ms
           - TransmitDataRPCTime: 12.286ms
           - UncompressedRowBatchSize: 65.53 MB (68713240)
        CodeGen:(Total: 35.956ms, non-child: 35.956ms, % non-child: 100.00%)
           - CodegenTime: 721.314us
           - CompileTime: 6.112ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 12.794ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 16.599ms
        SUBPLAN_NODE (id=6):(Total: 39.213ms, non-child: 5.759ms, %
non-child: 14.69%)
           - PeakMemoryUsage: 671.21 KB (687319)
           - RowsReturned: 27.65K (27648)
           - RowsReturnedRate: 705.06 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 18.599ms, non-child:
14.247ms, % non-child: 76.60%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 27.73K (27728)
             - ProbeTime: 0.000ns
             - RowsReturned: 14.17M (14172840)
             - RowsReturnedRate: 761.98 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 4.352ms, non-child: 4.352ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 27.73K (27728)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 229.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 14.853ms, non-child: 14.853ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
6:1/486.13 KB 2:2/1.23 MB 4:1/838.47 KB 3:1/260.54 KB 9:1/192.11 KB
1:2/1.73 MB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 8 out of 8
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
          File Formats: PARQUET/SNAPPY:48
          BytesRead(500.000ms): 1.78 MB, 1.78 MB, 1.78 MB
           - FooterProcessingTime: (Avg: 7.908ms ; Min: 7.090ms ; Max:
8.892ms ; Number of samples: 8)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.78 MB (1867919)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.78 MB (1867919)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.78 MB (1867919)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 56 (56)
           - DecompressionTime: 1.045ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 6 (6)
           - NumRowGroups: 8 (8)
           - NumScannerThreadsStarted: 8 (8)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.14 MB (5388373)
           - PerReadThreadRawHdfsThroughput: 923.56 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 13.628ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 61.73K (61732)
           - RowsReturned: 27.73K (27728)
           - RowsReturnedRate: 1.87 M/sec
           - ScanRangesComplete: 8 (8)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 149.221ms
             - MaterializeTupleTime(*): 7.339ms
             - ScannerThreadsSysTime: 999.000us
             - ScannerThreadsUserTime: 12.992ms
           - ScannerThreadsVoluntaryContextSwitches: 138 (138)
           - TotalRawHdfsReadTime(*): 1.928ms
           - TotalReadThroughput: 1.19 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000014
(host=slpr-dhc020.lpdomain.com:22000):(Total: 1s399ms, non-child: 1s282ms,
% non-child: 91.64%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:1/182.35 KB 3:1/1.25 MB 0:1/181.94 KB 7:1/374.64 KB 2:1/1.32 MB 8:2/1.62
MB
        MemoryUsage(500.000ms): 3.78 MB, 4.10 MB, 2.76 MB
        ThreadUsage(500.000ms): 2, 1, 1
         - AverageThreadTokens: 1.33
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 4.83 MB (5062033)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 81.51 MB (85472224)
         - RowsProduced: 29.83K (29834)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s274ms
         - TotalStorageWaitTime: 106.847ms
         - TotalThreadsInvoluntaryContextSwitches: 32 (32)
         - TotalThreadsTotalWallClockTime: 1s509ms
           - TotalThreadsSysTime: 4.999ms
           - TotalThreadsUserTime: 86.982ms
         - TotalThreadsVoluntaryContextSwitches: 178 (178)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s359ms
             - ExecTreeExecTime: 49.176ms
           - OpenTime: 20.060ms
             - ExecTreeOpenTime: 46.820us
           - PrepareTime: 19.886ms
             - ExecTreePrepareTime: 150.664us
        DataStreamSender (dst_id=11):(Total: 31.378ms, non-child: 31.378ms,
% non-child: 100.00%)
           - BytesSent: 35.99 MB (37739200)
           - NetworkThroughput(*): 2.25 GB/sec
           - OverallThroughput: 1.12 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 29.83K (29834)
           - SerializeBatchTime: 13.343ms
           - TransmitDataRPCTime: 15.610ms
           - UncompressedRowBatchSize: 70.51 MB (73936460)
        CodeGen:(Total: 36.307ms, non-child: 36.307ms, % non-child: 100.00%)
           - CodegenTime: 818.733us
           - CompileTime: 5.940ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 13.387ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 16.467ms
        SUBPLAN_NODE (id=6):(Total: 49.274ms, non-child: 9.236ms, %
non-child: 18.74%)
           - PeakMemoryUsage: 735.13 KB (752773)
           - RowsReturned: 29.70K (29696)
           - RowsReturnedRate: 602.66 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 29.023ms, non-child:
22.027ms, % non-child: 75.89%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 29.83K (29834)
             - ProbeTime: 0.000ns
             - RowsReturned: 15.23M (15228791)
             - RowsReturnedRate: 524.71 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 6.996ms, non-child: 6.996ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 29.83K (29834)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 142.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 11.015ms, non-child: 11.015ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:1/182.35 KB 3:1/1.25 MB 0:1/181.94 KB 7:1/374.64 KB 2:1/1.32 MB 8:2/1.62
MB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
          Hdfs Read Thread Concurrency Bucket: 0:66.67% 1:33.33% 2:0% 3:0%
4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
17:0%
          File Formats: PARQUET/SNAPPY:42
          BytesRead(500.000ms): 1.70 MB, 1.75 MB, 1.75 MB
           - FooterProcessingTime: (Avg: 10.602ms ; Min: 5.250ms ; Max:
38.505ms ; Number of samples: 7)
           - AverageHdfsReadThreadConcurrency: 0.33
           - AverageScannerThreadConcurrency: 0.33
           - BytesRead: 1.75 MB (1837043)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.75 MB (1837043)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.75 MB (1837043)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 49 (49)
           - DecompressionTime: 979.025us
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 6 (6)
           - NumRowGroups: 7 (7)
           - NumScannerThreadsStarted: 7 (7)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 4.55 MB (4775359)
           - PerReadThreadRawHdfsThroughput: 456.04 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 9.540ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 65.25K (65252)
           - RowsReturned: 29.83K (29834)
           - RowsReturnedRate: 2.71 M/sec
           - ScanRangesComplete: 7 (7)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 129.926ms
             - MaterializeTupleTime(*): 6.955ms
             - ScannerThreadsSysTime: 999.000us
             - ScannerThreadsUserTime: 12.993ms
           - ScannerThreadsVoluntaryContextSwitches: 106 (106)
           - TotalRawHdfsReadTime(*): 3.841ms
           - TotalReadThroughput: 1.17 MB/sec
      Instance 744de1b6228736fa:b54bfaa70000000c
(host=slpr-dhc019.lpdomain.com:22000):(Total: 1s392ms, non-child: 1s281ms,
% non-child: 92.04%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 9:1/1.02
MB 7:1/1.10 MB 0:1/1.22 MB 8:1/1.26 MB 1:2/395.98 KB
        MemoryUsage(500.000ms): 4.94 MB, 4.94 MB, 1.19 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 7.00 MB (7340476)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 75.63 MB (79303233)
         - RowsProduced: 31.25K (31245)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s191ms
         - TotalStorageWaitTime: 106.956ms
         - TotalThreadsInvoluntaryContextSwitches: 73 (73)
         - TotalThreadsTotalWallClockTime: 1s503ms
           - TotalThreadsSysTime: 2.999ms
           - TotalThreadsUserTime: 81.984ms
         - TotalThreadsVoluntaryContextSwitches: 183 (183)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s356ms
             - ExecTreeExecTime: 48.776ms
           - OpenTime: 17.984ms
             - ExecTreeOpenTime: 44.576us
           - PrepareTime: 17.956ms
             - ExecTreePrepareTime: 161.729us
        DataStreamSender (dst_id=11):(Total: 29.063ms, non-child: 29.063ms,
% non-child: 100.00%)
           - BytesSent: 37.68 MB (39513400)
           - NetworkThroughput(*): 2.30 GB/sec
           - OverallThroughput: 1.27 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 31.25K (31245)
           - SerializeBatchTime: 11.112ms
           - TransmitDataRPCTime: 16.017ms
           - UncompressedRowBatchSize: 73.85 MB (77434140)
        CodeGen:(Total: 32.932ms, non-child: 32.932ms, % non-child: 100.00%)
           - CodegenTime: 720.261us
           - CompileTime: 5.503ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 11.844ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 15.153ms
        SUBPLAN_NODE (id=6):(Total: 48.848ms, non-child: 7.468ms, %
non-child: 15.29%)
           - PeakMemoryUsage: 666.42 KB (682417)
           - RowsReturned: 30.72K (30720)
           - RowsReturnedRate: 628.88 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 25.718ms, non-child:
19.277ms, % non-child: 74.96%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 31.25K (31245)
             - ProbeTime: 0.000ns
             - RowsReturned: 15.88M (15882075)
             - RowsReturnedRate: 617.53 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 6.440ms, non-child: 6.440ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 31.25K (31245)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 155.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 15.661ms, non-child: 15.661ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
9:1/1.02 MB 7:1/1.10 MB 0:1/1.22 MB 8:1/1.26 MB 1:2/395.98 KB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
          File Formats: PARQUET/SNAPPY:36
          BytesRead(500.000ms): 1.69 MB, 1.69 MB, 1.69 MB
           - FooterProcessingTime: (Avg: 11.127ms ; Min: 4.612ms ; Max:
23.576ms ; Number of samples: 6)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.69 MB (1772933)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.69 MB (1772933)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.69 MB (1772933)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 42 (42)
           - DecompressionTime: 1.000ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 5 (5)
           - NumRowGroups: 6 (6)
           - NumScannerThreadsStarted: 6 (6)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 6.73 MB (7053786)
           - PerReadThreadRawHdfsThroughput: 1.12 GB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 14.492ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 67.31K (67314)
           - RowsReturned: 31.25K (31245)
           - RowsReturnedRate: 1.99 M/sec
           - ScanRangesComplete: 6 (6)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 129.305ms
             - MaterializeTupleTime(*): 9.107ms
             - ScannerThreadsSysTime: 999.000us
             - ScannerThreadsUserTime: 16.994ms
           - ScannerThreadsVoluntaryContextSwitches: 111 (111)
           - TotalRawHdfsReadTime(*): 1.475ms
           - TotalReadThroughput: 1.13 MB/sec
      Instance 744de1b6228736fa:b54bfaa700000001
(host=slpr-dhc013.lpdomain.com:22000):(Total: 1s292ms, non-child: 1s188ms,
% non-child: 91.97%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:1/172.94 KB 1:1/680.67 KB 7:1/1.19 MB 6:1/220.94 KB 4:2/2.54 MB
        MemoryUsage(500.000ms): 99.38 KB, 4.08 MB, 2.86 MB
        ThreadUsage(500.000ms): 1, 1, 1
         - AverageThreadTokens: 1.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 5.22 MB (5477186)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 76.04 MB (79737600)
         - RowsProduced: 29.82K (29816)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1s194ms
         - TotalStorageWaitTime: 84.889ms
         - TotalThreadsInvoluntaryContextSwitches: 9 (9)
         - TotalThreadsTotalWallClockTime: 1s379ms
           - TotalThreadsSysTime: 5.998ms
           - TotalThreadsUserTime: 75.986ms
         - TotalThreadsVoluntaryContextSwitches: 171 (171)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 1s250ms
             - ExecTreeExecTime: 39.392ms
           - OpenTime: 20.762ms
             - ExecTreeOpenTime: 58.843us
           - PrepareTime: 20.558ms
             - ExecTreePrepareTime: 182.009us
        DataStreamSender (dst_id=11):(Total: 26.737ms, non-child: 26.737ms,
% non-child: 100.00%)
           - BytesSent: 35.89 MB (37634460)
           - NetworkThroughput(*): 2.64 GB/sec
           - OverallThroughput: 1.31 GB/sec
           - PeakMemoryUsage: 59.38 KB (60800)
           - RowsReturned: 29.82K (29816)
           - SerializeBatchTime: 10.888ms
           - TransmitDataRPCTime: 13.267ms
           - UncompressedRowBatchSize: 70.47 MB (73892760)
        CodeGen:(Total: 37.505ms, non-child: 37.505ms, % non-child: 100.00%)
           - CodegenTime: 855.587us
           - CompileTime: 6.296ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.96 MB (2050180)
           - NumFunctions: 16 (16)
           - NumInstructions: 250 (250)
           - OptimizationTime: 13.802ms
           - PeakMemoryUsage: 125.00 KB (128000)
           - PrepareTime: 16.861ms
        SUBPLAN_NODE (id=6):(Total: 39.529ms, non-child: 6.476ms, %
non-child: 16.38%)
           - PeakMemoryUsage: 676.42 KB (692656)
           - RowsReturned: 29.70K (29696)
           - RowsReturnedRate: 751.23 K/sec
          NESTED_LOOP_JOIN_NODE (id=9):(Total: 21.183ms, non-child:
16.175ms, % non-child: 76.36%)
             - BuildRows: 0 (0)
             - BuildTime: 0.000ns
             - PeakMemoryUsage: 24.00 KB (24576)
             - ProbeRows: 29.82K (29816)
             - ProbeTime: 0.000ns
             - RowsReturned: 15.23M (15226460)
             - RowsReturnedRate: 718.80 M/sec
            Nested Loop Join Builder:
               - PeakMemoryUsage: 8.00 KB (8192)
            SINGULAR_ROW_SRC_NODE (id=7):
               - PeakMemoryUsage: 0
               - RowsReturned: 0 (0)
               - RowsReturnedRate: 0
          UNNEST_NODE (id=8):(Total: 5.007ms, non-child: 5.007ms, %
non-child: 100.00%)
             - AvgCollectionSize: 1.00
             - MaxCollectionSize: 1 (1)
             - MinCollectionSize: 1 (1)
             - NumCollections: 29.82K (29816)
             - PeakMemoryUsage: 0
             - RowsReturned: 1 (1)
             - RowsReturnedRate: 199.00 /sec
        HDFS_SCAN_NODE (id=5):(Total: 11.870ms, non-child: 11.870ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:1/172.94 KB 1:1/680.67 KB 7:1/1.19 MB 6:1/220.94 KB 4:2/2.54 MB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
          File Formats: PARQUET/SNAPPY:36
          BytesRead(500.000ms): 0, 1.64 MB, 1.64 MB
           - FooterProcessingTime: (Avg: 6.385ms ; Min: 5.928ms ; Max:
7.410ms ; Number of samples: 6)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 0.00
           - BytesRead: 1.64 MB (1721793)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 1.64 MB (1721793)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 1.64 MB (1721793)
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 42 (42)
           - DecompressionTime: 1.224ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 6 (6)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 5 (5)
           - NumRowGroups: 6 (6)
           - NumScannerThreadsStarted: 6 (6)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 4.87 MB (5102432)
           - PerReadThreadRawHdfsThroughput: 86.13 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 10.358ms
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 64.41K (64410)
           - RowsReturned: 29.82K (29816)
           - RowsReturnedRate: 2.51 M/sec
           - ScanRangesComplete: 6 (6)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 107.697ms
             - MaterializeTupleTime(*): 8.462ms
             - ScannerThreadsSysTime: 1.998ms
             - ScannerThreadsUserTime: 14.995ms
           - ScannerThreadsVoluntaryContextSwitches: 92 (92)
           - TotalRawHdfsReadTime(*): 19.064ms
           - TotalReadThroughput: 1.09 MB/sec

[slpr-dhc018.lpdomain.com:21000] >



On Thu, Feb 22, 2018 at 9:41 PM, Mostafa Mokhtar <mmokhtar@cloudera.com>
wrote:

> @Fawze,
>
> To best serve your question please share a profile for the problematic
> query.
>
> Thanks
> Mostafa
>
> On Thu, Feb 22, 2018 at 9:40 AM, Jim Apple <jbapple@cloudera.com> wrote:
>
>> I think compute stats will help estimates be more accurate.
>>
>> On Thu, Feb 22, 2018 at 4:48 AM, Fawze Abujaber <fawzeaj@gmail.com>
>> wrote:
>>
>>> Hi Guys,
>>>
>>> In Impala version 2.10 i see a lot that the Estimate per Mode peak
>>> memory can reach X30 times of the Per Node Peak Memory usage.
>>>
>>> For example one of the query the estimate was 9GB while the usage was
>>> 30GB.
>>>
>>> I have 3 questions:
>>>
>>> 1- Do you think the compute stats will help here?
>>> 2- How i can reduce this gap?
>>> 3- Does the resource pools based on the Estimate per Mode peak memory?
>>> so if for such query i put the default max memory limit to 5GB, will this
>>> query fail?
>>>
>>>
>>>
>>
>

Mime
View raw message