impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xu, Cheng A" <cheng.a...@intel.com>
Subject RE: performance issue on big table join
Date Fri, 27 Oct 2017 08:25:45 GMT
Append the profile log and it looks probe is taking quite a long time here.

Query (id=2438b42f267ed44:3e1ac3c700000000):
  DEBUG MODE WARNING: Query profile created while running a DEBUG build of Impala. Use RELEASE builds to measure query performance.
  Summary:
    Session ID: 214312dff6aa405d:71a5d38fb87083b2
    Session Type: BEESWAX
    Start Time: 2017-10-27 15:18:32.854201000
    End Time:
    Query Type: QUERY
    Query State: RUNNING
    Query Status: OK
    Impala Version: impalad version 2.11.0-SNAPSHOT DEBUG (build 8ad6d03310825418b6aa1d427200b07bc8bdb0bc)
    User: root
    Connected User: root
    Delegated User:
    Network Address: ::ffff:10.239.47.102:33193
    Default Db: tpcds_bin_partitioned_parquet_100
    Sql Statement: select ss_item_sk, sum(cs_net_profit), sum(ss_net_profit) from
store_sales ss, catalog_sales cs
where ss.ss_item_sk = cs.cs_item_sk and cs.cs_net_profit > ss.ss_net_profit
group by ss_item_sk
order by ss_item_sk
limit 100
    Coordinator: bdpe833n4:22000
    Query Options (set by configuration):
    Query Options (set by configuration and planner): MT_DOP=0
    Plan:
----------------
Max Per-Host Resource Reservation: Memory=35.94MB
Per-Host Resource Estimates: Memory=589.39MB

F03:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
|  Per-Host Resources: mem-estimate=0B mem-reservation=0B
PLAN-ROOT SINK
|  mem-estimate=0B mem-reservation=0B
|
07:MERGING-EXCHANGE [UNPARTITIONED]
|  order by: ss_item_sk ASC
|  limit: 100
|  mem-estimate=0B mem-reservation=0B
|  tuple-ids=3 row-size=24B cardinality=100
|
F02:PLAN FRAGMENT [HASH(ss.ss_item_sk)] hosts=5 instances=5
Per-Host Resources: mem-estimate=493.39MB mem-reservation=35.94MB
04:TOP-N [LIMIT=100]
|  order by: ss_item_sk ASC
|  mem-estimate=2.34KB mem-reservation=0B
|  tuple-ids=3 row-size=24B cardinality=100
|
03:AGGREGATE [FINALIZE]
|  output: sum(cs_net_profit), sum(ss_net_profit)
|  group by: ss_item_sk
|  mem-estimate=10.00MB mem-reservation=1.94MB spill-buffer=64.00KB
|  tuple-ids=2 row-size=24B cardinality=204925
|
02:HASH JOIN [INNER JOIN, PARTITIONED]
|  hash predicates: ss.ss_item_sk = cs.cs_item_sk
|  fk/pk conjuncts: none
|  other predicates: cs.cs_net_profit > ss.ss_net_profit
|  runtime filters: RF000 <- cs.cs_item_sk
|  mem-estimate=483.39MB mem-reservation=34.00MB spill-buffer=2.00MB
|  tuple-ids=0,1 row-size=32B cardinality=202370263193
|
|--06:EXCHANGE [HASH(cs.cs_item_sk)]
|  |  mem-estimate=0B mem-reservation=0B
|  |  tuple-ids=1 row-size=16B cardinality=143997065
|  |
|  F01:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
|  Per-Host Resources: mem-estimate=48.00MB mem-reservation=0B
|  01:SCAN HDFS [tpcds_bin_partitioned_parquet_100.catalog_sales cs, RANDOM]
|     partitions=1837/1837 files=1936 size=17.03GB
|     stats-rows=143997065 extrapolated-rows=disabled
|     table stats: rows=143997065 size=17.03GB
|     column stats: all
|     mem-estimate=48.00MB mem-reservation=0B
|     tuple-ids=1 row-size=16B cardinality=143997065
|
05:EXCHANGE [HASH(ss.ss_item_sk)]
|  mem-estimate=0B mem-reservation=0B
|  tuple-ids=0 row-size=16B cardinality=287997024
|
F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
Per-Host Resources: mem-estimate=48.00MB mem-reservation=0B
00:SCAN HDFS [tpcds_bin_partitioned_parquet_100.store_sales ss, RANDOM]
   partitions=1824/1824 files=1923 size=20.89GB
   runtime filters: RF000 -> ss.ss_item_sk
   stats-rows=287997024 extrapolated-rows=disabled
   table stats: rows=287997024 size=20.89GB
   column stats: all
   mem-estimate=48.00MB mem-reservation=0B
   tuple-ids=0 row-size=16B cardinality=287997024
----------------
    Estimated Per-Host Mem: 618018750
    Per Host Min Reservation: bdpe611n1:22000(35.94 MB) bdpe611n2:22000(35.94 MB) bdpe611n3:22000(35.94 MB) bdpe611n4:22000(35.94 MB) bdpe722n4:22000(35.94 MB) bdpe833n4:22000(0)
    Request Pool: default-pool
    Admission result: Admitted immediately
    Planner Timeline: 106.057ms
       - Analysis finished: 4.687ms (4.687ms)
       - Equivalence classes computed: 5.000ms (313.121us)
       - Single node plan created: 79.940ms (74.939ms)
       - Runtime filters computed: 80.184ms (244.176us)
       - Distributed plan created: 80.437ms (253.115us)
       - Planning finished: 106.057ms (25.620ms)
    Query Timeline: 1h7m
       - Query submitted: 115.266us (115.266us)
       - Planning finished: 207.412ms (207.297ms)
       - Submit for admission: 282.363ms (74.950ms)
       - Completed admission: 282.516ms (152.998us)
       - Ready to start on 6 backends: 318.275ms (35.759ms)
       - All 6 execution backends (16 fragment instances) started: 495.692ms (177.416ms)
       - First dynamic filter received: 21s281ms (20s785ms)
     - ComputeScanRangeAssignmentTimer: 54.075ms
  ImpalaServer:
     - ClientFetchWaitTimer: 0.000ns
     - RowMaterializationTimer: 0.000ns
  Execution Profile 2438b42f267ed44:3e1ac3c700000000:(Total: 195.134ms, non-child: 0.000ns, % non-child: 0.00%)
    Number of filters: 1
    Filter routing table:
 ID  Src. Node  Tgt. Node(s)  Target type  Partition filter  Pending (Expected)  First arrived  Completed   Enabled
-------------------------------------------------------------------------------------------------------------------
  0          2             0       REMOTE             false               5 (5)            N/A        N/A      true
    Backend startup latencies: Count: 6, min / max: 78ms / 131ms, 25th %-ile: 92ms, 50th %-ile: 126ms, 75th %-ile: 128ms, 90th %-ile: 128ms, 95th %-ile: 131ms, 99.9th %-ile: 131ms
     - FiltersReceived: 5 (5)
     - FinalizationTimer: 0.000ns
    Averaged Fragment F03:
    Coordinator Fragment F03:
      Instance 2438b42f267ed44:3e1ac3c700000000 (host=bdpe833n4:22000):(Total: 26.461ms, non-child: 0.000ns, % non-child: 0.00%)
        MemoryUsage(1m4s): 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB
         - AverageThreadTokens: 0.00
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 232.00 KB (237568)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 16.01 MB (16785690)
         - RowsProduced: 0 (0)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 0 (0)
         - TotalThreadsTotalWallClockTime: 0.000ns
           - TotalThreadsSysTime: 0.000ns
           - TotalThreadsUserTime: 0.000ns
         - TotalThreadsVoluntaryContextSwitches: 0 (0)
        Fragment Instance Lifecycle Timings:
           - OpenTime: 0.000ns
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 26.380ms
             - ExecTreePrepareTime: 83.673us
        PLAN_ROOT_SINK:
           - PeakMemoryUsage: 0
        CodeGen:(Total: 40.138ms, non-child: 40.138ms, % non-child: 100.00%)
           - CodegenTime: 2.170ms
           - CompileTime: 2.239ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 29 (29)
           - NumInstructions: 448 (448)
           - OptimizationTime: 11.294ms
           - PeakMemoryUsage: 224.00 KB (229376)
           - PrepareTime: 25.840ms
        EXCHANGE_NODE (id=7):
          ExecOption: Codegen Enabled
           - ConvertRowBatchTime: 0.000ns
           - MergeGetNext: 0.000ns
           - MergeGetNextBatch: 0.000ns
           - PeakMemoryUsage: 0
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
          DataStreamReceiver:
            BytesReceived(1m4s): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
             - BytesReceived: 0
             - DeserializeRowBatchTimer: 0.000ns
             - FirstBatchArrivalWaitTime: 0.000ns
             - PeakMemoryUsage: 0
             - SendersBlockedTimer: 0.000ns
             - SendersBlockedTotalTimer(*): 0.000ns
    Averaged Fragment F02:
    Fragment F02:
      Instance 2438b42f267ed44:3e1ac3c70000000e (host=bdpe611n1:22000):(Total: 60.735ms, non-child: 0.000ns, % non-child: 0.00%)
        MemoryUsage(1m4s): 1.51 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB
        ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.01
         - BloomFilterBytes: 16.00 MB (16777216)
         - PeakMemoryUsage: 2.15 GB (2303872045)
         - PeakReservation: 2.12 GB (2274426880)
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 2.36 GB (2530721031)
         - RowsProduced: 0 (0)
         - TotalNetworkReceiveTime: 10s323ms
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 74 (74)
         - TotalThreadsTotalWallClockTime: 23s913ms
           - TotalThreadsSysTime: 1s022ms
           - TotalThreadsUserTime: 13s752ms
         - TotalThreadsVoluntaryContextSwitches: 17.21K (17214)
        Fragment Instance Lifecycle Timings:
           - OpenTime: 0.000ns
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 60.675ms
             - ExecTreePrepareTime: 1.038ms
        DataStreamSender (dst_id=7):(Total: 277.868us, non-child: 277.868us, % non-child: 100.00%)
           - BytesSent: 0
           - NetworkThroughput(*): 0.00 /sec
           - OverallThroughput: 0.00 /sec
           - PeakMemoryUsage: 5.12 KB (5240)
           - RowsReturned: 0 (0)
           - SerializeBatchTime: 0.000ns
           - TransmitDataRPCTime: 0.000ns
           - UncompressedRowBatchSize: 0
        CodeGen:(Total: 403.767ms, non-child: 403.767ms, % non-child: 100.00%)
           - CodegenTime: 7.281ms
           - CompileTime: 25.928ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 149 (149)
           - NumInstructions: 4.39K (4394)
           - OptimizationTime: 338.848ms
           - PeakMemoryUsage: 2.15 MB (2249728)
           - PrepareTime: 37.870ms
        SORT_NODE (id=4):(Total: 1.036ms, non-child: 113.376us, % non-child: 10.93%)
          ExecOption: Codegen Enabled
           - InsertBatchTime: 0.000ns
           - PeakMemoryUsage: 4.00 KB (4096)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - TuplePoolReclamations: 0 (0)
        AGGREGATION_NODE (id=3):(Total: 923.490us, non-child: 0.000ns, % non-child: 0.00%)
          ExecOption: Codegen Enabled
           - BuildTime: 7m9s
           - GetResultsTime: 0.000ns
           - HTResizeTime: 9s233ms
           - HashBuckets: 0 (0)
          - LargestPartitionPercent: 0 (0)
           - MaxPartitionLevel: 0 (0)
           - NumRepartitions: 0 (0)
           - PartitionsCreated: 16 (16)
           - PeakMemoryUsage: 3.13 MB (3277926)
           - RowsRepartitioned: 0 (0)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - SpilledPartitions: 0 (0)
        HASH_JOIN_NODE (id=2):(Total: 59m41s, non-child: 59m25s, % non-child: 99.56%)
          ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
           - BuildRows: 28.78M (28780361)
           - BuildTime: 11s291ms
           - NumHashTableBuildsSkipped: 0 (0)
           - PeakMemoryUsage: 2.12 GB (2271291622)
           - ProbeRows: 19.04M (19038208)
           - ProbeRowsPartitioned: 0 (0)
           - ProbeTime: 59m6s
           - RowsReturned: 9.19B (9192516608)
           - RowsReturnedRate: 2.57 M/sec
          Buffer pool:
             - AllocTime: 41.602ms
             - CumulativeAllocationBytes: 2.12 GB (2271215616)
             - CumulativeAllocations: 587 (587)
             - PeakReservation: 2.12 GB (2271215616)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 2.12 GB (2271215616)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Hash Join Builder (join_node_id=2):
            ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled
            Runtime filters: 1 of 1 Runtime Filter Published
             - BuildRowsPartitionTime: 7s193ms
             - BuildRowsPartitioned: 28.78M (28780361)
             - HashBuckets: 67.11M (67108864)
             - HashCollisions: 0 (0)
             - HashTablesBuildTime: 4s069ms
             - LargestPartitionPercent: 6 (6)
             - MaxPartitionLevel: 0 (0)
             - NumRepartitions: 0 (0)
             - PartitionsCreated: 16 (16)
             - PeakMemoryUsage: 21.12 KB (21632)
             - RepartitionTime: 0.000ns
             - SpilledPartitions: 0 (0)
          EXCHANGE_NODE (id=6):(Total: 12s499ms, non-child: 12s499ms, % non-child: 100.00%)
             - ConvertRowBatchTime: 2s961ms
             - PeakMemoryUsage: 0
             - RowsReturned: 28.78M (28780361)
             - RowsReturnedRate: 2.30 M/sec
            DataStreamReceiver:
              BytesReceived(500.000ms): 4.83 MB, 13.21 MB, 20.00 MB, 25.94 MB, 34.47 MB, 43.07 MB, 53.29 MB, 60.58 MB, 69.64 MB, 79.96 MB, 90.95 MB, 102.65 MB, 114.45 MB, 126.38 MB, 138.11 MB, 149.89 MB, 160.54 MB, 171.15 MB, 183.00 MB, 194.76 MB, 206.61 MB, 218.42 MB, 230.30 MB, 241.95 MB, 252.93 MB, 263.84 MB, 275.62 MB, 287.39 MB, 299.12 MB, 310.70 MB, 322.34 MB, 333.89 MB, 345.54 MB, 357.32 MB, 368.84 MB, 378.03 MB, 387.25 MB, 395.88 MB, 400.75 MB, 403.03 MB, 404.79 MB, 404.79 MB, 404.79 MB, 404.79 MB, 404.79 MB, 404.79 MB, 404.79 MB, 404.79 MB
               - BytesReceived: 404.79 MB (424455941)
               - DeserializeRowBatchTimer: 1s093ms
               - FirstBatchArrivalWaitTime: 0.000ns
               - PeakMemoryUsage: 10.08 MB (10568925)
               - SendersBlockedTimer: 0.000ns
               - SendersBlockedTotalTimer(*): 0.000ns
        EXCHANGE_NODE (id=5):(Total: 3s449ms, non-child: 3s449ms, % non-child: 100.00%)
           - ConvertRowBatchTime: 2s103ms
           - PeakMemoryUsage: 0
           - RowsReturned: 19.04M (19038208)
           - RowsReturnedRate: 5.52 M/sec
          DataStreamReceiver:
            BytesReceived(1m4s): 7.95 MB, 11.14 MB, 15.32 MB, 18.94 MB, 22.47 MB, 26.87 MB, 32.41 MB, 37.89 MB, 42.43 MB, 46.75 MB, 51.04 MB, 55.32 MB, 59.45 MB, 63.75 MB, 67.99 MB, 72.17 MB, 76.59 MB, 80.87 MB, 84.89 MB, 89.18 MB, 93.54 MB, 97.90 MB, 102.21 MB, 106.48 MB, 110.87 MB, 115.28 MB, 119.65 MB, 124.01 MB, 128.33 MB, 132.74 MB, 137.14 MB, 141.41 MB, 145.62 MB, 149.66 MB, 154.18 MB, 159.13 MB, 164.06 MB, 168.96 MB, 173.81 MB, 178.62 MB, 183.34 MB, 187.96 MB, 192.59 MB, 197.31 MB, 201.98 MB, 206.65 MB, 211.33 MB, 215.70 MB, 220.12 MB, 224.81 MB, 229.49 MB, 234.18 MB, 238.89 MB, 243.60 MB, 248.33 MB, 253.07 MB, 257.80 MB, 262.53 MB, 267.27 MB, 272.02 MB, 276.27 MB, 280.55 MB, 284.87 MB
             - BytesReceived: 288.64 MB (302657277)
             - DeserializeRowBatchTimer: 1s534ms
             - FirstBatchArrivalWaitTime: 647.613ms
             - PeakMemoryUsage: 11.92 MB (12494925)
             - SendersBlockedTimer: 1h
             - SendersBlockedTotalTimer(*): 4h58m
          Buffer pool:
             - AllocTime: 677.240us
             - CumulativeAllocationBytes: 4.62 MB (4849664)
             - CumulativeAllocations: 58 (58)
             - PeakReservation: 3.06 MB (3211264)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 3.06 MB (3211264)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
      Instance 2438b42f267ed44:3e1ac3c70000000c (host=bdpe611n2:22000):(Total: 60.385ms, non-child: 0.000ns, % non-child: 0.00%)
        MemoryUsage(1m4s): 1.51 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB
        ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.01
         - BloomFilterBytes: 16.00 MB (16777216)
         - PeakMemoryUsage: 2.15 GB (2308066349)
         - PeakReservation: 2.12 GB (2278621184)
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 2.35 GB (2523002980)
         - RowsProduced: 0 (0)
         - TotalNetworkReceiveTime: 4m28s
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 46 (46)
         - TotalThreadsTotalWallClockTime: 23s375ms
           - TotalThreadsSysTime: 830.830ms
           - TotalThreadsUserTime: 12s866ms
         - TotalThreadsVoluntaryContextSwitches: 17.74K (17740)
        Fragment Instance Lifecycle Timings:
           - OpenTime: 0.000ns
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 60.246ms
             - ExecTreePrepareTime: 1.008ms
        DataStreamSender (dst_id=7):(Total: 338.446us, non-child: 338.446us, % non-child: 100.00%)
           - BytesSent: 0
           - NetworkThroughput(*): 0.00 /sec
           - OverallThroughput: 0.00 /sec
           - PeakMemoryUsage: 5.12 KB (5240)
           - RowsReturned: 0 (0)
           - SerializeBatchTime: 0.000ns
           - TransmitDataRPCTime: 0.000ns
           - UncompressedRowBatchSize: 0
        CodeGen:(Total: 418.144ms, non-child: 418.144ms, % non-child: 100.00%)
           - CodegenTime: 6.477ms
           - CompileTime: 26.204ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 149 (149)
           - NumInstructions: 4.39K (4394)
           - OptimizationTime: 353.902ms
           - PeakMemoryUsage: 2.15 MB (2249728)
           - PrepareTime: 36.915ms
        SORT_NODE (id=4):(Total: 1.007ms, non-child: 94.499us, % non-child: 9.38%)
          ExecOption: Codegen Enabled
           - InsertBatchTime: 0.000ns
           - PeakMemoryUsage: 4.00 KB (4096)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - TuplePoolReclamations: 0 (0)
        AGGREGATION_NODE (id=3):(Total: 912.507us, non-child: 0.000ns, % non-child: 0.00%)
          ExecOption: Codegen Enabled
           - BuildTime: 7m55s
           - GetResultsTime: 0.000ns
           - HTResizeTime: 9s948ms
           - HashBuckets: 0 (0)
           - LargestPartitionPercent: 0 (0)
           - MaxPartitionLevel: 0 (0)
           - NumRepartitions: 0 (0)
           - PartitionsCreated: 16 (16)
           - PeakMemoryUsage: 3.13 MB (3277926)
           - RowsRepartitioned: 0 (0)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - SpilledPartitions: 0 (0)
        HASH_JOIN_NODE (id=2):(Total: 58m52s, non-child: 54m19s, % non-child: 92.26%)
          ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
           - BuildRows: 28.94M (28939765)
           - BuildTime: 10s085ms
           - NumHashTableBuildsSkipped: 0 (0)
           - PeakMemoryUsage: 2.12 GB (2275485926)
           - ProbeRows: 19.66M (19658752)
           - ProbeRowsPartitioned: 0 (0)
           - ProbeTime: 54m
           - RowsReturned: 9.48B (9482271744)
           - RowsReturnedRate: 2.68 M/sec
          Buffer pool:
             - AllocTime: 34.354ms
             - CumulativeAllocationBytes: 2.12 GB (2275409920)
             - CumulativeAllocations: 589 (589)
             - PeakReservation: 2.12 GB (2275409920)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 2.12 GB (2275409920)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Hash Join Builder (join_node_id=2):
            ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled
            Runtime filters: 1 of 1 Runtime Filter Published
             - BuildRowsPartitionTime: 6s484ms
             - BuildRowsPartitioned: 28.94M (28939765)
             - HashBuckets: 67.11M (67108864)
             - HashCollisions: 0 (0)
             - HashTablesBuildTime: 3s570ms
             - LargestPartitionPercent: 6 (6)
             - MaxPartitionLevel: 0 (0)
             - NumRepartitions: 0 (0)
             - PartitionsCreated: 16 (16)
             - PeakMemoryUsage: 21.12 KB (21632)
             - RepartitionTime: 0.000ns
             - SpilledPartitions: 0 (0)
          EXCHANGE_NODE (id=6):(Total: 13s166ms, non-child: 13s166ms, % non-child: 100.00%)
             - ConvertRowBatchTime: 3s054ms
             - PeakMemoryUsage: 0
             - RowsReturned: 28.94M (28939765)
             - RowsReturnedRate: 2.20 M/sec
            DataStreamReceiver:
              BytesReceived(500.000ms): 0, 8.31 MB, 16.07 MB, 21.92 MB, 29.09 MB, 36.98 MB, 46.76 MB, 55.98 MB, 63.87 MB, 73.19 MB, 83.86 MB, 95.62 MB, 107.31 MB, 119.33 MB, 131.28 MB, 142.99 MB, 154.89 MB, 164.96 MB, 176.31 MB, 188.16 MB, 200.09 MB, 211.93 MB, 223.91 MB, 235.83 MB, 247.31 MB, 257.80 MB, 269.44 MB, 281.31 MB, 293.16 MB, 304.77 MB, 316.58 MB, 328.36 MB, 339.80 MB, 351.73 MB, 363.51 MB, 374.18 MB, 383.41 MB, 392.72 MB, 400.63 MB, 403.84 MB, 406.17 MB, 407.11 MB, 407.11 MB, 407.11 MB, 407.11 MB, 407.11 MB, 407.11 MB, 407.11 MB
               - BytesReceived: 407.11 MB (426884473)
               - DeserializeRowBatchTimer: 1s128ms
               - FirstBatchArrivalWaitTime: 0.000ns
               - PeakMemoryUsage: 11.76 MB (12326400)
               - SendersBlockedTimer: 0.000ns
               - SendersBlockedTotalTimer(*): 0.000ns
        EXCHANGE_NODE (id=5):(Total: 4m20s, non-child: 4m20s, % non-child: 100.00%)
           - ConvertRowBatchTime: 2s176ms
           - PeakMemoryUsage: 0
          - RowsReturned: 19.66M (19658752)
           - RowsReturnedRate: 75.42 K/sec
          DataStreamReceiver:
            BytesReceived(1m4s): 7.91 MB, 11.14 MB, 15.37 MB, 19.01 MB, 22.56 MB, 26.95 MB, 32.53 MB, 38.07 MB, 42.66 MB, 46.99 MB, 51.30 MB, 55.62 MB, 59.75 MB, 64.07 MB, 68.35 MB, 72.55 MB, 76.99 MB, 81.31 MB, 85.35 MB, 89.68 MB, 94.06 MB, 98.47 MB, 102.79 MB, 107.08 MB, 111.48 MB, 115.92 MB, 120.33 MB, 124.71 MB, 129.05 MB, 133.48 MB, 137.90 MB, 142.18 MB, 146.39 MB, 150.45 MB, 154.98 MB, 159.94 MB, 164.88 MB, 169.81 MB, 174.68 MB, 179.51 MB, 184.25 MB, 188.90 MB, 193.57 MB, 198.35 MB, 203.05 MB, 207.74 MB, 212.47 MB, 216.86 MB, 221.31 MB, 226.05 MB, 230.75 MB, 235.47 MB, 240.20 MB, 244.97 MB, 249.73 MB, 254.48 MB, 259.22 MB, 263.98 MB, 268.74 MB, 273.53 MB, 277.81 MB, 282.12 MB, 286.47 MB
             - BytesReceived: 290.43 MB (304536877)
             - DeserializeRowBatchTimer: 1s880ms
             - FirstBatchArrivalWaitTime: 611.434ms
             - PeakMemoryUsage: 11.92 MB (12494925)
             - SendersBlockedTimer: 2m23s
             - SendersBlockedTotalTimer(*): 11m9s
          Buffer pool:
             - AllocTime: 740.683us
             - CumulativeAllocationBytes: 4.62 MB (4849664)
             - CumulativeAllocations: 58 (58)
             - PeakReservation: 3.06 MB (3211264)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 3.06 MB (3211264)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
      Instance 2438b42f267ed44:3e1ac3c70000000b (host=bdpe611n3:22000):(Total: 65.589ms, non-child: 0.000ns, % non-child: 0.00%)
        MemoryUsage(1m4s): 1.50 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB
        ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.01
         - BloomFilterBytes: 16.00 MB (16777216)
         - PeakMemoryUsage: 2.14 GB (2296330746)
         - PeakReservation: 2.11 GB (2268135424)
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 2.33 GB (2505825451)
         - RowsProduced: 0 (0)
         - TotalNetworkReceiveTime: 3m38s
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 24 (24)
         - TotalThreadsTotalWallClockTime: 23s506ms
           - TotalThreadsSysTime: 848.021ms
           - TotalThreadsUserTime: 12s669ms
         - TotalThreadsVoluntaryContextSwitches: 18.45K (18448)
        Fragment Instance Lifecycle Timings:
           - OpenTime: 0.000ns
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 65.405ms
             - ExecTreePrepareTime: 951.512us
        DataStreamSender (dst_id=7):(Total: 280.611us, non-child: 280.611us, % non-child: 100.00%)
           - BytesSent: 0
           - NetworkThroughput(*): 0.00 /sec
           - OverallThroughput: 0.00 /sec
           - PeakMemoryUsage: 5.12 KB (5240)
           - RowsReturned: 0 (0)
           - SerializeBatchTime: 0.000ns
           - TransmitDataRPCTime: 0.000ns
           - UncompressedRowBatchSize: 0
        CodeGen:(Total: 401.447ms, non-child: 401.447ms, % non-child: 100.00%)
           - CodegenTime: 8.074ms
           - CompileTime: 25.532ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 149 (149)
           - NumInstructions: 4.39K (4394)
           - OptimizationTime: 335.130ms
           - PeakMemoryUsage: 2.15 MB (2249728)
           - PrepareTime: 39.643ms
        SORT_NODE (id=4):(Total: 949.746us, non-child: 96.968us, % non-child: 10.21%)
          ExecOption: Codegen Enabled
           - InsertBatchTime: 0.000ns
           - PeakMemoryUsage: 4.00 KB (4096)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - TuplePoolReclamations: 0 (0)
        AGGREGATION_NODE (id=3):(Total: 852.778us, non-child: 0.000ns, % non-child: 0.00%)
          ExecOption: Codegen Enabled
           - BuildTime: 7m48s
           - GetResultsTime: 0.000ns
           - HTResizeTime: 9s829ms
           - HashBuckets: 0 (0)
           - LargestPartitionPercent: 0 (0)
           - MaxPartitionLevel: 0 (0)
           - NumRepartitions: 0 (0)
           - PartitionsCreated: 16 (16)
           - PeakMemoryUsage: 3.13 MB (3277926)
           - RowsRepartitioned: 0 (0)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - SpilledPartitions: 0 (0)
        HASH_JOIN_NODE (id=2):(Total: 58m59s, non-child: 55m16s, % non-child: 93.69%)
          ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
           - BuildRows: 28.62M (28618322)
           - BuildTime: 10s036ms
           - NumHashTableBuildsSkipped: 0 (0)
           - PeakMemoryUsage: 2.11 GB (2265000166)
           - ProbeRows: 19.43M (19430400)
           - ProbeRowsPartitioned: 0 (0)
           - ProbeTime: 54m58s
           - RowsReturned: 9.37B (9368660992)
           - RowsReturnedRate: 2.65 M/sec
          Buffer pool:
             - AllocTime: 26.074ms
             - CumulativeAllocationBytes: 2.11 GB (2264924160)
             - CumulativeAllocations: 584 (584)
             - PeakReservation: 2.11 GB (2264924160)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 2.11 GB (2264924160)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Hash Join Builder (join_node_id=2):
            ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled
            Runtime filters: 1 of 1 Runtime Filter Published
             - BuildRowsPartitionTime: 6s314ms
             - BuildRowsPartitioned: 28.62M (28618322)
             - HashBuckets: 67.11M (67108864)
             - HashCollisions: 0 (0)
             - HashTablesBuildTime: 3s687ms
             - LargestPartitionPercent: 6 (6)
             - MaxPartitionLevel: 0 (0)
             - NumRepartitions: 0 (0)
             - PartitionsCreated: 16 (16)
             - PeakMemoryUsage: 21.12 KB (21632)
             - RepartitionTime: 0.000ns
             - SpilledPartitions: 0 (0)
          EXCHANGE_NODE (id=6):(Total: 13s349ms, non-child: 13s349ms, % non-child: 100.00%)
             - ConvertRowBatchTime: 2s954ms
             - PeakMemoryUsage: 0
             - RowsReturned: 28.62M (28618322)
             - RowsReturnedRate: 2.14 M/sec
            DataStreamReceiver:
              BytesReceived(500.000ms): 2.51 MB, 11.25 MB, 18.78 MB, 23.82 MB, 32.00 MB, 40.32 MB, 50.11 MB, 58.46 MB, 66.92 MB, 76.52 MB, 87.38 MB, 99.16 MB, 110.70 MB, 122.51 MB, 134.29 MB, 145.95 MB, 157.06 MB, 167.11 MB, 178.85 MB, 190.56 MB, 202.38 MB, 214.10 MB, 225.90 MB, 237.57 MB, 248.62 MB, 259.30 MB, 271.01 MB, 282.75 MB, 294.46 MB, 305.97 MB, 317.48 MB, 329.01 MB, 340.59 MB, 352.31 MB, 363.98 MB, 373.54 MB, 382.68 MB, 391.87 MB, 397.94 MB, 400.22 MB, 402.55 MB, 402.56 MB, 402.56 MB, 402.56 MB, 402.56 MB, 402.56 MB, 402.56 MB, 402.56 MB
               - BytesReceived: 402.56 MB (422114759)
               - DeserializeRowBatchTimer: 1s198ms
               - FirstBatchArrivalWaitTime: 0.000ns
               - PeakMemoryUsage: 11.20 MB (11748600)
               - SendersBlockedTimer: 0.000ns
               - SendersBlockedTotalTimer(*): 0.000ns
        EXCHANGE_NODE (id=5):(Total: 3m30s, non-child: 3m30s, % non-child: 100.00%)
           - ConvertRowBatchTime: 2s142ms
           - PeakMemoryUsage: 0
           - RowsReturned: 19.43M (19431414)
           - RowsReturnedRate: 92.42 K/sec
          DataStreamReceiver:
            BytesReceived(1m4s): 7.86 MB, 11.07 MB, 15.21 MB, 18.81 MB, 22.33 MB, 26.69 MB, 32.22 MB, 37.69 MB, 42.22 MB, 46.49 MB, 50.75 MB, 55.01 MB, 59.10 MB, 63.39 MB, 67.63 MB, 71.77 MB, 76.16 MB, 80.43 MB, 84.41 MB, 88.70 MB, 93.02 MB, 97.37 MB, 101.64 MB, 105.88 MB, 110.22 MB, 114.59 MB, 118.93 MB, 123.26 MB, 127.55 MB, 131.96 MB, 136.33 MB, 140.60 MB, 144.78 MB, 148.79 MB, 153.29 MB, 158.21 MB, 163.08 MB, 167.95 MB, 172.76 MB, 177.55 MB, 182.25 MB, 186.84 MB, 191.45 MB, 196.16 MB, 200.84 MB, 205.50 MB, 210.16 MB, 214.50 MB, 218.89 MB, 223.58 MB, 228.22 MB, 232.87 MB, 237.56 MB, 242.26 MB, 246.97 MB, 251.68 MB, 256.38 MB, 261.09 MB, 265.81 MB, 270.55 MB, 274.74 MB, 279.01 MB, 283.31 MB
             - BytesReceived: 287.04 MB (300979168)
             - DeserializeRowBatchTimer: 1s831ms
             - FirstBatchArrivalWaitTime: 621.536ms
             - PeakMemoryUsage: 11.87 MB (12446775)
             - SendersBlockedTimer: 0.000ns
             - SendersBlockedTotalTimer(*): 0.000ns
          Buffer pool:
             - AllocTime: 677.596us
             - CumulativeAllocationBytes: 4.62 MB (4849664)
             - CumulativeAllocations: 58 (58)
             - PeakReservation: 3.06 MB (3211264)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 3.06 MB (3211264)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
      Instance 2438b42f267ed44:3e1ac3c70000000f (host=bdpe611n4:22000):(Total: 51.527ms, non-child: 0.000ns, % non-child: 0.00%)
        MemoryUsage(1m4s): 1.50 GB, 2.15 GB, 2.16 GB, 2.16 GB, 2.16 GB, 2.16 GB, 2.16 GB, 2.16 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB
        ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.01
         - BloomFilterBytes: 16.00 MB (16777216)
         - PeakMemoryUsage: 2.16 GB (2314423341)
         - PeakReservation: 2.13 GB (2284978176)
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 2.34 GB (2515248771)
         - RowsProduced: 0 (0)
         - TotalNetworkReceiveTime: 5m3s
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 47 (47)
         - TotalThreadsTotalWallClockTime: 23s986ms
           - TotalThreadsSysTime: 930.342ms
           - TotalThreadsUserTime: 13s892ms
         - TotalThreadsVoluntaryContextSwitches: 17.42K (17421)
        Fragment Instance Lifecycle Timings:
           - OpenTime: 0.000ns
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 51.455ms
             - ExecTreePrepareTime: 875.534us
        DataStreamSender (dst_id=7):(Total: 287.278us, non-child: 287.278us, % non-child: 100.00%)
           - BytesSent: 0
           - NetworkThroughput(*): 0.00 /sec
           - OverallThroughput: 0.00 /sec
           - PeakMemoryUsage: 5.12 KB (5240)
           - RowsReturned: 0 (0)
           - SerializeBatchTime: 0.000ns
           - TransmitDataRPCTime: 0.000ns
           - UncompressedRowBatchSize: 0
        CodeGen:(Total: 406.407ms, non-child: 406.407ms, % non-child: 100.00%)
           - CodegenTime: 6.204ms
           - CompileTime: 25.846ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 149 (149)
           - NumInstructions: 4.39K (4394)
           - OptimizationTime: 347.117ms
           - PeakMemoryUsage: 2.15 MB (2249728)
           - PrepareTime: 32.535ms
        SORT_NODE (id=4):(Total: 859.398us, non-child: 119.435us, % non-child: 13.90%)
          ExecOption: Codegen Enabled
           - InsertBatchTime: 0.000ns
           - PeakMemoryUsage: 4.00 KB (4096)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - TuplePoolReclamations: 0 (0)
        AGGREGATION_NODE (id=3):(Total: 739.963us, non-child: 0.000ns, % non-child: 0.00%)
          ExecOption: Codegen Enabled
           - BuildTime: 7m49s
           - GetResultsTime: 0.000ns
           - HTResizeTime: 9s866ms
           - HashBuckets: 0 (0)
           - LargestPartitionPercent: 0 (0)
           - MaxPartitionLevel: 0 (0)
           - NumRepartitions: 0 (0)
           - PartitionsCreated: 16 (16)
           - PeakMemoryUsage: 3.19 MB (3343462)
           - RowsRepartitioned: 0 (0)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - SpilledPartitions: 0 (0)
        HASH_JOIN_NODE (id=2):(Total: 58m59s, non-child: 53m50s, % non-child: 91.27%)
          ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
           - BuildRows: 29.07M (29066489)
           - BuildTime: 11s309ms
           - NumHashTableBuildsSkipped: 0 (0)
           - PeakMemoryUsage: 2.13 GB (2281777382)
           - ProbeRows: 19.74M (19742720)
           - ProbeRowsPartitioned: 0 (0)
           - ProbeTime: 53m31s
           - RowsReturned: 9.52B (9516909568)
           - RowsReturnedRate: 2.69 M/sec
          Buffer pool:
             - AllocTime: 43.372ms
             - CumulativeAllocationBytes: 2.12 GB (2281701376)
             - CumulativeAllocations: 592 (592)
             - PeakReservation: 2.12 GB (2281701376)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 2.12 GB (2281701376)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Hash Join Builder (join_node_id=2):
            ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled
            Runtime filters: 1 of 1 Runtime Filter Published
             - BuildRowsPartitionTime: 7s172ms
             - BuildRowsPartitioned: 29.07M (29066489)
             - HashBuckets: 67.11M (67108864)
             - HashCollisions: 0 (0)
             - HashTablesBuildTime: 4s108ms
             - LargestPartitionPercent: 6 (6)
             - MaxPartitionLevel: 0 (0)
             - NumRepartitions: 0 (0)
             - PartitionsCreated: 16 (16)
             - PeakMemoryUsage: 21.12 KB (21632)
             - RepartitionTime: 0.000ns
             - SpilledPartitions: 0 (0)
          EXCHANGE_NODE (id=6):(Total: 12s559ms, non-child: 12s559ms, % non-child: 100.00%)
             - ConvertRowBatchTime: 2s958ms
             - PeakMemoryUsage: 0
             - RowsReturned: 29.07M (29066489)
             - RowsReturnedRate: 2.31 M/sec
            DataStreamReceiver:
              BytesReceived(500.000ms): 0, 6.96 MB, 14.73 MB, 21.22 MB, 28.25 MB, 35.79 MB, 45.44 MB, 55.30 MB, 62.80 MB, 72.11 MB, 82.67 MB, 94.24 MB, 106.02 MB, 117.98 MB, 130.01 MB, 141.71 MB, 153.75 MB, 164.15 MB, 175.18 MB, 187.17 MB, 199.08 MB, 210.99 MB, 222.96 MB, 235.02 MB, 246.78 MB, 257.52 MB, 268.85 MB, 280.75 MB, 292.65 MB, 304.29 MB, 316.18 MB, 328.03 MB, 339.47 MB, 351.43 MB, 363.32 MB, 374.42 MB, 383.69 MB, 393.01 MB, 401.27 MB, 405.23 MB, 407.56 MB, 408.86 MB, 408.86 MB, 408.86 MB, 408.86 MB, 408.86 MB, 408.86 MB, 408.86 MB, 408.86 MB
               - BytesReceived: 408.86 MB (428719786)
               - DeserializeRowBatchTimer: 1s039ms
               - FirstBatchArrivalWaitTime: 0.000ns
               - PeakMemoryUsage: 9.16 MB (9605925)
               - SendersBlockedTimer: 0.000ns
               - SendersBlockedTotalTimer(*): 0.000ns
        EXCHANGE_NODE (id=5):(Total: 4m56s, non-child: 4m56s, % non-child: 100.00%)
           - ConvertRowBatchTime: 2s194ms
           - PeakMemoryUsage: 0
           - RowsReturned: 19.74M (19742720)
           - RowsReturnedRate: 66.51 K/sec
          DataStreamReceiver:
            BytesReceived(1m4s): 7.95 MB, 11.24 MB, 15.42 MB, 19.05 MB, 22.62 MB, 27.06 MB, 32.66 MB, 38.24 MB, 42.86 MB, 47.22 MB, 51.55 MB, 55.88 MB, 60.06 MB, 64.38 MB, 68.68 MB, 72.89 MB, 77.33 MB, 81.66 MB, 85.73 MB, 90.08 MB, 94.49 MB, 98.90 MB, 103.25 MB, 107.56 MB, 111.98 MB, 116.40 MB, 120.83 MB, 125.23 MB, 129.58 MB, 134.04 MB, 138.47 MB, 142.79 MB, 147.04 MB, 151.12 MB, 155.68 MB, 160.66 MB, 165.62 MB, 170.58 MB, 175.48 MB, 180.33 MB, 185.10 MB, 189.79 MB, 194.46 MB, 199.24 MB, 203.96 MB, 208.67 MB, 213.39 MB, 217.78 MB, 222.25 MB, 227.00 MB, 231.75 MB, 236.47 MB, 241.24 MB, 246.00 MB, 250.79 MB, 255.57 MB, 260.35 MB, 265.13 MB, 269.91 MB, 274.72 MB, 279.00 MB, 283.33 MB, 287.69 MB
             - BytesReceived: 291.66 MB (305826020)
             - DeserializeRowBatchTimer: 1s758ms
             - FirstBatchArrivalWaitTime: 684.861ms
             - PeakMemoryUsage: 11.92 MB (12494925)
             - SendersBlockedTimer: 6m37s
             - SendersBlockedTotalTimer(*): 32m12s
          Buffer pool:
             - AllocTime: 528.670us
             - CumulativeAllocationBytes: 4.69 MB (4915200)
             - CumulativeAllocations: 59 (59)
             - PeakReservation: 3.12 MB (3276800)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 3.12 MB (3276800)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
      Instance 2438b42f267ed44:3e1ac3c70000000d (host=bdpe722n4:22000):(Total: 60.833ms, non-child: 0.000ns, % non-child: 0.00%)
        MemoryUsage(1m4s): 1.51 GB, 2.14 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB
        ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.01
         - BloomFilterBytes: 16.00 MB (16777216)
         - PeakMemoryUsage: 2.14 GB (2296402971)
         - PeakReservation: 2.11 GB (2268135424)
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 2.34 GB (2507859104)
         - RowsProduced: 0 (0)
         - TotalNetworkReceiveTime: 7m2s
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 54 (54)
         - TotalThreadsTotalWallClockTime: 23s068ms
           - TotalThreadsSysTime: 648.538ms
           - TotalThreadsUserTime: 12s929ms
         - TotalThreadsVoluntaryContextSwitches: 17.43K (17431)
        Fragment Instance Lifecycle Timings:
           - OpenTime: 0.000ns
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 60.685ms
             - ExecTreePrepareTime: 893.957us
        DataStreamSender (dst_id=7):(Total: 278.197us, non-child: 278.197us, % non-child: 100.00%)
           - BytesSent: 0
           - NetworkThroughput(*): 0.00 /sec
           - OverallThroughput: 0.00 /sec
           - PeakMemoryUsage: 5.12 KB (5240)
           - RowsReturned: 0 (0)
           - SerializeBatchTime: 0.000ns
           - TransmitDataRPCTime: 0.000ns
           - UncompressedRowBatchSize: 0
        CodeGen:(Total: 407.057ms, non-child: 407.057ms, % non-child: 100.00%)
           - CodegenTime: 7.258ms
           - CompileTime: 26.797ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 149 (149)
           - NumInstructions: 4.39K (4394)
           - OptimizationTime: 342.023ms
           - PeakMemoryUsage: 2.15 MB (2249728)
           - PrepareTime: 36.965ms
        SORT_NODE (id=4):(Total: 892.289us, non-child: 84.246us, % non-child: 9.44%)
          ExecOption: Codegen Enabled
           - InsertBatchTime: 0.000ns
           - PeakMemoryUsage: 4.00 KB (4096)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - TuplePoolReclamations: 0 (0)
        AGGREGATION_NODE (id=3):(Total: 808.043us, non-child: 0.000ns, % non-child: 0.00%)
          ExecOption: Codegen Enabled
           - BuildTime: 7m38s
           - GetResultsTime: 0.000ns
           - HTResizeTime: 9s600ms
           - HashBuckets: 0 (0)
           - LargestPartitionPercent: 0 (0)
           - MaxPartitionLevel: 0 (0)
           - NumRepartitions: 0 (0)
           - PartitionsCreated: 16 (16)
           - PeakMemoryUsage: 3.13 MB (3277926)
           - RowsRepartitioned: 0 (0)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - SpilledPartitions: 0 (0)
        HASH_JOIN_NODE (id=2):(Total: 59m13s, non-child: 52m4s, % non-child: 87.95%)
          ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
           - BuildRows: 28.59M (28592128)
           - BuildTime: 9s937ms
           - NumHashTableBuildsSkipped: 0 (0)
           - PeakMemoryUsage: 2.11 GB (2265000166)
           - ProbeRows: 19.42M (19423232)
           - ProbeRowsPartitioned: 0 (0)
           - ProbeTime: 51m47s
           - RowsReturned: 9.36B (9362539520)
           - RowsReturnedRate: 2.64 M/sec
          Buffer pool:
             - AllocTime: 9.410ms
             - CumulativeAllocationBytes: 2.11 GB (2264924160)
             - CumulativeAllocations: 584 (584)
             - PeakReservation: 2.11 GB (2264924160)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 2.11 GB (2264924160)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Hash Join Builder (join_node_id=2):
            ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled
            Runtime filters: 1 of 1 Runtime Filter Published
             - BuildRowsPartitionTime: 6s656ms
             - BuildRowsPartitioned: 28.59M (28592128)
             - HashBuckets: 67.11M (67108864)
             - HashCollisions: 0 (0)
             - HashTablesBuildTime: 3s246ms
             - LargestPartitionPercent: 6 (6)
             - MaxPartitionLevel: 0 (0)
             - NumRepartitions: 0 (0)
             - PartitionsCreated: 16 (16)
             - PeakMemoryUsage: 21.12 KB (21632)
             - RepartitionTime: 0.000ns
             - SpilledPartitions: 0 (0)
          EXCHANGE_NODE (id=6):(Total: 13s011ms, non-child: 13s011ms, % non-child: 100.00%)
             - ConvertRowBatchTime: 3s049ms
             - PeakMemoryUsage: 0
             - RowsReturned: 28.59M (28592128)
             - RowsReturnedRate: 2.20 M/sec
            DataStreamReceiver:
              BytesReceived(500.000ms): 7.57 MB, 15.13 MB, 21.41 MB, 28.33 MB, 35.80 MB, 45.54 MB, 54.95 MB, 62.50 MB, 71.47 MB, 81.98 MB, 93.58 MB, 105.10 MB, 117.00 MB, 128.77 MB, 140.32 MB, 152.05 MB, 162.20 MB, 173.23 MB, 185.00 MB, 196.73 MB, 208.40 MB, 220.22 MB, 232.05 MB, 243.48 MB, 253.96 MB, 265.26 MB, 277.00 MB, 288.71 MB, 300.16 MB, 311.85 MB, 323.50 MB, 334.73 MB, 346.51 MB, 358.21 MB, 368.93 MB, 378.08 MB, 387.25 MB, 395.23 MB, 398.76 MB, 401.06 MB, 402.18 MB, 402.18 MB, 402.18 MB, 402.18 MB, 402.18 MB, 402.18 MB, 402.18 MB
               - BytesReceived: 402.18 MB (421714313)
               - DeserializeRowBatchTimer: 1s124ms
               - FirstBatchArrivalWaitTime: 0.000ns
               - PeakMemoryUsage: 11.11 MB (11652300)
               - SendersBlockedTimer: 0.000ns
               - SendersBlockedTotalTimer(*): 0.000ns
        EXCHANGE_NODE (id=5):(Total: 6m55s, non-child: 6m55s, % non-child: 100.00%)
           - ConvertRowBatchTime: 2s171ms
           - PeakMemoryUsage: 0
           - RowsReturned: 19.42M (19423232)
           - RowsReturnedRate: 46.76 K/sec
          DataStreamReceiver:
            BytesReceived(1m4s): 7.94 MB, 11.11 MB, 15.24 MB, 18.82 MB, 22.33 MB, 26.70 MB, 32.20 MB, 37.66 MB, 42.18 MB, 46.47 MB, 50.72 MB, 54.98 MB, 59.07 MB, 63.35 MB, 67.57 MB, 71.73 MB, 76.12 MB, 80.39 MB, 84.38 MB, 88.64 MB, 92.98 MB, 97.32 MB, 101.61 MB, 105.82 MB, 110.18 MB, 114.54 MB, 118.90 MB, 123.23 MB, 127.50 MB, 131.89 MB, 136.26 MB, 140.51 MB, 144.68 MB, 148.70 MB, 153.19 MB, 158.10 MB, 162.97 MB, 167.85 MB, 172.66 MB, 177.42 MB, 182.12 MB, 186.71 MB, 191.31 MB, 196.02 MB, 200.66 MB, 205.29 MB, 209.95 MB, 214.27 MB, 218.66 MB, 223.34 MB, 227.99 MB, 232.64 MB, 237.33 MB, 242.02 MB, 246.73 MB, 251.43 MB, 256.12 MB, 260.82 MB, 265.55 MB, 270.28 MB, 274.49 MB, 278.76 MB, 283.04 MB
             - BytesReceived: 286.91 MB (300843911)
             - DeserializeRowBatchTimer: 1s677ms
             - FirstBatchArrivalWaitTime: 622.022ms
             - PeakMemoryUsage: 11.92 MB (12494925)
             - SendersBlockedTimer: 0.000ns
             - SendersBlockedTotalTimer(*): 0.000ns
          Buffer pool:
             - AllocTime: 713.547us
             - CumulativeAllocationBytes: 4.56 MB (4784128)
             - CumulativeAllocations: 57 (57)
             - PeakReservation: 3.06 MB (3211264)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 3.06 MB (3211264)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
    Averaged Fragment F00:
    Fragment F00:
      Instance 2438b42f267ed44:3e1ac3c700000004 (host=bdpe611n1:22000):(Total: 55.453ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:386/4.18 GB
        Filter 0 arrival: 21s284ms
        MemoryUsage(1m4s): 143.94 MB, 154.84 MB, 154.86 MB, 154.84 MB, 154.85 MB, 154.86 MB, 154.28 MB, 154.73 MB, 154.85 MB, 153.83 MB, 166.99 MB, 191.46 MB, 191.65 MB, 191.81 MB, 196.52 MB, 195.88 MB, 195.24 MB, 195.23 MB, 195.07 MB, 195.59 MB, 201.53 MB, 205.04 MB, 199.80 MB, 199.99 MB, 200.11 MB, 201.09 MB, 202.76 MB, 200.50 MB, 203.26 MB, 210.82 MB, 210.56 MB, 212.15 MB, 212.13 MB, 209.29 MB, 207.42 MB, 207.43 MB, 207.43 MB, 207.41 MB, 207.14 MB, 211.74 MB, 210.14 MB, 209.88 MB, 207.49 MB, 188.98 MB, 189.98 MB, 191.98 MB, 189.05 MB, 189.91 MB, 187.61 MB, 187.70 MB, 186.83 MB, 187.44 MB, 187.41 MB, 185.74 MB, 192.02 MB, 192.00 MB, 190.85 MB, 193.70 MB, 192.45 MB, 191.96 MB, 191.04 MB, 195.83 MB, 195.18 MB
        ThreadUsage(1m4s): 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40
         - AverageThreadTokens: 39.99
         - BloomFilterBytes: 16.00 MB (16777216)
         - PeakMemoryUsage: 216.37 MB (226881728)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 2.36 GB (2530721031)
         - RowsProduced: 21.88M (21879143)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1h7m
         - TotalStorageWaitTime: 52s749ms
         - TotalThreadsInvoluntaryContextSwitches: 0 (0)
         - TotalThreadsTotalWallClockTime: 32.249ms
           - TotalThreadsSysTime: 0.000ns
           - TotalThreadsUserTime: 31.550ms
         - TotalThreadsVoluntaryContextSwitches: 3 (3)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 1s504ms
           - OpenTime: 32.252ms
             - ExecTreeOpenTime: 915.429us
           - PrepareTime: 23.084ms
             - ExecTreePrepareTime: 4.885ms
        DataStreamSender (dst_id=5):(Total: 10s820ms, non-child: 10s820ms, % non-child: 100.00%)
           - BytesSent: 323.04 MB (338729014)
           - NetworkThroughput(*): 129.37 MB/sec
           - OverallThroughput: 29.86 MB/sec
           - PeakMemoryUsage: 177.62 KB (181880)
           - RowsReturned: 21.88M (21878119)
           - SerializeBatchTime: 8s315ms
           - TransmitDataRPCTime: 2s497ms
           - UncompressedRowBatchSize: 438.21 MB (459496759)
        CodeGen:(Total: 47.723ms, non-child: 47.723ms, % non-child: 100.00%)
           - CodegenTime: 1.816ms
           - CompileTime: 4.683ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 29 (29)
           - NumInstructions: 437 (437)
           - OptimizationTime: 25.986ms
           - PeakMemoryUsage: 218.50 KB (223744)
           - PrepareTime: 16.431ms
        HDFS_SCAN_NODE (id=0):(Total: 1s476ms, non-child: 1s476ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:386/4.18 GB
          ExecOption: PARQUET Codegen Enabled
          Runtime filters: Not all filters arrived (arrived: [], missing [0]), waited for 963ms
          BytesRead(1m4s): 78.87 MB, 82.05 MB, 82.05 MB, 82.05 MB, 82.05 MB, 82.05 MB, 82.05 MB, 84.65 MB, 84.73 MB, 85.20 MB, 113.53 MB, 136.74 MB, 136.88 MB, 138.24 MB, 141.97 MB, 141.99 MB, 143.33 MB, 143.33 MB, 143.33 MB, 149.84 MB, 158.62 MB, 165.59 MB, 173.98 MB, 179.09 MB, 181.20 MB, 183.28 MB, 184.44 MB, 186.89 MB, 192.38 MB, 203.07 MB, 209.82 MB, 221.26 MB, 230.63 MB, 234.38 MB, 237.83 MB, 237.87 MB, 237.87 MB, 237.87 MB, 237.87 MB, 242.55 MB, 245.86 MB, 252.57 MB, 253.92 MB, 274.27 MB, 285.76 MB, 287.18 MB, 288.10 MB, 289.84 MB, 290.32 MB, 291.22 MB, 291.68 MB, 292.60 MB, 292.60 MB, 293.47 MB, 299.67 MB, 302.75 MB, 309.14 MB, 317.08 MB, 321.24 MB, 324.43 MB, 325.38 MB, 331.44 MB, 332.20 MB
           - FooterProcessingTime: (Avg: 123.978ms ; Min: 998.381us ; Max: 1s024ms ; Number of samples: 162)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 38.99
           - BytesRead: 337.49 MB (353888074)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 2 (2)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 0 (0)
           - NumRowGroups: 162 (162)
           - NumScannerThreadsStarted: 39 (39)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 200.19 MB (209912215)
           - PerReadThreadRawHdfsThroughput: 61.99 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 0.000ns
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 21.97M (21970270)
           - RowsReturned: 21.88M (21879143)
           - RowsReturnedRate: 14.82 M/sec
           - ScanRangesComplete: 123 (123)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 0.000ns
             - MaterializeTupleTime(*): 6s513ms
             - ScannerThreadsSysTime: 0.000ns
             - ScannerThreadsUserTime: 0.000ns
           - ScannerThreadsVoluntaryContextSwitches: 0 (0)
           - TotalRawHdfsReadTime(*): 5s444ms
           - TotalReadThroughput: 85.28 KB/sec
          Filter 0 (16.00 MB):
             - Rows processed: 1.93M (1931264)
             - Rows rejected: 0 (0)
             - Rows total: 17.32M (17318247)
      Instance 2438b42f267ed44:3e1ac3c700000002 (host=bdpe611n2:22000):(Total: 84.539ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:385/4.17 GB
        Filter 0 arrival: 21s443ms
        MemoryUsage(1m4s): 150.81 MB, 163.60 MB, 163.59 MB, 163.58 MB, 163.60 MB, 163.60 MB, 163.59 MB, 163.61 MB, 163.60 MB, 163.16 MB, 163.58 MB, 163.59 MB, 162.54 MB, 167.00 MB, 190.04 MB, 190.06 MB, 190.06 MB, 190.05 MB, 190.07 MB, 192.55 MB, 192.56 MB, 192.55 MB, 192.96 MB, 194.76 MB, 194.78 MB, 195.84 MB, 214.78 MB, 213.38 MB, 204.45 MB, 208.05 MB, 208.28 MB, 208.27 MB, 208.30 MB, 208.29 MB, 208.28 MB, 208.09 MB, 208.86 MB, 212.46 MB, 208.51 MB, 208.58 MB, 197.71 MB, 195.12 MB, 195.36 MB, 193.94 MB, 193.15 MB, 193.14 MB, 192.79 MB, 200.50 MB, 200.50 MB, 202.30 MB, 201.78 MB, 208.82 MB, 208.41 MB, 209.08 MB, 210.73 MB, 211.13 MB, 211.13 MB, 211.77 MB, 213.36 MB, 210.88 MB, 209.36 MB, 209.28 MB, 197.86 MB
        ThreadUsage(1m4s): 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40
         - AverageThreadTokens: 39.99
         - BloomFilterBytes: 16.00 MB (16777216)
         - PeakMemoryUsage: 216.79 MB (227319848)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 2.35 GB (2523002980)
         - RowsProduced: 19.28M (19282007)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1h7m
         - TotalStorageWaitTime: 1m
         - TotalThreadsInvoluntaryContextSwitches: 1 (1)
         - TotalThreadsTotalWallClockTime: 40.998ms
           - TotalThreadsSysTime: 970.000us
           - TotalThreadsUserTime: 39.651ms
         - TotalThreadsVoluntaryContextSwitches: 0 (0)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 1s617ms
           - OpenTime: 41.001ms
             - ExecTreeOpenTime: 722.978us
           - PrepareTime: 43.375ms
             - ExecTreePrepareTime: 9.931ms
        DataStreamSender (dst_id=5):(Total: 12s640ms, non-child: 12s640ms, % non-child: 100.00%)
           - BytesSent: 285.91 MB (299796899)
           - NetworkThroughput(*): 90.33 MB/sec
           - OverallThroughput: 22.62 MB/sec
           - PeakMemoryUsage: 177.62 KB (181880)
           - RowsReturned: 19.28M (19280983)
           - SerializeBatchTime: 9s466ms
           - TransmitDataRPCTime: 3s165ms
           - UncompressedRowBatchSize: 386.17 MB (404924313)
        CodeGen:(Total: 70.323ms, non-child: 70.323ms, % non-child: 100.00%)
           - CodegenTime: 3.270ms
           - CompileTime: 3.944ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 29 (29)
           - NumInstructions: 437 (437)
           - OptimizationTime: 34.725ms
           - PeakMemoryUsage: 218.50 KB (223744)
           - PrepareTime: 30.589ms
        HDFS_SCAN_NODE (id=0):(Total: 1s595ms, non-child: 1s595ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:385/4.17 GB
          ExecOption: PARQUET Codegen Enabled
          Runtime filters: Not all filters arrived (arrived: [], missing [0]), waited for 923ms
          BytesRead(1m4s): 82.82 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.24 MB, 88.44 MB, 88.44 MB, 88.44 MB, 112.19 MB, 132.16 MB, 132.16 MB, 132.16 MB, 132.16 MB, 132.49 MB, 134.82 MB, 134.82 MB, 134.82 MB, 135.88 MB, 137.49 MB, 137.49 MB, 145.43 MB, 163.21 MB, 164.69 MB, 182.08 MB, 187.12 MB, 188.30 MB, 188.30 MB, 188.30 MB, 188.30 MB, 188.30 MB, 188.30 MB, 194.83 MB, 197.84 MB, 200.01 MB, 216.59 MB, 228.42 MB, 233.02 MB, 233.33 MB, 233.61 MB, 234.67 MB, 234.67 MB, 236.64 MB, 243.52 MB, 243.52 MB, 246.88 MB, 259.84 MB, 276.82 MB, 285.14 MB, 290.53 MB, 293.70 MB, 294.16 MB, 294.16 MB, 295.38 MB, 296.84 MB, 297.51 MB, 300.96 MB, 308.61 MB, 313.48 MB
           - FooterProcessingTime: (Avg: 167.786ms ; Min: 1.095ms ; Max: 1s132ms ; Number of samples: 153)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 39.00
           - BytesRead: 323.72 MB (339446916)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 2 (2)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 0 (0)
           - NumRowGroups: 153 (153)
           - NumScannerThreadsStarted: 39 (39)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 200.59 MB (210333951)
           - PerReadThreadRawHdfsThroughput: 72.86 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 0.000ns
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 19.37M (19367427)
           - RowsReturned: 19.28M (19282007)
           - RowsReturnedRate: 12.08 M/sec
           - ScanRangesComplete: 114 (114)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 0.000ns
             - MaterializeTupleTime(*): 7s403ms
             - ScannerThreadsSysTime: 0.000ns
             - ScannerThreadsUserTime: 0.000ns
           - ScannerThreadsVoluntaryContextSwitches: 0 (0)
           - TotalRawHdfsReadTime(*): 4s443ms
           - TotalReadThroughput: 81.79 KB/sec
          Filter 0 (16.00 MB):
             - Rows processed: 1.75M (1754112)
             - Rows rejected: 0 (0)
             - Rows total: 16.09M (16090035)
      Instance 2438b42f267ed44:3e1ac3c700000001 (host=bdpe611n3:22000):(Total: 85.005ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:384/4.18 GB
        Filter 0 arrival: 21s619ms
        MemoryUsage(1m4s): 134.97 MB, 145.25 MB, 145.23 MB, 145.21 MB, 145.24 MB, 145.24 MB, 145.22 MB, 145.21 MB, 145.22 MB, 145.21 MB, 144.74 MB, 139.05 MB, 175.44 MB, 176.40 MB, 176.37 MB, 176.38 MB, 176.26 MB, 176.45 MB, 177.04 MB, 178.64 MB, 178.64 MB, 178.64 MB, 178.61 MB, 178.17 MB, 173.00 MB, 187.41 MB, 191.83 MB, 189.62 MB, 187.75 MB, 187.95 MB, 188.47 MB, 194.64 MB, 197.19 MB, 197.16 MB, 197.17 MB, 197.17 MB, 196.54 MB, 198.07 MB, 202.24 MB, 194.24 MB, 184.85 MB, 176.52 MB, 179.96 MB, 183.74 MB, 183.73 MB, 182.92 MB, 183.73 MB, 187.84 MB, 194.18 MB, 193.12 MB, 192.58 MB, 194.24 MB, 196.05 MB, 195.43 MB, 191.75 MB, 192.35 MB, 192.34 MB, 192.33 MB, 192.56 MB, 196.64 MB, 196.46 MB, 197.65 MB, 194.81 MB
        ThreadUsage(1m4s): 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40
         - AverageThreadTokens: 39.99
         - BloomFilterBytes: 16.00 MB (16777216)
         - PeakMemoryUsage: 205.33 MB (215307504)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 2.33 GB (2505825451)
         - RowsProduced: 19.06M (19056293)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1h7m
         - TotalStorageWaitTime: 45s650ms
         - TotalThreadsInvoluntaryContextSwitches: 0 (0)
         - TotalThreadsTotalWallClockTime: 43.650ms
           - TotalThreadsSysTime: 1.981ms
           - TotalThreadsUserTime: 40.673ms
         - TotalThreadsVoluntaryContextSwitches: 1 (1)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 1s611ms
           - OpenTime: 43.660ms
             - ExecTreeOpenTime: 770.272us
           - PrepareTime: 41.072ms
             - ExecTreePrepareTime: 10.438ms
        DataStreamSender (dst_id=5):(Total: 13s402ms, non-child: 13s402ms, % non-child: 100.00%)
           - BytesSent: 280.48 MB (294099431)
           - NetworkThroughput(*): 92.33 MB/sec
           - OverallThroughput: 20.93 MB/sec
           - PeakMemoryUsage: 177.62 KB (181880)
           - RowsReturned: 19.06M (19055269)
           - SerializeBatchTime: 10s355ms
           - TransmitDataRPCTime: 3s037ms
           - UncompressedRowBatchSize: 381.63 MB (400170968)
        CodeGen:(Total: 70.305ms, non-child: 70.305ms, % non-child: 100.00%)
           - CodegenTime: 3.083ms
           - CompileTime: 4.354ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 29 (29)
           - NumInstructions: 437 (437)
           - OptimizationTime: 37.275ms
           - PeakMemoryUsage: 218.50 KB (223744)
           - PrepareTime: 27.772ms
        HDFS_SCAN_NODE (id=0):(Total: 1s589ms, non-child: 1s589ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:384/4.18 GB
          ExecOption: PARQUET Codegen Enabled
          Runtime filters: Not all filters arrived (arrived: [], missing [0]), waited for 923ms
          BytesRead(1m4s): 73.07 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 88.81 MB, 135.10 MB, 135.82 MB, 135.82 MB, 135.82 MB, 135.82 MB, 137.02 MB, 138.45 MB, 139.87 MB, 139.87 MB, 139.87 MB, 139.87 MB, 140.02 MB, 144.54 MB, 161.79 MB, 165.31 MB, 167.45 MB, 170.50 MB, 170.86 MB, 173.03 MB, 180.97 MB, 182.62 MB, 182.62 MB, 182.62 MB, 182.62 MB, 183.09 MB, 188.19 MB, 193.27 MB, 201.88 MB, 216.28 MB, 223.21 MB, 231.16 MB, 235.53 MB, 235.53 MB, 236.15 MB, 238.19 MB, 242.95 MB, 249.19 MB, 250.34 MB, 261.98 MB, 275.16 MB, 281.47 MB, 282.85 MB, 287.02 MB, 288.56 MB, 288.56 MB, 288.56 MB, 289.35 MB, 292.31 MB, 294.73 MB, 302.91 MB, 311.58 MB
           - FooterProcessingTime: (Avg: 116.132ms ; Min: 1.215ms ; Max: 845.660ms ; Number of samples: 161)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 38.99
           - BytesRead: 319.01 MB (334507959)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 2 (2)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 0 (0)
          - NumRowGroups: 161 (161)
           - NumScannerThreadsStarted: 39 (39)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 189.15 MB (198337991)
           - PerReadThreadRawHdfsThroughput: 75.30 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 0.000ns
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 19.15M (19145381)
           - RowsReturned: 19.06M (19056293)
           - RowsReturnedRate: 11.99 M/sec
           - ScanRangesComplete: 122 (122)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 0.000ns
             - MaterializeTupleTime(*): 9s050ms
             - ScannerThreadsSysTime: 0.000ns
             - ScannerThreadsUserTime: 0.000ns
           - ScannerThreadsVoluntaryContextSwitches: 0 (0)
           - TotalRawHdfsReadTime(*): 4s236ms
           - TotalReadThroughput: 80.52 KB/sec
          Filter 0 (16.00 MB):
             - Rows processed: 1.90M (1902592)
             - Rows rejected: 0 (0)
             - Rows total: 15.55M (15552165)
      Instance 2438b42f267ed44:3e1ac3c700000005 (host=bdpe611n4:22000):(Total: 71.177ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:383/4.18 GB
        Filter 0 arrival: 21s843ms
        MemoryUsage(1m4s): 139.40 MB, 151.29 MB, 151.32 MB, 151.29 MB, 151.32 MB, 151.31 MB, 151.31 MB, 151.31 MB, 151.30 MB, 151.32 MB, 151.29 MB, 151.30 MB, 151.29 MB, 151.31 MB, 146.82 MB, 162.00 MB, 176.59 MB, 176.75 MB, 176.78 MB, 176.77 MB, 176.78 MB, 176.41 MB, 178.32 MB, 178.72 MB, 178.72 MB, 178.73 MB, 178.72 MB, 177.66 MB, 175.91 MB, 191.75 MB, 184.02 MB, 181.95 MB, 177.38 MB, 182.42 MB, 182.99 MB, 183.21 MB, 183.22 MB, 183.20 MB, 183.23 MB, 184.59 MB, 184.76 MB, 192.95 MB, 196.09 MB, 195.33 MB, 194.24 MB, 193.29 MB, 194.14 MB, 194.14 MB, 193.45 MB, 196.21 MB, 196.09 MB, 198.99 MB, 203.00 MB, 198.52 MB, 191.68 MB, 190.21 MB, 192.16 MB, 192.28 MB, 192.11 MB, 192.39 MB, 192.12 MB, 192.84 MB, 194.61 MB
        ThreadUsage(1m4s): 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40
         - AverageThreadTokens: 39.99
         - BloomFilterBytes: 16.00 MB (16777216)
         - PeakMemoryUsage: 203.41 MB (213288576)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 2.34 GB (2515248771)
         - RowsProduced: 18.56M (18555600)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1h7m
         - TotalStorageWaitTime: 1m
         - TotalThreadsInvoluntaryContextSwitches: 0 (0)
         - TotalThreadsTotalWallClockTime: 35.550ms
           - TotalThreadsSysTime: 0.000ns
           - TotalThreadsUserTime: 35.291ms
         - TotalThreadsVoluntaryContextSwitches: 0 (0)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 1s504ms
           - OpenTime: 35.553ms
             - ExecTreeOpenTime: 735.786us
           - PrepareTime: 35.470ms
             - ExecTreePrepareTime: 8.224ms
        DataStreamSender (dst_id=5):(Total: 10s272ms, non-child: 10s272ms, % non-child: 100.00%)
           - BytesSent: 273.41 MB (286691030)
           - NetworkThroughput(*): 98.09 MB/sec
           - OverallThroughput: 26.62 MB/sec
           - PeakMemoryUsage: 177.62 KB (181880)
           - RowsReturned: 18.55M (18554576)
           - SerializeBatchTime: 7s477ms
           - TransmitDataRPCTime: 2s787ms
           - UncompressedRowBatchSize: 371.62 MB (389673155)
        CodeGen:(Total: 58.194ms, non-child: 58.194ms, % non-child: 100.00%)
           - CodegenTime: 2.512ms
           - CompileTime: 3.759ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 29 (29)
           - NumInstructions: 437 (437)
           - OptimizationTime: 30.129ms
           - PeakMemoryUsage: 218.50 KB (223744)
           - PrepareTime: 23.586ms
        HDFS_SCAN_NODE (id=0):(Total: 1s489ms, non-child: 1s489ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:383/4.18 GB
          ExecOption: PARQUET Codegen Enabled
          Runtime filters: Not all filters arrived (arrived: [], missing [0]), waited for 943ms
          BytesRead(1m4s): 80.26 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 85.37 MB, 120.85 MB, 133.32 MB, 133.46 MB, 133.46 MB, 133.46 MB, 133.46 MB, 133.46 MB, 135.87 MB, 136.14 MB, 136.14 MB, 136.14 MB, 136.14 MB, 136.19 MB, 141.98 MB, 161.14 MB, 162.47 MB, 171.68 MB, 175.43 MB, 182.38 MB, 184.33 MB, 185.02 MB, 185.02 MB, 185.02 MB, 185.02 MB, 186.91 MB, 192.14 MB, 202.80 MB, 208.74 MB, 220.51 MB, 233.96 MB, 240.41 MB, 241.97 MB, 241.97 MB, 242.08 MB, 245.66 MB, 245.66 MB, 252.62 MB, 256.43 MB, 261.55 MB, 271.20 MB, 283.57 MB, 286.51 MB, 287.54 MB, 288.70 MB, 288.94 MB, 288.94 MB, 291.34 MB, 292.95 MB
           - FooterProcessingTime: (Avg: 161.210ms ; Min: 906.207us ; Max: 998.197ms ; Number of samples: 150)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 38.99
           - BytesRead: 292.95 MB (307182818)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 2 (2)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 0 (0)
           - NumRowGroups: 150 (150)
           - NumScannerThreadsStarted: 39 (39)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 187.24 MB (196335447)
           - PerReadThreadRawHdfsThroughput: 57.66 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 0.000ns
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 18.65M (18646736)
           - RowsReturned: 18.56M (18555600)
           - RowsReturnedRate: 12.46 M/sec
           - ScanRangesComplete: 111 (111)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 0.000ns
             - MaterializeTupleTime(*): 5s686ms
             - ScannerThreadsSysTime: 0.000ns
             - ScannerThreadsUserTime: 0.000ns
           - ScannerThreadsVoluntaryContextSwitches: 0 (0)
           - TotalRawHdfsReadTime(*): 5s080ms
           - TotalReadThroughput: 74.01 KB/sec
          Filter 0 (16.00 MB):
             - Rows processed: 1.68M (1679360)
             - Rows rejected: 0 (0)
             - Rows total: 13.77M (13769424)
      Instance 2438b42f267ed44:3e1ac3c700000003 (host=bdpe722n4:22000):(Total: 82.178ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:385/4.17 GB
        Filter 0 arrival: 21s965ms
        MemoryUsage(1m4s): 152.01 MB, 163.30 MB, 163.31 MB, 163.31 MB, 162.33 MB, 160.76 MB, 161.08 MB, 161.11 MB, 161.11 MB, 161.09 MB, 157.17 MB, 178.12 MB, 194.19 MB, 194.21 MB, 194.21 MB, 193.78 MB, 194.19 MB, 198.23 MB, 198.22 MB, 198.23 MB, 198.24 MB, 198.12 MB, 198.66 MB, 210.96 MB, 209.29 MB, 192.40 MB, 190.51 MB, 190.58 MB, 191.98 MB, 194.02 MB, 194.48 MB, 194.46 MB, 194.45 MB, 194.45 MB, 194.46 MB, 192.65 MB, 198.75 MB, 194.95 MB, 204.31 MB, 192.23 MB, 189.19 MB, 193.58 MB, 194.35 MB, 194.33 MB, 194.33 MB, 194.34 MB, 195.14 MB, 203.49 MB, 203.70 MB, 205.54 MB, 197.37 MB, 196.96 MB, 196.98 MB, 196.47 MB, 192.56 MB, 192.96 MB, 188.01 MB, 188.65 MB, 188.66 MB, 188.66 MB, 189.08 MB, 190.12 MB, 190.84 MB
        ThreadUsage(1m4s): 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40
         - AverageThreadTokens: 39.99
         - BloomFilterBytes: 16.00 MB (16777216)
         - PeakMemoryUsage: 212.08 MB (222381024)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 2.34 GB (2507859104)
         - RowsProduced: 18.98M (18978066)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 1h7m
         - TotalStorageWaitTime: 50s972ms
         - TotalThreadsInvoluntaryContextSwitches: 1 (1)
         - TotalThreadsTotalWallClockTime: 43.018ms
           - TotalThreadsSysTime: 0.000ns
           - TotalThreadsUserTime: 42.330ms
         - TotalThreadsVoluntaryContextSwitches: 1 (1)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 1s645ms
           - OpenTime: 43.020ms
             - ExecTreeOpenTime: 685.669us
           - PrepareTime: 39.009ms
             - ExecTreePrepareTime: 9.755ms
        DataStreamSender (dst_id=5):(Total: 13s018ms, non-child: 13s018ms, % non-child: 100.00%)
           - BytesSent: 280.86 MB (294500459)
           - NetworkThroughput(*): 120.91 MB/sec
           - OverallThroughput: 21.57 MB/sec
           - PeakMemoryUsage: 177.62 KB (181880)
           - RowsReturned: 18.98M (18977042)
           - SerializeBatchTime: 10s686ms
           - TransmitDataRPCTime: 2s322ms
           - UncompressedRowBatchSize: 380.09 MB (398552808)
        CodeGen:(Total: 68.436ms, non-child: 68.436ms, % non-child: 100.00%)
           - CodegenTime: 3.099ms
           - CompileTime: 3.775ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 29 (29)
           - NumInstructions: 437 (437)
           - OptimizationTime: 37.348ms
           - PeakMemoryUsage: 218.50 KB (223744)
           - PrepareTime: 26.399ms
        HDFS_SCAN_NODE (id=0):(Total: 1s622ms, non-child: 1s622ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:385/4.17 GB
          ExecOption: PARQUET Codegen Enabled
          Runtime filters: Not all filters arrived (arrived: [], missing [0]), waited for 923ms
          BytesRead(1m4s): 80.94 MB, 84.23 MB, 84.23 MB, 84.23 MB, 84.23 MB, 85.30 MB, 85.60 MB, 85.60 MB, 85.60 MB, 85.60 MB, 85.78 MB, 119.30 MB, 134.23 MB, 134.23 MB, 134.23 MB, 135.07 MB, 136.34 MB, 139.31 MB, 139.31 MB, 139.31 MB, 139.31 MB, 139.31 MB, 144.43 MB, 159.83 MB, 160.44 MB, 165.29 MB, 172.66 MB, 173.43 MB, 175.33 MB, 177.99 MB, 179.39 MB, 179.39 MB, 179.39 MB, 179.39 MB, 179.39 MB, 181.08 MB, 188.56 MB, 192.87 MB, 218.82 MB, 228.29 MB, 230.98 MB, 235.85 MB, 237.00 MB, 237.00 MB, 237.00 MB, 237.00 MB, 238.84 MB, 247.04 MB, 252.21 MB, 263.47 MB, 273.90 MB, 280.85 MB, 282.80 MB, 283.47 MB, 285.87 MB, 286.22 MB, 287.03 MB, 287.57 MB, 287.57 MB, 287.57 MB, 288.69 MB, 292.91 MB, 298.01 MB
           - FooterProcessingTime: (Avg: 158.157ms ; Min: 1.028ms ; Max: 901.023ms ; Number of samples: 150)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 38.99
           - BytesRead: 306.69 MB (321585641)
          - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 0 (0)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 2 (2)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 0 (0)
           - NumRowGroups: 150 (150)
           - NumScannerThreadsStarted: 39 (39)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 195.90 MB (205411511)
           - PerReadThreadRawHdfsThroughput: 66.40 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 0.000ns
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 19.07M (19073298)
           - RowsReturned: 18.98M (18978066)
           - RowsReturnedRate: 11.70 M/sec
           - ScanRangesComplete: 111 (111)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 0.000ns
             - MaterializeTupleTime(*): 7s622ms
             - ScannerThreadsSysTime: 0.000ns
             - ScannerThreadsUserTime: 0.000ns
           - ScannerThreadsVoluntaryContextSwitches: 0 (0)
           - TotalRawHdfsReadTime(*): 4s619ms
           - TotalReadThroughput: 77.49 KB/sec
          Filter 0 (16.00 MB):
             - Rows processed: 1.74M (1743872)
             - Rows rejected: 0 (0)
             - Rows total: 14.88M (14875904)
    Averaged Fragment F01:
    Fragment F01:
      Instance 2438b42f267ed44:3e1ac3c700000006 (host=bdpe611n1:22000):(Total: 18s739ms, non-child: 7s511ms, % non-child: 40.09%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:388/3.41 GB
        MemoryUsage(500.000ms): 1.76 MB, 6.03 MB, 842.40 KB, 12.20 MB, 1.27 MB, 1.35 MB, 754.27 KB, 2.28 MB, 6.07 MB, 16.39 MB, 15.51 MB, 5.56 MB, 5.44 MB, 9.52 MB, 3.21 MB, 12.83 MB, 40.04 MB, 81.45 MB, 83.83 MB, 88.26 MB, 81.70 MB, 86.63 MB, 81.17 MB, 61.96 MB, 55.29 MB, 67.06 MB, 76.19 MB, 67.45 MB, 64.55 MB, 74.25 MB, 82.55 MB, 84.46 MB, 94.95 MB, 90.90 MB, 94.23 MB, 77.24 MB, 42.21 MB
        ThreadUsage(500.000ms): 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 28, 14
         - AverageThreadTokens: 38.97
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 98.20 MB (102973750)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 710.68 MB (745201229)
         - RowsProduced: 28.80M (28804452)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 12.756ms
         - TotalStorageWaitTime: 5m38s
         - TotalThreadsInvoluntaryContextSwitches: 143 (143)
         - TotalThreadsTotalWallClockTime: 12m3s
           - TotalThreadsSysTime: 494.514ms
           - TotalThreadsUserTime: 23s480ms
         - TotalThreadsVoluntaryContextSwitches: 27.76K (27760)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 18s679ms
             - ExecTreeExecTime: 1s708ms
           - OpenTime: 19.453ms
             - ExecTreeOpenTime: 793.045us
           - PrepareTime: 40.244ms
             - ExecTreePrepareTime: 11.587ms
        DataStreamSender (dst_id=6):(Total: 9s474ms, non-child: 9s474ms, % non-child: 100.00%)
           - BytesSent: 405.14 MB (424823878)
           - NetworkThroughput(*): 232.29 MB/sec
           - OverallThroughput: 42.76 MB/sec
           - PeakMemoryUsage: 177.62 KB (181880)
           - RowsReturned: 28.80M (28804452)
           - SerializeBatchTime: 7s725ms
           - TransmitDataRPCTime: 1s744ms
           - UncompressedRowBatchSize: 576.99 MB (605013144)
        CodeGen:(Total: 45.247ms, non-child: 45.247ms, % non-child: 100.00%)
           - CodegenTime: 1.192ms
           - CompileTime: 2.651ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 14.958ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 26.660ms
        HDFS_SCAN_NODE (id=1):(Total: 1s707ms, non-child: 1s707ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:388/3.41 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 388 out of 388
          Hdfs Read Thread Concurrency Bucket: 0:29.73% 1:70.27% 2:0% 3:0% 4:0% 5:0%
          File Formats: PARQUET/NONE:776
          BytesRead(500.000ms): 7.00 MB, 19.43 MB, 28.44 MB, 39.55 MB, 48.13 MB, 56.26 MB, 65.00 MB, 71.00 MB, 84.05 MB, 98.90 MB, 110.39 MB, 120.82 MB, 134.11 MB, 147.25 MB, 157.92 MB, 175.50 MB, 199.36 MB, 231.07 MB, 238.86 MB, 247.73 MB, 255.24 MB, 271.52 MB, 280.39 MB, 291.89 MB, 307.52 MB, 322.65 MB, 336.39 MB, 344.78 MB, 359.34 MB, 376.17 MB, 401.50 MB, 410.95 MB, 426.16 MB, 434.93 MB, 448.52 MB, 452.67 MB, 452.67 MB
           - FooterProcessingTime: (Avg: 429.574ms ; Min: 734.820us ; Max: 1s454ms ; Number of samples: 388)
           - AverageHdfsReadThreadConcurrency: 0.70
           - AverageScannerThreadConcurrency: 37.97
           - BytesRead: 452.67 MB (474662103)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 452.67 MB (474662103)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 933 (933)
           - CachedFileHandlesMissCount: 231 (231)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 2 (2)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 388 (388)
           - NumScannerThreadsStarted: 39 (39)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 98.02 MB (102782239)
           - PerReadThreadRawHdfsThroughput: 30.11 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 1s425ms
           - RowBatchQueuePutWaitTime: 5m45s
           - RowsRead: 28.80M (28804452)
           - RowsReturned: 28.80M (28804452)
           - RowsReturnedRate: 16.87 M/sec
           - ScanRangesComplete: 388 (388)
           - ScannerThreadsInvoluntaryContextSwitches: 83 (83)
           - ScannerThreadsTotalWallClockTime: 11m45s
             - MaterializeTupleTime(*): 5s156ms
             - ScannerThreadsSysTime: 314.594ms
             - ScannerThreadsUserTime: 6s399ms
           - ScannerThreadsVoluntaryContextSwitches: 27.67K (27673)
           - TotalRawHdfsReadTime(*): 15s034ms
           - TotalReadThroughput: 24.47 MB/sec
      Instance 2438b42f267ed44:3e1ac3c700000009 (host=bdpe611n2:22000):(Total: 20s236ms, non-child: 7s497ms, % non-child: 37.05%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:387/3.40 GB
        MemoryUsage(500.000ms): 37.62 KB, 673.14 KB, 2.18 MB, 746.30 KB, 490.39 KB, 458.45 KB, 9.22 MB, 2.08 MB, 1.28 MB, 4.18 MB, 7.23 MB, 19.96 MB, 49.62 MB, 41.85 MB, 41.28 MB, 22.38 MB, 710.30 KB, 858.45 KB, 24.58 MB, 28.19 MB, 31.91 MB, 38.47 MB, 40.97 MB, 30.28 MB, 31.27 MB, 35.69 MB, 51.93 MB, 70.03 MB, 74.24 MB, 89.80 MB, 89.11 MB, 77.88 MB, 76.28 MB, 68.25 MB, 68.93 MB, 59.45 MB, 63.09 MB, 68.49 MB, 67.46 MB, 46.51 MB, 24.61 MB
        ThreadUsage(500.000ms): 1, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 22, 8
         - AverageThreadTokens: 37.83
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 93.42 MB (97953773)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 727.59 MB (762937435)
         - RowsProduced: 28.79M (28788883)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 12.764ms
         - TotalStorageWaitTime: 7m21s
         - TotalThreadsInvoluntaryContextSwitches: 388 (388)
         - TotalThreadsTotalWallClockTime: 13m2s
           - TotalThreadsSysTime: 487.302ms
           - TotalThreadsUserTime: 23s689ms
         - TotalThreadsVoluntaryContextSwitches: 27.90K (27903)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 20s177ms
             - ExecTreeExecTime: 3s275ms
           - OpenTime: 19.161ms
             - ExecTreeOpenTime: 777.817us
           - PrepareTime: 39.621ms
             - ExecTreePrepareTime: 10.609ms
        DataStreamSender (dst_id=6):(Total: 9s423ms, non-child: 9s423ms, % non-child: 100.00%)
           - BytesSent: 404.95 MB (424617979)
           - NetworkThroughput(*): 231.84 MB/sec
           - OverallThroughput: 42.97 MB/sec
           - PeakMemoryUsage: 177.62 KB (181880)
           - RowsReturned: 28.79M (28788883)
           - SerializeBatchTime: 7s671ms
           - TransmitDataRPCTime: 1s746ms
           - UncompressedRowBatchSize: 576.67 MB (604686135)
        CodeGen:(Total: 44.918ms, non-child: 44.918ms, % non-child: 100.00%)
           - CodegenTime: 1.156ms
           - CompileTime: 2.415ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 14.818ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 26.800ms
        HDFS_SCAN_NODE (id=1):(Total: 3s270ms, non-child: 3s270ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:387/3.40 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 387 out of 387
          Hdfs Read Thread Concurrency Bucket: 0:7.5% 1:92.5% 2:0% 3:0% 4:0% 5:0%
          File Formats: PARQUET/NONE:774
          BytesRead(500.000ms): 0, 7.63 MB, 12.75 MB, 17.28 MB, 23.94 MB, 30.49 MB, 41.65 MB, 50.18 MB, 56.42 MB, 62.53 MB, 70.59 MB, 88.79 MB, 112.31 MB, 119.81 MB, 131.30 MB, 137.71 MB, 146.37 MB, 150.41 MB, 171.55 MB, 182.59 MB, 195.88 MB, 212.64 MB, 224.92 MB, 236.00 MB, 251.59 MB, 265.33 MB, 285.94 MB, 307.73 MB, 316.23 MB, 335.93 MB, 346.77 MB, 355.72 MB, 366.58 MB, 378.46 MB, 392.91 MB, 404.08 MB, 418.19 MB, 431.16 MB, 441.04 MB, 449.77 MB, 452.59 MB
           - FooterProcessingTime: (Avg: 563.541ms ; Min: 880.869us ; Max: 1s508ms ; Number of samples: 387)
           - AverageHdfsReadThreadConcurrency: 0.93
           - AverageScannerThreadConcurrency: 37.75
           - BytesRead: 452.59 MB (474576310)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 452.59 MB (474576310)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 945 (945)
           - CachedFileHandlesMissCount: 216 (216)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
          - MaxCompressedTextFileLength: 0
           - NumColumns: 2 (2)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 387 (387)
           - NumScannerThreadsStarted: 39 (39)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 93.23 MB (97762262)
           - PerReadThreadRawHdfsThroughput: 24.96 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 2s988ms
           - RowBatchQueuePutWaitTime: 4m49s
           - RowsRead: 28.79M (28788883)
           - RowsReturned: 28.79M (28788883)
           - RowsReturnedRate: 8.80 M/sec
           - ScanRangesComplete: 387 (387)
           - ScannerThreadsInvoluntaryContextSwitches: 140 (140)
           - ScannerThreadsTotalWallClockTime: 12m42s
             - MaterializeTupleTime(*): 5s310ms
             - ScannerThreadsSysTime: 360.294ms
             - ScannerThreadsUserTime: 6s621ms
           - ScannerThreadsVoluntaryContextSwitches: 27.80K (27802)
           - TotalRawHdfsReadTime(*): 18s131ms
           - TotalReadThroughput: 22.08 MB/sec
      Instance 2438b42f267ed44:3e1ac3c700000007 (host=bdpe611n3:22000):(Total: 17s317ms, non-child: 7s255ms, % non-child: 41.90%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:386/3.40 GB
        MemoryUsage(500.000ms): 13.78 MB, 14.74 MB, 23.47 MB, 24.69 MB, 7.56 MB, 786.34 KB, 11.93 MB, 9.53 MB, 19.48 MB, 27.44 MB, 37.02 MB, 51.12 MB, 61.02 MB, 77.49 MB, 89.82 MB, 91.64 MB, 95.21 MB, 95.21 MB, 96.97 MB, 91.19 MB, 92.26 MB, 95.63 MB, 87.48 MB, 86.90 MB, 86.48 MB, 67.31 MB, 43.66 MB, 54.83 MB, 74.72 MB, 65.69 MB, 58.68 MB, 64.94 MB, 69.01 MB, 50.47 MB, 16.42 MB
        ThreadUsage(500.000ms): 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 39, 24, 5
         - AverageThreadTokens: 38.51
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 102.32 MB (107294254)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 640.03 MB (671116894)
         - RowsProduced: 28.79M (28786856)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 17.008ms
         - TotalStorageWaitTime: 3m55s
         - TotalThreadsInvoluntaryContextSwitches: 77 (77)
         - TotalThreadsTotalWallClockTime: 11m9s
           - TotalThreadsSysTime: 427.669ms
           - TotalThreadsUserTime: 23s514ms
         - TotalThreadsVoluntaryContextSwitches: 30.17K (30173)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 17s255ms
             - ExecTreeExecTime: 630.366ms
           - OpenTime: 21.983ms
             - ExecTreeOpenTime: 794.560us
           - PrepareTime: 40.089ms
             - ExecTreePrepareTime: 10.105ms
        DataStreamSender (dst_id=6):(Total: 9s393ms, non-child: 9s393ms, % non-child: 100.00%)
           - BytesSent: 404.96 MB (424630554)
           - NetworkThroughput(*): 223.64 MB/sec
           - OverallThroughput: 43.11 MB/sec
           - PeakMemoryUsage: 177.62 KB (181880)
           - RowsReturned: 28.79M (28786856)
           - SerializeBatchTime: 7s577ms
           - TransmitDataRPCTime: 1s810ms
           - UncompressedRowBatchSize: 576.63 MB (604643556)
        CodeGen:(Total: 48.826ms, non-child: 48.826ms, % non-child: 100.00%)
           - CodegenTime: 1.294ms
           - CompileTime: 3.504ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 16.485ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 27.871ms
        HDFS_SCAN_NODE (id=1):(Total: 620.395ms, non-child: 620.395ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:386/3.40 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 386 out of 386
          Hdfs Read Thread Concurrency Bucket: 0:34.29% 1:65.71% 2:0% 3:0% 4:0% 5:0%
          File Formats: PARQUET/NONE:772
          BytesRead(500.000ms): 9.30 MB, 22.92 MB, 39.97 MB, 53.28 MB, 59.49 MB, 67.43 MB, 82.92 MB, 95.58 MB, 112.51 MB, 129.79 MB, 143.33 MB, 163.12 MB, 180.09 MB, 197.56 MB, 212.59 MB, 228.42 MB, 240.94 MB, 248.82 MB, 264.78 MB, 278.10 MB, 296.45 MB, 308.04 MB, 316.73 MB, 326.30 MB, 335.87 MB, 344.96 MB, 358.22 MB, 375.37 MB, 394.93 MB, 403.47 MB, 416.35 MB, 429.51 MB, 452.42 MB, 452.42 MB, 452.42 MB
           - FooterProcessingTime: (Avg: 298.680ms ; Min: 724.423us ; Max: 1s068ms ; Number of samples: 386)
           - AverageHdfsReadThreadConcurrency: 0.66
           - AverageScannerThreadConcurrency: 37.51
           - BytesRead: 452.42 MB (474401701)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 452.42 MB (474401701)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 958 (958)
           - CachedFileHandlesMissCount: 200 (200)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 2 (2)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 386 (386)
           - NumScannerThreadsStarted: 39 (39)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 102.14 MB (107102743)
           - PerReadThreadRawHdfsThroughput: 37.03 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 280.019ms
           - RowBatchQueuePutWaitTime: 6m43s
           - RowsRead: 28.79M (28786856)
           - RowsReturned: 28.79M (28786856)
           - RowsReturnedRate: 46.40 M/sec
           - ScanRangesComplete: 386 (386)
           - ScannerThreadsInvoluntaryContextSwitches: 45 (45)
           - ScannerThreadsTotalWallClockTime: 10m52s
             - MaterializeTupleTime(*): 5s044ms
             - ScannerThreadsSysTime: 300.723ms
             - ScannerThreadsUserTime: 6s655ms
           - ScannerThreadsVoluntaryContextSwitches: 30.11K (30109)
           - TotalRawHdfsReadTime(*): 12s217ms
           - TotalReadThroughput: 25.85 MB/sec
      Instance 2438b42f267ed44:3e1ac3c700000008 (host=bdpe611n4:22000):(Total: 19s105ms, non-child: 7s527ms, % non-child: 39.40%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:387/3.41 GB
        MemoryUsage(500.000ms): 37.62 KB, 6.22 MB, 754.27 KB, 314.40 KB, 1.78 MB, 10.99 MB, 5.16 MB, 3.19 MB, 426.35 KB, 10.26 MB, 20.30 MB, 29.34 MB, 32.28 MB, 38.28 MB, 48.56 MB, 51.51 MB, 55.28 MB, 64.16 MB, 62.72 MB, 50.63 MB, 48.81 MB, 56.92 MB, 74.28 MB, 80.01 MB, 84.90 MB, 79.66 MB, 68.48 MB, 58.46 MB, 60.82 MB, 62.81 MB, 56.09 MB, 67.62 MB, 74.75 MB, 91.05 MB, 87.45 MB, 93.06 MB, 90.38 MB, 57.38 MB, 17.24 MB
        ThreadUsage(500.000ms): 1, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 36, 24, 6
         - AverageThreadTokens: 37.62
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 97.56 MB (102297694)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 714.74 MB (749458108)
         - RowsProduced: 28.83M (28834382)
        - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 12.032ms
         - TotalStorageWaitTime: 5m32s
         - TotalThreadsInvoluntaryContextSwitches: 204 (204)
         - TotalThreadsTotalWallClockTime: 12m22s
           - TotalThreadsSysTime: 534.741ms
           - TotalThreadsUserTime: 25s347ms
         - TotalThreadsVoluntaryContextSwitches: 29.07K (29067)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 19s054ms
             - ExecTreeExecTime: 2s076ms
           - OpenTime: 17.312ms
             - ExecTreeOpenTime: 737.427us
           - PrepareTime: 33.913ms
             - ExecTreePrepareTime: 8.403ms
        DataStreamSender (dst_id=6):(Total: 9s471ms, non-child: 9s471ms, % non-child: 100.00%)
           - BytesSent: 405.60 MB (425303423)
           - NetworkThroughput(*): 231.66 MB/sec
           - OverallThroughput: 42.83 MB/sec
           - PeakMemoryUsage: 177.62 KB (181880)
           - RowsReturned: 28.83M (28834382)
           - SerializeBatchTime: 7s713ms
           - TransmitDataRPCTime: 1s750ms
           - UncompressedRowBatchSize: 577.59 MB (605641802)
        CodeGen:(Total: 38.865ms, non-child: 38.865ms, % non-child: 100.00%)
           - CodegenTime: 982.787us
           - CompileTime: 2.512ms
          - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 13.084ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 22.489ms
        HDFS_SCAN_NODE (id=1):(Total: 2s068ms, non-child: 2s068ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:387/3.41 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 387 out of 387
          Hdfs Read Thread Concurrency Bucket: 0:15.79% 1:84.21% 2:0% 3:0% 4:0% 5:0%
          File Formats: PARQUET/NONE:774
          BytesRead(500.000ms): 0, 8.51 MB, 15.71 MB, 21.21 MB, 28.56 MB, 42.67 MB, 52.30 MB, 61.61 MB, 65.44 MB, 78.45 MB, 96.17 MB, 111.02 MB, 124.62 MB, 138.56 MB, 156.16 MB, 171.67 MB, 186.45 MB, 202.00 MB, 212.05 MB, 220.39 MB, 239.72 MB, 258.14 MB, 274.14 MB, 287.71 MB, 296.55 MB, 309.36 MB, 318.52 MB, 334.06 MB, 344.03 MB, 357.73 MB, 367.59 MB, 388.09 MB, 403.68 MB, 420.25 MB, 426.52 MB, 443.22 MB, 452.72 MB, 452.72 MB, 452.72 MB
           - FooterProcessingTime: (Avg: 421.574ms ; Min: 767.385us ; Max: 1s455ms ; Number of samples: 387)
           - AverageHdfsReadThreadConcurrency: 0.84
           - AverageScannerThreadConcurrency: 37.58
           - BytesRead: 452.72 MB (474706225)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 452.72 MB (474706225)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 946 (946)
           - CachedFileHandlesMissCount: 215 (215)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 2 (2)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 387 (387)
           - NumScannerThreadsStarted: 39 (39)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 97.39 MB (102122567)
           - PerReadThreadRawHdfsThroughput: 28.93 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 1s774ms
           - RowBatchQueuePutWaitTime: 6m4s
           - RowsRead: 28.83M (28834382)
           - RowsReturned: 28.83M (28834382)
           - RowsReturnedRate: 13.94 M/sec
           - ScanRangesComplete: 387 (387)
           - ScannerThreadsInvoluntaryContextSwitches: 99 (99)
           - ScannerThreadsTotalWallClockTime: 12m3s
             - MaterializeTupleTime(*): 6s575ms
             - ScannerThreadsSysTime: 367.774ms
             - ScannerThreadsUserTime: 8s229ms
           - ScannerThreadsVoluntaryContextSwitches: 28.99K (28988)
           - TotalRawHdfsReadTime(*): 15s649ms
           - TotalReadThroughput: 23.22 MB/sec
      Instance 2438b42f267ed44:3e1ac3c70000000a (host=bdpe722n4:22000):(Total: 19s224ms, non-child: 7s613ms, % non-child: 39.60%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:388/3.40 GB
        MemoryUsage(500.000ms): 1.21 MB, 6.36 MB, 2.28 MB, 490.42 KB, 3.44 MB, 2.52 MB, 32.22 MB, 34.53 MB, 41.90 MB, 58.63 MB, 80.14 MB, 84.30 MB, 61.27 MB, 55.51 MB, 47.88 MB, 43.02 MB, 48.51 MB, 42.41 MB, 40.27 MB, 40.90 MB, 36.74 MB, 24.47 MB, 3.74 MB, 1.31 MB, 434.39 KB, 16.05 MB, 9.53 MB, 22.27 MB, 8.35 MB, 7.53 MB, 826.35 KB, 11.06 MB, 14.73 MB, 19.32 MB, 15.94 MB, 16.71 MB, 31.90 MB, 23.77 MB
        ThreadUsage(500.000ms): 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 28, 17, 8
         - AverageThreadTokens: 38.24
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 84.69 MB (88798822)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 680.93 MB (714007426)
         - RowsProduced: 28.78M (28782492)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 11.845ms
         - TotalStorageWaitTime: 8m15s
         - TotalThreadsInvoluntaryContextSwitches: 179 (179)
         - TotalThreadsTotalWallClockTime: 12m12s
           - TotalThreadsSysTime: 449.831ms
           - TotalThreadsUserTime: 23s998ms
         - TotalThreadsVoluntaryContextSwitches: 27.49K (27492)
        Fragment Instance Lifecycle Timings:
           - ExecTime: 19s166ms
             - ExecTreeExecTime: 2s209ms
           - OpenTime: 20.637ms
             - ExecTreeOpenTime: 833.069us
           - PrepareTime: 37.334ms
             - ExecTreePrepareTime: 9.378ms
        DataStreamSender (dst_id=6):(Total: 9s361ms, non-child: 9s361ms, % non-child: 100.00%)
           - BytesSent: 404.85 MB (424513438)
           - NetworkThroughput(*): 247.99 MB/sec
           - OverallThroughput: 43.25 MB/sec
           - PeakMemoryUsage: 177.62 KB (181880)
           - RowsReturned: 28.78M (28782492)
           - SerializeBatchTime: 7s723ms
           - TransmitDataRPCTime: 1s632ms
           - UncompressedRowBatchSize: 576.55 MB (604551892)
        CodeGen:(Total: 45.388ms, non-child: 45.388ms, % non-child: 100.00%)
           - CodegenTime: 1.188ms
           - CompileTime: 2.912ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.84 MB (1929896)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 15.672ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 25.893ms
        HDFS_SCAN_NODE (id=1):(Total: 2s203ms, non-child: 2s203ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:388/3.40 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 388 out of 388
          Hdfs Read Thread Concurrency Bucket: 0:5.263% 1:94.74% 2:0% 3:0% 4:0% 5:0% 6:0%
          File Formats: PARQUET/NONE:776
          BytesRead(500.000ms): 10.52 MB, 20.29 MB, 25.80 MB, 30.25 MB, 35.98 MB, 46.06 MB, 68.92 MB, 80.86 MB, 97.49 MB, 116.13 MB, 136.11 MB, 144.39 MB, 151.96 MB, 160.84 MB, 174.40 MB, 187.23 MB, 199.41 MB, 211.46 MB, 226.74 MB, 236.98 MB, 248.14 MB, 258.62 MB, 265.77 MB, 276.36 MB, 282.98 MB, 299.42 MB, 311.21 MB, 327.77 MB, 334.76 MB, 348.54 MB, 358.54 MB, 374.73 MB, 388.41 MB, 402.72 MB, 415.56 MB, 429.66 MB, 448.27 MB, 452.55 MB
           - FooterProcessingTime: (Avg: 628.978ms ; Min: 1.447ms ; Max: 1s481ms ; Number of samples: 388)
           - AverageHdfsReadThreadConcurrency: 0.95
           - AverageScannerThreadConcurrency: 37.24
           - BytesRead: 452.55 MB (474535168)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 452.55 MB (474535168)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 944 (944)
           - CachedFileHandlesMissCount: 220 (220)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 2 (2)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 388 (388)
           - NumScannerThreadsStarted: 39 (39)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 84.50 MB (88607311)
           - PerReadThreadRawHdfsThroughput: 26.27 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 1s927ms
           - RowBatchQueuePutWaitTime: 3m13s
           - RowsRead: 28.78M (28782492)
           - RowsReturned: 28.78M (28782492)
           - RowsReturnedRate: 13.06 M/sec
           - ScanRangesComplete: 388 (388)
           - ScannerThreadsInvoluntaryContextSwitches: 82 (82)
           - ScannerThreadsTotalWallClockTime: 11m53s
             - MaterializeTupleTime(*): 5s432ms
             - ScannerThreadsSysTime: 322.882ms
             - ScannerThreadsUserTime: 6s878ms
           - ScannerThreadsVoluntaryContextSwitches: 27.40K (27396)
           - TotalRawHdfsReadTime(*): 17s225ms
           - TotalReadThroughput: 23.82 MB/sec


Best Regards
Ferdinand Xu

From: Xu, Cheng A [mailto:cheng.a.xu@intel.com]
Sent: Friday, October 27, 2017 2:57 PM
To: user@impala.incubator.apache.org; mmokhtar@cloudera.com
Subject: RE: performance issue on big table join

Hi Junjie,
The following error is caused by low version of Hive metastore. Can be resolved by upgrade script.

Hi Mostafa
About “compute stats”, is it used to help Impala determine join type (broadcast join or partitioned join) since we have better stats?
https://www.cloudera.com/documentation/enterprise/5-6-x/topics/impala_perf_joins.html#perf_joins
From the plan picture as attached, the join type for store_sales and catalog_sales is already partitioned type.

Also from this picture, the broadcast for catalog_sales is very fast and I think this table is used for build and probe store_sales table. The MAX and avg for broadcast and exchange is much lower than that in Hash join(Max is 2h, avg 2h). Not sure whether we can increase the parallelism for the probe part. If only single thread mode is supported now, any tuning tips to improve this case (two big table join)? Thank you!

Best Regards
Ferdinand Xu

From: 俊杰陈 [mailto:cjjnjust@gmail.com]
Sent: Friday, October 27, 2017 1:38 PM
To: user@impala.incubator.apache.org<mailto:user@impala.incubator.apache.org>; mmokhtar@cloudera.com<mailto:mmokhtar@cloudera.com>
Subject: Re: performance issue on big table join

Thanks, let me put here.

Yes, the query is intended to verify parallelism of partitioned join. We want to know how many fragment instances started for a hash join on a single node, and how many threads in a fragment instance perform the join. I'm not sure whether there is only one thread participate in hash join, since thus it can not maximize the CPU utilization.

The compute stats met following error:

Query: compute stats store_sales
WARNINGS: ImpalaRuntimeException: Error making 'updateTableColumnStatistics' RPC to Hive Metastore:
CAUSED BY: MetaException: Insert of object "org.apache.hadoop.hive.metastore.model.MTableColumnStatistics@49fe55a1<mailto:org.apache.hadoop.hive.metastore.model.MTableColumnStatistics@49fe55a1>" using statement "INSERT INTO `TAB_COL_STATS` (`CS_ID`,`AVG_COL_LEN`,`BIT_VECTOR`,`COLUMN_NAME`,`COLUMN_TYPE`,`DB_NAME`,`BIG_DECIMAL_HIGH_VALUE`,`BIG_DECIMAL_LOW_VALUE`,`DOUBLE_HIGH_VALUE`,`DOUBLE_LOW_VALUE`,`LAST_ANALYZED`,`LONG_HIGH_VALUE`,`LONG_LOW_VALUE`,`MAX_COL_LEN`,`NUM_DISTINCTS`,`NUM_FALSES`,`NUM_NULLS`,`NUM_TRUES`,`TBL_ID`,`TABLE_NAME`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : Unknown column 'BIT_VECTOR' in 'field list'


2017-10-27 12:02 GMT+08:00 Mostafa Mokhtar <mmokhtar@cloudera.com<mailto:mmokhtar@cloudera.com>>:
Hi,

Looks like you are joining store_sales with catalog_sales on item_sk, this
kind of join condition is a many to many, which means the output number of
rows will be much larger then input number of rows, not sure if this is
intended.

Also did you run "compute stats [TABLE_NAME]" on both tables?

For a more comprehensive query try TPCDS Q17

select  i_item_id

       ,i_item_desc

       ,s_state

       ,count(ss_quantity) as store_sales_quantitycount

       ,avg(ss_quantity) as store_sales_quantityave

       ,stddev_samp(ss_quantity) as store_sales_quantitystdev

       ,stddev_samp(ss_quantity)/avg(ss_quantity) as store_sales_quantitycov

       ,count(sr_return_quantity) as store_returns_quantitycount

       ,avg(sr_return_quantity) as store_returns_quantityave

       ,stddev_samp(sr_return_quantity) as store_returns_quantitystdev

       ,stddev_samp(sr_return_quantity)/avg(sr_return_quantity) as
store_returns_quantitycov

       ,count(cs_quantity) as catalog_sales_quantitycount
,avg(cs_quantity) as catalog_sales_quantityave

       ,stddev_samp(cs_quantity) as catalog_sales_quantitystdev

       ,stddev_samp(cs_quantity)/avg(cs_quantity) as catalog_sales_quantitycov

 from store_sales

     ,store_returns

     ,catalog_sales

     ,date_dim d1

     ,date_dim d2

     ,date_dim d3

     ,store

     ,item

 where d1.d_quarter_name = '2000Q1'

   and d1.d_date_sk = ss_sold_date_sk

   and i_item_sk = ss_item_sk

   and s_store_sk = ss_store_sk

   and ss_customer_sk = sr_customer_sk

   and ss_item_sk = sr_item_sk

   and ss_ticket_number = sr_ticket_number

   and sr_returned_date_sk = d2.d_date_sk

   and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3')

   and sr_customer_sk = cs_bill_customer_sk

   and sr_item_sk = cs_item_sk

   and cs_sold_date_sk = d3.d_date_sk

   and d3.d_quarter_name in ('2000Q1','2000Q2','2000Q3')

 group by i_item_id

         ,i_item_desc

         ,s_state

 order by i_item_id

         ,i_item_desc

         ,s_state

limit 100;


I recommend moving this kind of discussion on
user@impala.incubator.apache.org<mailto:user@impala.incubator.apache.org>.

On Thu, Oct 26, 2017 at 7:25 PM, 俊杰陈 <cjjnjust@gmail.com<mailto:cjjnjust@gmail.com>> wrote:

> The profile file is damaged. Here is a screenshot for exec summary
>
> ​
>
> 2017-10-27 10:04 GMT+08:00 俊杰陈 <cjjnjust@gmail.com<mailto:cjjnjust@gmail.com>>:
>
>> Hi Devs
>>
>> I met a performance issue on big table join. The query takes more than 3
>> hours on Impala and only 3 minutes on Spark SQL on the same 5 nodes
>> cluster. when running query,  the left scanner and exchange node are very
>> slow.  Did I miss some key arguments?
>>
>> you can see profile file in attachment.
>>
>>
>> ​
>> --
>> Thanks & Best Regards
>>
>
>
>
> --
> Thanks & Best Regards
>



--
Thanks & Best Regards
Mime
View raw message