kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From farkas@tf-bic.sk <far...@tf-bic.sk>
Subject Re: poor performance on insert into range partitions and scaling
Date Wed, 01 Aug 2018 14:16:22 GMT


On 2018/07/31 20:59:28, Mike Percy <mpercy@apache.org> wrote: 
> Can you post a query profile from Impala for one of the slow insert jobs?
> 
> Mike
> 
> On Tue, Jul 31, 2018 at 12:56 PM Tomas Farkas <farkas@tf-bic.sk> wrote:
> 
> > Hi,
> > wanted share with you the preliminary results of my Kudu testing on AWS
> > Created a set of performance tests for evaluation of different instance
> > types in AWS and different configurations (Kudu separated from Impala, Kudu
> > and Impala on the same nodes); different drive (st1 and gp2) settings and
> > here my results:
> >
> > I was quite dissapointed by the inserts in Step3 see attached sqls,
> >
> > Any hints, ideas, why this does not scale?
> > Thanks
> >
> >
> >
> 


Impala profile:

Query (id=7e441f58868a8585:7452021000000000):
  Summary:
    Session ID: be47b44e2dde497b:fd4a9bd251e2c897
    Session Type: BEESWAX
    Start Time: 2018-08-01 15:40:30.976897000
    End Time: 2018-08-01 15:43:17.236925000
    Query Type: DML
    Query State: FINISHED
    Query Status: OK
    Impala Version: impalad version 2.12.0-cdh5.15.0 RELEASE (build 23f574543323301846b41fa5433690df32efe085)
    User: impala@KUDUTEST.DOMAIN.LOCAL
    Connected User: impala@KUDUTEST.DOMAIN.LOCAL
    Delegated User: 
    Network Address: 10.197.7.176:45846
    Default Db: default
    Sql Statement: insert into test.test_kudu_range 
select 
  id,
  name,
  ban,
  2018,
  iban,
  eventdate,
  eventid,
  city, 
  state,
  lat,
  lon
from test.test_prq
    Coordinator: ip-10-197-10-88.eu-west-1.compute.internal:22000
    Query Options (set by configuration): 
    Query Options (set by configuration and planner): MT_DOP=0
    Plan: 
----------------
Max Per-Host Resource Reservation: Memory=4.00MB
Per-Host Resource Estimates: Memory=832.00MB

F01:PLAN FRAGMENT [KUDU(KuduPartition(id, name, ban, 2018))] hosts=6 instances=6
|  Per-Host Resources: mem-estimate=128.00MB mem-reservation=4.00MB
INSERT INTO KUDU [test.test_kudu_range]
|  mem-estimate=0B mem-reservation=0B
|
02:PARTIAL SORT
|  order by: KuduPartition(id, name, ban, 2018) ASC NULLS LAST, id ASC NULLS LAST, name ASC NULLS LAST, ban ASC NULLS LAST
|  materialized: KuduPartition(id, name, ban, 2018)
|  mem-estimate=128.00MB mem-reservation=4.00MB spill-buffer=2.00MB
|  tuple-ids=2 row-size=248B cardinality=45000000
|
01:EXCHANGE [KUDU(KuduPartition(id, name, ban, 2018))]
|  mem-estimate=0B mem-reservation=0B
|  tuple-ids=0 row-size=244B cardinality=45000000
|
F00:PLAN FRAGMENT [RANDOM] hosts=6 instances=6
Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B
00:SCAN HDFS [test.test_prq, RANDOM]
   partitions=1/1 files=24 size=4.93GB
   stored statistics:
     table: rows=45000000 size=4.93GB
     columns: all
   extrapolated-rows=disabled
   mem-estimate=704.00MB mem-reservation=0B
   tuple-ids=0 row-size=244B cardinality=45000000
----------------
    Estimated Per-Host Mem: 872415232
    Per Host Min Reservation: ip-10-197-10-88.eu-west-1.compute.internal:22000(4.00 MB) ip-10-197-11-142.eu-west-1.compute.internal:22000(4.00 MB) ip-10-197-29-94.eu-west-1.compute.internal:22000(4.00 MB) ip-10-197-3-207.eu-west-1.compute.internal:22000(4.00 MB) ip-10-197-30-21.eu-west-1.compute.internal:22000(4.00 MB) ip-10-197-7-97.eu-west-1.compute.internal:22000(4.00 MB) 
    Request Pool: root.impala
    Admission result: Admitted immediately
    ExecSummary: 
Operator          #Hosts   Avg Time   Max Time   #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail                                   
---------------------------------------------------------------------------------------------------------------------------------------
02:PARTIAL SORT        6   12s495ms   26s075ms  45.00M      45.00M    3.49 GB      128.00 MB                                           
01:EXCHANGE            6  232.833ms  517.001ms  45.00M      45.00M   13.72 MB              0  KUDU(KuduPartition(id, name, ban, 2018)) 
00:SCAN HDFS           6   96.500ms  117.000ms  45.00M      45.00M  774.11 MB      704.00 MB  test.test_prq                            
    Errors: Key already present in Kudu table 'impala::test.test_kudu_range'. (1 of 15442 similar)

    Query Compilation: 4.022ms
       - Metadata of all 2 tables cached: 380.433us (380.433us)
       - Analysis finished: 1.141ms (761.253us)
       - Value transfer graph computed: 1.254ms (113.129us)
       - Single node plan created: 1.780ms (525.298us)
       - Runtime filters computed: 1.819ms (39.444us)
       - Distributed plan created: 2.508ms (688.869us)
       - Lineage info computed: 2.678ms (169.662us)
       - Planning finished: 4.022ms (1.343ms)
    Query Timeline: 2m46s
       - Query submitted: 0.000ns (0.000ns)
       - Planning finished: 5.000ms (5.000ms)
       - Submit for admission: 6.000ms (1.000ms)
       - Completed admission: 6.000ms (0.000ns)
       - Ready to start on 6 backends: 6.000ms (0.000ns)
       - All 6 execution backends (12 fragment instances) started: 7.000ms (1.000ms)
       - Released admission control resources: 2m45s (2m45s)
       - DML data written: 2m45s (1.000ms)
       - DML Metastore update finished: 2m45s (0.000ns)
       - Request finished: 2m45s (0.000ns)
       - Unregister query: 2m46s (431.003ms)
     - ComputeScanRangeAssignmentTimer: 1.000ms
  ImpalaServer:
     - ClientFetchWaitTimer: 431.003ms
     - MetastoreUpdateTimer: 0.000ns
     - RowMaterializationTimer: 0.000ns
  Execution Profile 7e441f58868a8585:7452021000000000:(Total: 2m45s, non-child: 0.000ns, % non-child: 0.00%)
    Number of filters: 0
    Filter routing table: 
 ID  Src. Node  Tgt. Node(s)  Target type  Partition filter  Pending (Expected)  First arrived  Completed   Enabled
-------------------------------------------------------------------------------------------------------------------

    Backend startup latencies: Count: 6, min / max: 0 / 1ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms
    DML Stats: 
Partition: Default
NumModifiedRows: 44984558
NumRowErrors: 15442
  BytesWritten: 0
    Per Node Peak Memory Usage: ip-10-197-10-88.eu-west-1.compute.internal:22000(704.86 MB) ip-10-197-11-142.eu-west-1.compute.internal:22000(3.52 GB) ip-10-197-29-94.eu-west-1.compute.internal:22000(3.53 GB) ip-10-197-3-207.eu-west-1.compute.internal:22000(720.28 MB) ip-10-197-30-21.eu-west-1.compute.internal:22000(3.76 GB) ip-10-197-7-97.eu-west-1.compute.internal:22000(675.40 MB) 
     - FiltersReceived: 0 (0)
     - FinalizationTimer: 0.000ns
     - NumBackends: 6 (6)
     - NumFragmentInstances: 12 (12)
     - NumFragments: 2 (2)
    Averaged Fragment F01:(Total: 1m30s, non-child: 127.333ms, % non-child: 0.14%)
      split sizes:  min: 0, max: 0, avg: 0, stddev: 0
      completion times: min:19s223ms  max:2m45s  mean: 1m30s  stddev:1m11s
      execution rates: min:0.00 /sec  max:0.00 /sec  mean:0.00 /sec  stddev:0.00 /sec
      num instances: 6
       - AverageThreadTokens: 1.00 
       - BloomFilterBytes: 0
       - PeakMemoryUsage: 1.78 GB (1906470029)
       - PeakReservation: 1.75 GB (1878349141)
       - PeakUsedReservation: 0
       - PerHostPeakMemUsage: 2.14 GB (2301629572)
       - RowsProduced: 7.50M (7500000)
       - TotalNetworkReceiveTime: 11s040ms
       - TotalNetworkSendTime: 0.000ns
       - TotalStorageWaitTime: 0.000ns
       - TotalThreadsInvoluntaryContextSwitches: 755 (755)
       - TotalThreadsTotalWallClockTime: 1m30s
         - TotalThreadsSysTime: 4s416ms
         - TotalThreadsUserTime: 37s775ms
       - TotalThreadsVoluntaryContextSwitches: 5.94K (5937)
      Buffer pool:
         - AllocTime: 0.000ns
         - CumulativeAllocationBytes: 0
         - CumulativeAllocations: 0 (0)
         - PeakReservation: 0
         - PeakUnpinnedBytes: 0
         - PeakUsedReservation: 0
         - ReadIoBytes: 0
         - ReadIoOps: 0 (0)
         - ReadIoWaitTime: 0.000ns
         - ReservationLimit: 0
         - WriteIoBytes: 0
         - WriteIoOps: 0 (0)
         - WriteIoWaitTime: 0.000ns
      Fragment Instance Lifecycle Timings:
         - ExecTime: 1m21s
           - ExecTreeExecTime: 14s101ms
         - OpenTime: 9s615ms
           - ExecTreeOpenTime: 9s571ms
         - PrepareTime: 49.500ms
           - ExecTreePrepareTime: 0.000ns
      KuduTableSink:(Total: 1m7s, non-child: 1m7s, % non-child: 100.00%)
         - KuduApplyTimer: 57s540ms
         - NumRowErrors: 2.57K (2573)
         - PeakMemoryUsage: 20.00 MB (20971520)
         - RowsProcessedRate: 55.98 K/sec
         - TotalNumRows: 7.50M (7500000)
      SORT_NODE (id=2):(Total: 23s666ms, non-child: 12s495ms, % non-child: 52.80%)
         - InMemorySortTime: 4s505ms
         - PeakMemoryUsage: 1.75 GB (1878365525)
         - RowsReturned: 7.50M (7500000)
         - RowsReturnedRate: 265.96 K/sec
         - RunsCreated: 1 (1)
         - SortDataSize: 1.74 GB (1873230714)
        Buffer pool:
           - AllocTime: 147.833ms
           - CumulativeAllocationBytes: 1.75 GB (1878349141)
           - CumulativeAllocations: 895 (895)
           - PeakReservation: 1.75 GB (1878349141)
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 1.75 GB (1878349141)
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
      EXCHANGE_NODE (id=1):(Total: 11s171ms, non-child: 11s171ms, % non-child: 100.00%)
         - ConvertRowBatchTime: 106.833ms
         - PeakMemoryUsage: 6.85 MB (7179227)
         - RowsReturned: 7.50M (7500000)
         - RowsReturnedRate: 2.42 M/sec
        Buffer pool:
           - AllocTime: 59.333ms
           - CumulativeAllocationBytes: 2.35 GB (2518056960)
           - CumulativeAllocations: 122.95K (122953)
           - PeakReservation: 6.79 MB (7124309)
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 6.79 MB (7124309)
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Dequeue:
           - FirstBatchWaitTime: 9s570ms
           - TotalBytesDequeued: 1.77 GB (1903230714)
           - TotalGetBatchTime: 11s037ms
             - DataWaitTime: 10s938ms
        Enqueue:
           - DeserializeRowBatchTime: 1s764ms
           - TotalBatchesEnqueued: 61.48K (61476)
           - TotalBatchesReceived: 61.48K (61476)
           - TotalBytesReceived: 1.24 GB (1333898634)
           - TotalEarlySenders: 0 (0)
           - TotalEosReceived: 6 (6)
           - TotalHasDeferredRPCsTime: 1s897ms
           - TotalRPCsDeferred: 11.56K (11562)
      CodeGen:(Total: 44.000ms, non-child: 44.000ms, % non-child: 100.00%)
         - CodegenTime: 833.334us
         - CompileTime: 5.666ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 1.95 MB (2039944)
         - NumFunctions: 32 (32)
         - NumInstructions: 638 (638)
         - OptimizationTime: 24.666ms
         - PeakMemoryUsage: 319.00 KB (326656)
         - PrepareTime: 12.833ms
    Fragment F01:
      Instance 7e441f58868a8585:745202100000000a (host=ip-10-197-29-94.eu-west-1.compute.internal:22000):(Total: 2m45s, non-child: 130.000ms, % non-child: 0.08%)
        Fragment Instance Lifecycle Event Timeline: 2m45s
           - Prepare Finished: 53.000ms (53.000ms)
           - Open Finished: 108.000ms (55.000ms)
           - First Batch Produced: 28s518ms (28s410ms)
           - First Batch Sent: 28s522ms (4.000ms)
           - ExecInternal Finished: 2m45s (2m17s)
        MemoryUsage(4s000ms): 406.92 MB, 1.19 GB, 2.03 GB, 2.79 GB, 3.41 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB
        ThreadUsage(4s000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.00 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 3.53 GB (3787104739)
         - PeakReservation: 3.49 GB (3751804928)
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 3.53 GB (3787104739)
         - RowsProduced: 15.00M (15003125)
         - TotalNetworkReceiveTime: 2s139ms
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 1.92K (1923)
         - TotalThreadsTotalWallClockTime: 2m45s
           - TotalThreadsSysTime: 8s959ms
           - TotalThreadsUserTime: 1m18s
         - TotalThreadsVoluntaryContextSwitches: 6.17K (6169)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 2m45s
             - ExecTreeExecTime: 28s483ms
           - OpenTime: 55.000ms
             - ExecTreeOpenTime: 7.000ms
           - PrepareTime: 52.000ms
             - ExecTreePrepareTime: 0.000ns
        KuduTableSink:(Total: 2m17s, non-child: 2m17s, % non-child: 100.00%)
           - KuduApplyTimer: 1m57s
           - NumRowErrors: 5.16K (5159)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 109.38 K/sec
           - TotalNumRows: 15.00M (15003125)
        SORT_NODE (id=2):(Total: 28s481ms, non-child: 26s075ms, % non-child: 91.55%)
          SortType: Partial
          ExecOption: Codegen Enabled
           - NumRowsPerRun: (Avg: 15.00M (15003125) ; Min: 15.00M (15003125) ; Max: 15.00M (15003125) ; Number of samples: 1)
           - InMemorySortTime: 9s293ms
           - PeakMemoryUsage: 3.49 GB (3751821312)
           - RowsReturned: 15.00M (15003125)
           - RowsReturnedRate: 526.77 K/sec
           - RunsCreated: 1 (1)
           - SortDataSize: 3.49 GB (3747240938)
          Buffer pool:
             - AllocTime: 103.000ms
             - CumulativeAllocationBytes: 3.49 GB (3751804928)
             - CumulativeAllocations: 1.79K (1789)
             - PeakReservation: 3.49 GB (3751804928)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 3.49 GB (3751804928)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
        EXCHANGE_NODE (id=1):(Total: 2s406ms, non-child: 517.001ms, % non-child: 21.49%)
           - ConvertRowBatchTime: 225.000ms
           - PeakMemoryUsage: 13.72 MB (14390403)
           - RowsReturned: 15.00M (15003125)
           - RowsReturnedRate: 6.24 M/sec
          Buffer pool:
             - AllocTime: 124.000ms
             - CumulativeAllocationBytes: 4.69 GB (5037178880)
             - CumulativeAllocations: 245.96K (245960)
             - PeakReservation: 13.64 MB (14303232)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 13.64 MB (14303232)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Dequeue:
            BytesDequeued(4s000ms): 489.70 MB, 1.28 GB, 2.12 GB, 2.89 GB, 3.47 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB
             - FirstBatchWaitTime: 7.000ms
             - TotalBytesDequeued: 3.55 GB (3807253438)
             - TotalGetBatchTime: 2s134ms
               - DataWaitTime: 1s890ms
          Enqueue:
            BytesReceived(4s000ms): 347.95 MB, 917.64 MB, 1.49 GB, 2.03 GB, 2.43 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB
            DeferredQueueSize(4s000ms): 0, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
             - DispatchTime: (Avg: 16.831us ; Min: 3.269us ; Max: 5.228ms ; Number of samples: 122980)
             - DeserializeRowBatchTime: 3s661ms
             - TotalBatchesEnqueued: 122.98K (122980)
             - TotalBatchesReceived: 122.98K (122980)
             - TotalBytesReceived: 2.49 GB (2668357754)
             - TotalEarlySenders: 0 (0)
             - TotalEosReceived: 6 (6)
             - TotalHasDeferredRPCsTime: 6s631ms
             - TotalRPCsDeferred: 36.40K (36395)
        CodeGen:(Total: 46.000ms, non-child: 46.000ms, % non-child: 100.00%)
           - CodegenTime: 1.000ms
           - CompileTime: 8.000ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 32 (32)
           - NumInstructions: 638 (638)
           - OptimizationTime: 26.000ms
           - PeakMemoryUsage: 319.00 KB (326656)
           - PrepareTime: 12.000ms
      Instance 7e441f58868a8585:7452021000000006 (host=ip-10-197-30-21.eu-west-1.compute.internal:22000):(Total: 2m41s, non-child: 152.998ms, % non-child: 0.09%)
        Fragment Instance Lifecycle Event Timeline: 2m41s
           - Prepare Finished: 48.999ms (48.999ms)
           - Open Finished: 106.999ms (57.999ms)
           - First Batch Produced: 28s199ms (28s092ms)
           - First Batch Sent: 28s202ms (2.999ms)
           - ExecInternal Finished: 2m41s (2m12s)
        MemoryUsage(4s000ms): 496.29 MB, 1.27 GB, 2.10 GB, 2.86 GB, 3.45 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB, 3.53 GB
        ThreadUsage(4s000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.00 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 3.53 GB (3786981859)
         - PeakReservation: 3.49 GB (3751804928)
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 3.76 GB (4037232486)
         - RowsProduced: 15.01M (15006663)
         - TotalNetworkReceiveTime: 3s501ms
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 1.34K (1340)
         - TotalThreadsTotalWallClockTime: 2m41s
           - TotalThreadsSysTime: 8s804ms
           - TotalThreadsUserTime: 1m14s
         - TotalThreadsVoluntaryContextSwitches: 15.37K (15372)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 2m41s
             - ExecTreeExecTime: 28s172ms
           - OpenTime: 57.999ms
             - ExecTreeOpenTime: 6.999ms
           - PrepareTime: 48.999ms
             - ExecTreePrepareTime: 0.000ns
        KuduTableSink:(Total: 2m12s, non-child: 2m12s, % non-child: 100.00%)
           - KuduApplyTimer: 1m54s
           - NumRowErrors: 5.11K (5108)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 112.98 K/sec
           - TotalNumRows: 15.01M (15006663)
        SORT_NODE (id=2):(Total: 28s162ms, non-child: 24s414ms, % non-child: 86.69%)
          SortType: Partial
          ExecOption: Codegen Enabled
           - NumRowsPerRun: (Avg: 15.01M (15006663) ; Min: 15.01M (15006663) ; Max: 15.01M (15006663) ; Number of samples: 1)
           - InMemorySortTime: 8s976ms
           - PeakMemoryUsage: 3.49 GB (3751821312)
           - RowsReturned: 15.01M (15006663)
           - RowsReturnedRate: 532.85 K/sec
           - RunsCreated: 1 (1)
           - SortDataSize: 3.49 GB (3748143215)
          Buffer pool:
             - AllocTime: 485.996ms
             - CumulativeAllocationBytes: 3.49 GB (3751804928)
             - CumulativeAllocations: 1.79K (1789)
             - PeakReservation: 3.49 GB (3751804928)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 3.49 GB (3751804928)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
        EXCHANGE_NODE (id=1):(Total: 3s747ms, non-child: 444.996ms, % non-child: 11.87%)
           - ConvertRowBatchTime: 199.998ms
           - PeakMemoryUsage: 13.65 MB (14313119)
           - RowsReturned: 15.01M (15006663)
           - RowsReturnedRate: 4.00 M/sec
          Buffer pool:
             - AllocTime: 125.999ms
             - CumulativeAllocationBytes: 4.69 GB (5038350336)
             - CumulativeAllocations: 246.02K (246016)
             - PeakReservation: 13.52 MB (14180352)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 13.52 MB (14180352)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Dequeue:
            BytesDequeued(4s000ms): 466.52 MB, 1.25 GB, 2.10 GB, 2.87 GB, 3.47 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB, 3.55 GB
             - FirstBatchWaitTime: 5.999ms
             - TotalBytesDequeued: 3.55 GB (3808169867)
             - TotalGetBatchTime: 3s494ms
               - DataWaitTime: 3s302ms
          Enqueue:
            BytesReceived(4s000ms): 330.36 MB, 901.24 MB, 1.48 GB, 2.01 GB, 2.43 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB, 2.49 GB
            DeferredQueueSize(4s000ms): 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
             - DispatchTime: (Avg: 18.758us ; Min: 4.110us ; Max: 9.230ms ; Number of samples: 123008)
             - DeserializeRowBatchTime: 3s491ms
             - TotalBatchesEnqueued: 123.01K (123008)
             - TotalBatchesReceived: 123.01K (123008)
             - TotalBytesReceived: 2.49 GB (2668950891)
             - TotalEarlySenders: 0 (0)
             - TotalEosReceived: 6 (6)
             - TotalHasDeferredRPCsTime: 2s186ms
             - TotalRPCsDeferred: 15.90K (15904)
        CodeGen:(Total: 50.999ms, non-child: 50.999ms, % non-child: 100.00%)
           - CodegenTime: 999.992us
           - CompileTime: 4.999ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 32 (32)
           - NumInstructions: 638 (638)
           - OptimizationTime: 26.999ms
           - PeakMemoryUsage: 319.00 KB (326656)
           - PrepareTime: 17.999ms
      Instance 7e441f58868a8585:7452021000000008 (host=ip-10-197-11-142.eu-west-1.compute.internal:22000):(Total: 2m40s, non-child: 333.001ms, % non-child: 0.21%)
        Fragment Instance Lifecycle Event Timeline: 2m40s
           - Prepare Finished: 49.000ms (49.000ms)
           - Open Finished: 107.000ms (58.000ms)
           - First Batch Produced: 27s989ms (27s882ms)
           - First Batch Sent: 27s992ms (3.000ms)
           - ExecInternal Finished: 2m40s (2m12s)
        MemoryUsage(4s000ms): 500.63 MB, 1.27 GB, 2.11 GB, 2.87 GB, 3.44 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB, 3.52 GB
        ThreadUsage(4s000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.00 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 3.52 GB (3782869475)
         - PeakReservation: 3.49 GB (3747610624)
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 3.52 GB (3782869475)
         - RowsProduced: 14.99M (14990212)
         - TotalNetworkReceiveTime: 3s205ms
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 1.27K (1268)
         - TotalThreadsTotalWallClockTime: 2m40s
           - TotalThreadsSysTime: 8s733ms
           - TotalThreadsUserTime: 1m13s
         - TotalThreadsVoluntaryContextSwitches: 14.08K (14076)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 2m40s
             - ExecTreeExecTime: 27s952ms
           - OpenTime: 58.000ms
             - ExecTreeOpenTime: 17.000ms
           - PrepareTime: 48.000ms
             - ExecTreePrepareTime: 0.000ns
        KuduTableSink:(Total: 2m12s, non-child: 2m12s, % non-child: 100.00%)
           - KuduApplyTimer: 1m53s
           - NumRowErrors: 5.17K (5175)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 113.55 K/sec
           - TotalNumRows: 14.99M (14990212)
        SORT_NODE (id=2):(Total: 27s960ms, non-child: 24s481ms, % non-child: 87.56%)
          SortType: Partial
          ExecOption: Codegen Enabled
           - NumRowsPerRun: (Avg: 14.99M (14990212) ; Min: 14.99M (14990212) ; Max: 14.99M (14990212) ; Number of samples: 1)
           - InMemorySortTime: 8s765ms
           - PeakMemoryUsage: 3.49 GB (3747627008)
           - RowsReturned: 14.99M (14990212)
           - RowsReturnedRate: 536.13 K/sec
           - RunsCreated: 1 (1)
           - SortDataSize: 3.49 GB (3744000131)
          Buffer pool:
             - AllocTime: 298.001ms
             - CumulativeAllocationBytes: 3.49 GB (3747610624)
             - CumulativeAllocations: 1.79K (1787)
             - PeakReservation: 3.49 GB (3747610624)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 3.49 GB (3747610624)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
        EXCHANGE_NODE (id=1):(Total: 3s479ms, non-child: 435.002ms, % non-child: 12.50%)
           - ConvertRowBatchTime: 216.001ms
           - PeakMemoryUsage: 13.71 MB (14371845)
           - RowsReturned: 14.99M (14990212)
           - RowsReturnedRate: 4.31 M/sec
          Buffer pool:
             - AllocTime: 106.000ms
             - CumulativeAllocationBytes: 4.69 GB (5032812544)
             - CumulativeAllocations: 245.75K (245746)
             - PeakReservation: 13.60 MB (14262272)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 13.60 MB (14262272)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Dequeue:
            BytesDequeued(4s000ms): 472.38 MB, 1.25 GB, 2.10 GB, 2.87 GB, 3.46 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB, 3.54 GB
             - FirstBatchWaitTime: 17.000ms
             - TotalBytesDequeued: 3.54 GB (3803960979)
             - TotalGetBatchTime: 3s198ms
               - DataWaitTime: 3s044ms
          Enqueue:
            BytesReceived(4s000ms): 334.01 MB, 903.81 MB, 1.48 GB, 2.02 GB, 2.43 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB, 2.48 GB
            DeferredQueueSize(4s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
             - DispatchTime: (Avg: 17.899us ; Min: 3.826us ; Max: 5.040ms ; Number of samples: 122873)
             - DeserializeRowBatchTime: 3s434ms
             - TotalBatchesEnqueued: 122.87K (122873)
             - TotalBatchesReceived: 122.87K (122873)
             - TotalBytesReceived: 2.48 GB (2666083162)
             - TotalEarlySenders: 0 (0)
             - TotalEosReceived: 6 (6)
             - TotalHasDeferredRPCsTime: 2s563ms
             - TotalRPCsDeferred: 17.08K (17077)
        CodeGen:(Total: 40.000ms, non-child: 40.000ms, % non-child: 100.00%)
           - CodegenTime: 1.000ms
           - CompileTime: 5.000ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 32 (32)
           - NumInstructions: 638 (638)
           - OptimizationTime: 23.000ms
           - PeakMemoryUsage: 319.00 KB (326656)
           - PrepareTime: 11.000ms
      Instance 7e441f58868a8585:745202100000000b (host=ip-10-197-10-88.eu-west-1.compute.internal:22000):(Total: 19s224ms, non-child: 49.000ms, % non-child: 0.25%)
        Fragment Instance Lifecycle Event Timeline: 19s224ms
           - Prepare Finished: 49.000ms (49.000ms)
           - Open Finished: 19s224ms (19s175ms)
           - First Batch Produced: 19s224ms (0.000ns)
           - First Batch Sent: 19s224ms (0.000ns)
           - ExecInternal Finished: 19s224ms (0.000ns)
        MemoryUsage(500.000ms): 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB
        ThreadUsage(500.000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.00 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 26.02 MB (27288035)
         - PeakReservation: 6.00 MB (6291456)
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 704.86 MB (739094888)
         - RowsProduced: 0 (0)
         - TotalNetworkReceiveTime: 19s135ms
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 0 (0)
         - TotalThreadsTotalWallClockTime: 19s175ms
           - TotalThreadsSysTime: 0.000ns
           - TotalThreadsUserTime: 40.199ms
         - TotalThreadsVoluntaryContextSwitches: 2 (2)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 0.000ns
           - OpenTime: 19s175ms
             - ExecTreeOpenTime: 19s135ms
           - PrepareTime: 49.000ms
             - ExecTreePrepareTime: 0.000ns
        KuduTableSink:
           - KuduApplyTimer: 0.000ns
           - NumRowErrors: 0 (0)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 0
           - TotalNumRows: 0 (0)
        SORT_NODE (id=2):(Total: 19s135ms, non-child: 0.000ns, % non-child: 0.00%)
          SortType: Partial
          ExecOption: Codegen Enabled
           - NumRowsPerRun: (Avg: 0 (0) ; Min: 0 (0) ; Max: 0 (0) ; Number of samples: 1)
           - InMemorySortTime: 0.000ns
           - PeakMemoryUsage: 6.02 MB (6307840)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - RunsCreated: 1 (1)
           - SortDataSize: 0
          Buffer pool:
             - AllocTime: 0.000ns
             - CumulativeAllocationBytes: 6.00 MB (6291456)
             - CumulativeAllocations: 3 (3)
             - PeakReservation: 6.00 MB (6291456)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 6.00 MB (6291456)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
        EXCHANGE_NODE (id=1):(Total: 19s135ms, non-child: 0.000ns, % non-child: 0.00%)
           - ConvertRowBatchTime: 0.000ns
           - PeakMemoryUsage: 0
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
          Buffer pool:
             - AllocTime: 0.000ns
             - CumulativeAllocationBytes: 0
             - CumulativeAllocations: 0 (0)
             - PeakReservation: 0
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 0
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Dequeue:
            BytesDequeued(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
             - FirstBatchWaitTime: 19s135ms
             - TotalBytesDequeued: 0
             - TotalGetBatchTime: 19s135ms
               - DataWaitTime: 19s135ms
          Enqueue:
            BytesReceived(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
            DeferredQueueSize(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
            - DispatchTime: 0.000ns (Number of samples: 0)
             - DeserializeRowBatchTime: 0.000ns
             - TotalBatchesEnqueued: 0 (0)
             - TotalBatchesReceived: 0 (0)
             - TotalBytesReceived: 0
             - TotalEarlySenders: 0 (0)
             - TotalEosReceived: 6 (6)
             - TotalHasDeferredRPCsTime: 0.000ns
             - TotalRPCsDeferred: 0 (0)
        CodeGen:(Total: 40.000ms, non-child: 40.000ms, % non-child: 100.00%)
           - CodegenTime: 1.000ms
           - CompileTime: 4.000ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 32 (32)
           - NumInstructions: 638 (638)
           - OptimizationTime: 23.000ms
           - PeakMemoryUsage: 319.00 KB (326656)
           - PrepareTime: 12.000ms
      Instance 7e441f58868a8585:7452021000000007 (host=ip-10-197-3-207.eu-west-1.compute.internal:22000):(Total: 19s223ms, non-child: 48.000ms, % non-child: 0.25%)
        Fragment Instance Lifecycle Event Timeline: 19s223ms
           - Prepare Finished: 48.000ms (48.000ms)
           - Open Finished: 19s223ms (19s175ms)
           - First Batch Produced: 19s223ms (0.000ns)
           - First Batch Sent: 19s223ms (0.000ns)
           - ExecInternal Finished: 19s223ms (0.000ns)
        MemoryUsage(500.000ms): 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB
        ThreadUsage(500.000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.00 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 26.02 MB (27288035)
         - PeakReservation: 6.00 MB (6291456)
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 720.28 MB (755267876)
         - RowsProduced: 0 (0)
         - TotalNetworkReceiveTime: 19s135ms
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 0 (0)
         - TotalThreadsTotalWallClockTime: 19s175ms
           - TotalThreadsSysTime: 2.013ms
           - TotalThreadsUserTime: 37.565ms
         - TotalThreadsVoluntaryContextSwitches: 3 (3)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 0.000ns
           - OpenTime: 19s175ms
             - ExecTreeOpenTime: 19s135ms
           - PrepareTime: 48.000ms
             - ExecTreePrepareTime: 0.000ns
        KuduTableSink:
           - KuduApplyTimer: 0.000ns
           - NumRowErrors: 0 (0)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 0
           - TotalNumRows: 0 (0)
        SORT_NODE (id=2):(Total: 19s135ms, non-child: 0.000ns, % non-child: 0.00%)
          SortType: Partial
          ExecOption: Codegen Enabled
           - NumRowsPerRun: (Avg: 0 (0) ; Min: 0 (0) ; Max: 0 (0) ; Number of samples: 1)
           - InMemorySortTime: 0.000ns
           - PeakMemoryUsage: 6.02 MB (6307840)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - RunsCreated: 1 (1)
           - SortDataSize: 0
          Buffer pool:
             - AllocTime: 0.000ns
             - CumulativeAllocationBytes: 6.00 MB (6291456)
             - CumulativeAllocations: 3 (3)
             - PeakReservation: 6.00 MB (6291456)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 6.00 MB (6291456)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
        EXCHANGE_NODE (id=1):(Total: 19s135ms, non-child: 0.000ns, % non-child: 0.00%)
           - ConvertRowBatchTime: 0.000ns
           - PeakMemoryUsage: 0
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
          Buffer pool:
             - AllocTime: 0.000ns
             - CumulativeAllocationBytes: 0
             - CumulativeAllocations: 0 (0)
             - PeakReservation: 0
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 0
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Dequeue:
            BytesDequeued(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
             - FirstBatchWaitTime: 19s135ms
             - TotalBytesDequeued: 0
             - TotalGetBatchTime: 19s135ms
               - DataWaitTime: 19s135ms
          Enqueue:
            BytesReceived(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
            DeferredQueueSize(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
            - DispatchTime: 0.000ns (Number of samples: 0)
             - DeserializeRowBatchTime: 0.000ns
             - TotalBatchesEnqueued: 0 (0)
             - TotalBatchesReceived: 0 (0)
             - TotalBytesReceived: 0
             - TotalEarlySenders: 0 (0)
             - TotalEosReceived: 6 (6)
             - TotalHasDeferredRPCsTime: 0.000ns
             - TotalRPCsDeferred: 0 (0)
        CodeGen:(Total: 40.000ms, non-child: 40.000ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 5.000ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 32 (32)
           - NumInstructions: 638 (638)
           - OptimizationTime: 22.000ms
           - PeakMemoryUsage: 319.00 KB (326656)
           - PrepareTime: 12.000ms
      Instance 7e441f58868a8585:7452021000000009 (host=ip-10-197-7-97.eu-west-1.compute.internal:22000):(Total: 19s222ms, non-child: 50.999ms, % non-child: 0.27%)
        Fragment Instance Lifecycle Event Timeline: 19s222ms
           - Prepare Finished: 50.999ms (50.999ms)
           - Open Finished: 19s222ms (19s171ms)
           - First Batch Produced: 19s222ms (0.000ns)
           - First Batch Sent: 19s222ms (0.000ns)
           - ExecInternal Finished: 19s222ms (0.000ns)
        MemoryUsage(500.000ms): 335.00 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB, 16.47 KB
        ThreadUsage(500.000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.00 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 26.02 MB (27288035)
         - PeakReservation: 6.00 MB (6291456)
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 675.40 MB (708207971)
         - RowsProduced: 0 (0)
         - TotalNetworkReceiveTime: 19s124ms
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 4 (4)
         - TotalThreadsTotalWallClockTime: 19s171ms
           - TotalThreadsSysTime: 1.006ms
           - TotalThreadsUserTime: 45.570ms
         - TotalThreadsVoluntaryContextSwitches: 2 (2)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 0.000ns
           - OpenTime: 19s171ms
             - ExecTreeOpenTime: 19s124ms
           - PrepareTime: 50.999ms
             - ExecTreePrepareTime: 0.000ns
        KuduTableSink:
           - KuduApplyTimer: 0.000ns
           - NumRowErrors: 0 (0)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 0
           - TotalNumRows: 0 (0)
        SORT_NODE (id=2):(Total: 19s124ms, non-child: 0.000ns, % non-child: 0.00%)
          SortType: Partial
          ExecOption: Codegen Enabled
           - NumRowsPerRun: (Avg: 0 (0) ; Min: 0 (0) ; Max: 0 (0) ; Number of samples: 1)
           - InMemorySortTime: 0.000ns
           - PeakMemoryUsage: 6.02 MB (6307840)
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
           - RunsCreated: 1 (1)
           - SortDataSize: 0
          Buffer pool:
             - AllocTime: 0.000ns
             - CumulativeAllocationBytes: 6.00 MB (6291456)
             - CumulativeAllocations: 3 (3)
             - PeakReservation: 6.00 MB (6291456)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 6.00 MB (6291456)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
        EXCHANGE_NODE (id=1):(Total: 19s124ms, non-child: 0.000ns, % non-child: 0.00%)
           - ConvertRowBatchTime: 0.000ns
           - PeakMemoryUsage: 0
           - RowsReturned: 0 (0)
           - RowsReturnedRate: 0
          Buffer pool:
             - AllocTime: 0.000ns
             - CumulativeAllocationBytes: 0
             - CumulativeAllocations: 0 (0)
             - PeakReservation: 0
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 0
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Dequeue:
            BytesDequeued(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
             - FirstBatchWaitTime: 19s124ms
             - TotalBytesDequeued: 0
             - TotalGetBatchTime: 19s124ms
               - DataWaitTime: 19s124ms
          Enqueue:
            BytesReceived(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
            DeferredQueueSize(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
            - DispatchTime: 0.000ns (Number of samples: 0)
             - DeserializeRowBatchTime: 0.000ns
             - TotalBatchesEnqueued: 0 (0)
             - TotalBatchesReceived: 0 (0)
             - TotalBytesReceived: 0
             - TotalEarlySenders: 0 (0)
             - TotalEosReceived: 6 (6)
             - TotalHasDeferredRPCsTime: 0.000ns
             - TotalRPCsDeferred: 0 (0)
        CodeGen:(Total: 46.999ms, non-child: 46.999ms, % non-child: 100.00%)
           - CodegenTime: 999.999us
           - CompileTime: 6.999ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 32 (32)
           - NumInstructions: 638 (638)
           - OptimizationTime: 26.999ms
           - PeakMemoryUsage: 319.00 KB (326656)
           - PrepareTime: 11.999ms
    Averaged Fragment F00:(Total: 16s864ms, non-child: 20.666ms, % non-child: 0.12%)
      split sizes:  min: 663.66 MB, max: 918.23 MB, avg: 841.66 MB, stddev: 86.23 MB
      completion times: min:13s276ms  max:19s224ms  mean: 16s864ms  stddev:1s777ms
      execution rates: min:47.38 MB/sec  max:53.54 MB/sec  mean:49.97 MB/sec  stddev:2.15 MB/sec
      num instances: 6
       - AverageThreadTokens: 4.48 
       - BloomFilterBytes: 0
       - PeakMemoryUsage: 687.06 MB (720431332)
       - PeakReservation: 0
       - PeakUsedReservation: 0
       - PerHostPeakMemUsage: 1.99 GB (2136129084)
       - RowsProduced: 7.50M (7500000)
       - TotalNetworkReceiveTime: 0.000ns
       - TotalNetworkSendTime: 5s256ms
       - TotalStorageWaitTime: 131.500ms
       - TotalThreadsInvoluntaryContextSwitches: 1.25K (1252)
       - TotalThreadsTotalWallClockTime: 1m16s
         - TotalThreadsSysTime: 591.044ms
         - TotalThreadsUserTime: 13s372ms
       - TotalThreadsVoluntaryContextSwitches: 17.10K (17097)
      Buffer pool:
         - AllocTime: 0.000ns
         - CumulativeAllocationBytes: 0
         - CumulativeAllocations: 0 (0)
         - PeakReservation: 0
         - PeakUnpinnedBytes: 0
         - PeakUsedReservation: 0
         - ReadIoBytes: 0
         - ReadIoOps: 0 (0)
         - ReadIoWaitTime: 0.000ns
         - ReservationLimit: 0
         - WriteIoBytes: 0
         - WriteIoOps: 0 (0)
         - WriteIoWaitTime: 0.000ns
      Fragment Instance Lifecycle Timings:
         - ExecTime: 16s790ms
           - ExecTreeExecTime: 101.333ms
         - OpenTime: 23.166ms
           - ExecTreeOpenTime: 0.000ns
         - PrepareTime: 50.500ms
           - ExecTreePrepareTime: 0.000ns
      KrpcDataStreamSender (dst_id=1):(Total: 16s724ms, non-child: 16s724ms, % non-child: 100.00%)
         - EosSent: 6 (6)
         - NetworkThroughput: 98.51 MB/sec
         - PeakMemoryUsage: 185.72 KB (190176)
         - RowsSent: 7.50M (7500000)
         - RpcFailure: 0 (0)
         - RpcRetry: 0 (0)
         - SerializeBatchTime: 5s558ms
         - TotalBytesSent: 1.24 GB (1333898634)
         - UncompressedRowBatchSize: 1.77 GB (1903230714)
      HDFS_SCAN_NODE (id=0):(Total: 96.500ms, non-child: 96.500ms, % non-child: 100.00%)
         - AverageHdfsReadThreadConcurrency: 0.01 
         - AverageScannerThreadConcurrency: 3.50 
         - BytesRead: 842.04 MB (882945552)
         - BytesReadDataNodeCache: 0
         - BytesReadLocal: 842.04 MB (882945552)
         - BytesReadRemoteUnexpected: 0
         - BytesReadShortCircuit: 842.04 MB (882945552)
         - CachedFileHandlesHitCount: 132 (132)
         - CachedFileHandlesMissCount: 0 (0)
         - CollectionItemsRead: 0 (0)
         - DecompressionTime: 1s248ms
         - MaxCompressedTextFileLength: 0
         - NumColumns: 10 (10)
         - NumDictFilteredRowGroups: 0 (0)
         - NumDisksAccessed: 2 (2)
         - NumRowGroups: 4 (4)
         - NumScannerThreadsStarted: 4 (4)
         - NumScannersWithNoReads: 0 (0)
         - NumStatsFilteredRowGroups: 0 (0)
         - PeakMemoryUsage: 686.69 MB (720051139)
         - PerReadThreadRawHdfsThroughput: 2.16 GB/sec
         - RemoteScanRanges: 0 (0)
         - RowBatchQueueGetWaitTime: 35.000ms
         - RowBatchQueuePutWaitTime: 57s002ms
         - RowsRead: 7.50M (7500000)
         - RowsReturned: 7.50M (7500000)
         - RowsReturnedRate: 80.32 M/sec
         - ScanRangesComplete: 4 (4)
         - ScannerThreadsInvoluntaryContextSwitches: 156 (156)
         - ScannerThreadsTotalWallClockTime: 59s411ms
           - MaterializeTupleTime(*): 818.668ms
           - ScannerThreadsSysTime: 129.643ms
           - ScannerThreadsUserTime: 2s185ms
         - ScannerThreadsVoluntaryContextSwitches: 7.27K (7270)
         - TotalRawHdfsOpenFileTime(*): 499.996us
         - TotalRawHdfsReadTime(*): 381.001ms
         - TotalReadThroughput: 50.04 MB/sec
      CodeGen:(Total: 23.166ms, non-child: 23.166ms, % non-child: 100.00%)
         - CodegenTime: 500.003us
         - CompileTime: 4.000ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 1.95 MB (2039944)
         - NumFunctions: 7 (7)
         - NumInstructions: 105 (105)
         - OptimizationTime: 5.666ms
         - PeakMemoryUsage: 52.50 KB (53760)
         - PrepareTime: 13.000ms
    Fragment F00:
      Instance 7e441f58868a8585:7452021000000000 (host=ip-10-197-30-21.eu-west-1.compute.internal:22000):(Total: 19s223ms, non-child: 27.999ms, % non-child: 0.15%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2/406.63 MB 1:2/507.55 MB 
        Fragment Instance Lifecycle Event Timeline: 19s223ms
           - Prepare Finished: 49.999ms (49.999ms)
           - Open Finished: 80.999ms (30.999ms)
           - First Batch Produced: 114.999ms (33.999ms)
           - First Batch Sent: 116.999ms (1.999ms)
           - ExecInternal Finished: 19s223ms (19s106ms)
        MemoryUsage(500.000ms): 714.21 MB, 720.67 MB, 721.47 MB, 721.63 MB, 721.93 MB, 722.25 MB, 722.25 MB, 716.29 MB, 708.21 MB, 708.59 MB, 708.70 MB, 708.69 MB, 708.61 MB, 708.85 MB, 700.93 MB, 693.00 MB, 661.31 MB, 637.85 MB, 621.94 MB, 621.88 MB, 605.94 MB, 582.18 MB, 582.13 MB, 582.18 MB, 570.18 MB, 560.14 MB, 481.03 MB, 475.95 MB, 476.17 MB, 459.81 MB, 444.04 MB, 419.97 MB, 356.36 MB, 340.45 MB, 308.42 MB, 300.68 MB, 236.62 MB, 140.70 MB
        ThreadUsage(500.000ms): 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3
         - AverageThreadTokens: 4.68 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 722.47 MB (757560400)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 3.76 GB (4037232486)
         - RowsProduced: 8.13M (8134450)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 5s841ms
         - TotalStorageWaitTime: 134.998ms
         - TotalThreadsInvoluntaryContextSwitches: 2.32K (2324)
         - TotalThreadsTotalWallClockTime: 1m29s
           - TotalThreadsSysTime: 856.226ms
           - TotalThreadsUserTime: 15s315ms
         - TotalThreadsVoluntaryContextSwitches: 17.82K (17825)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 19s142ms
             - ExecTreeExecTime: 122.999ms
           - OpenTime: 30.999ms
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 49.999ms
             - ExecTreePrepareTime: 0.000ns
        KrpcDataStreamSender (dst_id=1):(Total: 19s050ms, non-child: 13s208ms, % non-child: 69.33%)
          BytesSent(500.000ms): 22.84 MB, 64.52 MB, 101.06 MB, 132.84 MB, 170.18 MB, 195.54 MB, 231.15 MB, 267.65 MB, 293.64 MB, 315.66 MB, 342.16 MB, 381.38 MB, 409.68 MB, 437.93 MB, 473.86 MB, 515.57 MB, 557.72 MB, 601.94 MB, 649.50 MB, 684.68 MB, 724.47 MB, 757.78 MB, 781.73 MB, 807.80 MB, 836.11 MB, 860.73 MB, 886.71 MB, 920.85 MB, 961.65 MB, 994.13 MB, 1.00 GB, 1.03 GB, 1.06 GB, 1.10 GB, 1.15 GB, 1.21 GB, 1.26 GB, 1.31 GB
           - EosSent: 6 (6)
           - NetworkThroughput: 99.22 MB/sec
           - PeakMemoryUsage: 185.72 KB (190176)
           - RowsSent: 8.13M (8134450)
           - RpcFailure: 0 (0)
           - RpcRetry: 0 (0)
           - SerializeBatchTime: 6s371ms
           - TotalBytesSent: 1.35 GB (1447999819)
           - UncompressedRowBatchSize: 1.93 GB (2067278508)
        HDFS_SCAN_NODE (id=0):(Total: 113.999ms, non-child: 113.999ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2/406.63 MB 1:2/507.55 MB 
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4 out of 4
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 
          File Formats: PARQUET/SNAPPY:4 
          BytesRead(500.000ms): 621.00 MB, 621.00 MB, 621.00 MB, 637.00 MB, 653.00 MB, 653.00 MB, 653.00 MB, 678.59 MB, 694.59 MB, 718.59 MB, 734.59 MB, 734.59 MB, 734.59 MB, 742.59 MB, 766.59 MB, 766.59 MB, 782.59 MB, 814.81 MB, 830.81 MB, 836.69 MB, 836.69 MB, 844.69 MB, 844.69 MB, 860.69 MB, 863.99 MB, 868.33 MB, 871.81 MB, 879.81 MB, 895.81 MB, 895.81 MB, 901.16 MB, 914.51 MB, 914.51 MB, 914.51 MB, 914.57 MB, 914.57 MB, 914.57 MB, 914.57 MB
           - FooterProcessingTime: (Avg: 999.992us ; Min: 999.992us ; Max: 999.992us ; Number of samples: 4)
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 3.68 
           - BytesRead: 914.57 MB (958994524)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 914.57 MB (958994524)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 914.57 MB (958994524)
           - CachedFileHandlesHitCount: 142 (142)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 1s404ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 10 (10)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 2 (2)
           - NumRowGroups: 4 (4)
           - NumScannerThreadsStarted: 4 (4)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 722.10 MB (757173382)
           - PerReadThreadRawHdfsThroughput: 2.13 GB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 33.999ms
           - RowBatchQueuePutWaitTime: 1m7s
           - RowsRead: 8.13M (8134450)
           - RowsReturned: 8.13M (8134450)
           - RowsReturnedRate: 71.36 M/sec
           - ScanRangesComplete: 4 (4)
           - ScannerThreadsInvoluntaryContextSwitches: 220 (220)
           - ScannerThreadsTotalWallClockTime: 1m10s
             - MaterializeTupleTime(*): 997.992ms
             - ScannerThreadsSysTime: 178.638ms
             - ScannerThreadsUserTime: 2s517ms
           - ScannerThreadsVoluntaryContextSwitches: 7.90K (7898)
           - TotalRawHdfsOpenFileTime(*): 2.999ms
           - TotalRawHdfsReadTime(*): 418.996ms
           - TotalReadThroughput: 48.14 MB/sec
        CodeGen:(Total: 30.999ms, non-child: 30.999ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 4.999ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 6.999ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 16.999ms
      Instance 7e441f58868a8585:7452021000000003 (host=ip-10-197-7-97.eu-west-1.compute.internal:22000):(Total: 17s461ms, non-child: 16.999ms, % non-child: 0.10%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/106.53 MB 0:1/253.72 MB 2:2/467.03 MB 
        Fragment Instance Lifecycle Event Timeline: 17s461ms
           - Prepare Finished: 51.999ms (51.999ms)
           - Open Finished: 73.999ms (21.999ms)
           - First Batch Produced: 109.999ms (35.999ms)
           - First Batch Sent: 111.999ms (1.999ms)
           - ExecInternal Finished: 17s461ms (17s349ms)
        MemoryUsage(500.000ms): 28.20 MB, 667.82 MB, 670.49 MB, 670.61 MB, 670.68 MB, 670.84 MB, 670.97 MB, 663.27 MB, 649.43 MB, 649.40 MB, 649.51 MB, 649.48 MB, 649.57 MB, 641.18 MB, 641.64 MB, 617.73 MB, 582.14 MB, 565.86 MB, 501.82 MB, 501.70 MB, 477.77 MB, 465.95 MB, 458.21 MB, 448.17 MB, 441.20 MB, 441.33 MB, 433.04 MB, 417.51 MB, 393.42 MB, 305.86 MB, 298.02 MB, 228.39 MB, 207.19 MB, 183.50 MB, 135.72 MB
        ThreadUsage(500.000ms): 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3
         - AverageThreadTokens: 4.40 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 671.38 MB (703996800)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 675.40 MB (708207971)
         - RowsProduced: 7.35M (7354152)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 6s756ms
         - TotalStorageWaitTime: 79.999ms
         - TotalThreadsInvoluntaryContextSwitches: 243 (243)
         - TotalThreadsTotalWallClockTime: 1m15s
           - TotalThreadsSysTime: 445.885ms
           - TotalThreadsUserTime: 12s510ms
         - TotalThreadsVoluntaryContextSwitches: 18.23K (18232)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 17s387ms
             - ExecTreeExecTime: 87.999ms
           - OpenTime: 21.999ms
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 51.999ms
             - ExecTreePrepareTime: 0.000ns
        KrpcDataStreamSender (dst_id=1):(Total: 17s338ms, non-child: 10s580ms, % non-child: 61.02%)
          BytesSent(500.000ms): 0, 44.05 MB, 87.14 MB, 122.34 MB, 161.11 MB, 195.23 MB, 226.66 MB, 269.07 MB, 302.79 MB, 330.26 MB, 352.78 MB, 389.26 MB, 427.13 MB, 452.09 MB, 481.46 MB, 516.94 MB, 561.69 MB, 602.17 MB, 648.44 MB, 693.18 MB, 730.98 MB, 771.50 MB, 799.23 MB, 824.71 MB, 849.85 MB, 880.43 MB, 905.90 MB, 936.61 MB, 975.60 MB, 1.00 GB, 1.04 GB, 1.07 GB, 1.10 GB, 1.14 GB, 1.18 GB
           - EosSent: 6 (6)
           - NetworkThroughput: 85.91 MB/sec
           - PeakMemoryUsage: 185.72 KB (190176)
           - RowsSent: 7.35M (7354152)
           - RpcFailure: 0 (0)
           - RpcRetry: 0 (0)
           - SerializeBatchTime: 5s306ms
           - TotalBytesSent: 1.22 GB (1309413246)
           - UncompressedRowBatchSize: 1.74 GB (1869500126)
        HDFS_SCAN_NODE (id=0):(Total: 83.999ms, non-child: 83.999ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/106.53 MB 0:1/253.72 MB 2:2/467.03 MB 
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4 out of 4
          Hdfs Read Thread Concurrency Bucket: 0:97.14% 1:0% 2:0% 3:2.857% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 
          File Formats: PARQUET/SNAPPY:4 
          BytesRead(500.000ms): 4.14 MB, 576.20 MB, 576.20 MB, 576.20 MB, 608.20 MB, 608.20 MB, 608.20 MB, 632.20 MB, 665.79 MB, 681.79 MB, 681.79 MB, 681.79 MB, 681.79 MB, 697.79 MB, 705.79 MB, 705.79 MB, 720.60 MB, 744.60 MB, 760.60 MB, 766.48 MB, 782.48 MB, 793.78 MB, 793.78 MB, 797.27 MB, 806.25 MB, 814.25 MB, 814.25 MB, 814.25 MB, 819.60 MB, 827.60 MB, 827.60 MB, 827.66 MB, 827.66 MB, 827.66 MB, 827.66 MB
           - FooterProcessingTime: (Avg: 999.999us ; Min: 999.999us ; Max: 999.999us ; Number of samples: 4)
           - AverageHdfsReadThreadConcurrency: 0.09 
           - AverageScannerThreadConcurrency: 3.40 
           - BytesRead: 827.66 MB (867865141)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 827.66 MB (867865141)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 827.66 MB (867865141)
           - CachedFileHandlesHitCount: 132 (132)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 1s181ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 10 (10)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 3 (3)
           - NumRowGroups: 4 (4)
           - NumScannerThreadsStarted: 4 (4)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 670.94 MB (703529916)
           - PerReadThreadRawHdfsThroughput: 2.07 GB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 35.999ms
           - RowBatchQueuePutWaitTime: 56s280ms
           - RowsRead: 7.35M (7354152)
           - RowsReturned: 7.35M (7354152)
           - RowsReturnedRate: 87.55 M/sec
           - ScanRangesComplete: 4 (4)
           - ScannerThreadsInvoluntaryContextSwitches: 138 (138)
           - ScannerThreadsTotalWallClockTime: 58s549ms
             - MaterializeTupleTime(*): 813.999ms
             - ScannerThreadsSysTime: 111.444ms
             - ScannerThreadsUserTime: 2s065ms
           - ScannerThreadsVoluntaryContextSwitches: 7.13K (7126)
           - TotalRawHdfsOpenFileTime(*): 0.000ns
           - TotalRawHdfsReadTime(*): 390.999ms
           - TotalReadThroughput: 47.29 MB/sec
        CodeGen:(Total: 21.999ms, non-child: 21.999ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 3.999ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 4.999ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 12.999ms
      Instance 7e441f58868a8585:7452021000000001 (host=ip-10-197-3-207.eu-west-1.compute.internal:22000):(Total: 17s149ms, non-child: 18.000ms, % non-child: 0.10%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:2/467.06 MB 2:2/451.17 MB 
        Fragment Instance Lifecycle Event Timeline: 17s149ms
           - Prepare Finished: 50.000ms (50.000ms)
           - Open Finished: 71.000ms (21.000ms)
           - First Batch Produced: 106.001ms (35.000ms)
           - First Batch Sent: 108.001ms (2.000ms)
           - ExecInternal Finished: 17s149ms (17s041ms)
        MemoryUsage(500.000ms): 709.14 MB, 714.39 MB, 715.27 MB, 715.40 MB, 715.64 MB, 715.54 MB, 715.79 MB, 715.99 MB, 716.04 MB, 715.96 MB, 716.14 MB, 716.15 MB, 700.17 MB, 676.50 MB, 668.59 MB, 642.89 MB, 634.99 MB, 635.14 MB, 618.75 MB, 602.92 MB, 599.26 MB, 581.11 MB, 564.91 MB, 550.86 MB, 543.13 MB, 519.24 MB, 503.23 MB, 447.54 MB, 375.65 MB, 359.90 MB, 274.91 MB, 207.42 MB, 167.55 MB, 36.20 MB
        ThreadUsage(500.000ms): 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 3, 3, 2
         - AverageThreadTokens: 4.76 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 716.26 MB (751056705)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 720.28 MB (755267876)
         - RowsProduced: 8.18M (8182685)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 5s229ms
         - TotalStorageWaitTime: 104.001ms
         - TotalThreadsInvoluntaryContextSwitches: 270 (270)
         - TotalThreadsTotalWallClockTime: 1m21s
           - TotalThreadsSysTime: 469.158ms
           - TotalThreadsUserTime: 13s793ms
         - TotalThreadsVoluntaryContextSwitches: 18.38K (18377)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 17s078ms
             - ExecTreeExecTime: 86.000ms
           - OpenTime: 21.000ms
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 50.000ms
             - ExecTreePrepareTime: 0.000ns
        KrpcDataStreamSender (dst_id=1):(Total: 17s031ms, non-child: 11s801ms, % non-child: 69.29%)
          BytesSent(500.000ms): 33.87 MB, 78.11 MB, 123.33 MB, 161.93 MB, 205.16 MB, 238.71 MB, 280.35 MB, 326.33 MB, 361.42 MB, 387.00 MB, 424.14 MB, 464.72 MB, 500.18 MB, 537.81 MB, 584.75 MB, 626.81 MB, 667.53 MB, 706.43 MB, 746.18 MB, 787.36 MB, 832.82 MB, 871.11 MB, 906.61 MB, 945.48 MB, 978.47 MB, 1014.22 MB, 1.03 GB, 1.07 GB, 1.13 GB, 1.17 GB, 1.21 GB, 1.25 GB, 1.29 GB, 1.34 GB
           - EosSent: 6 (6)
           - NetworkThroughput: 97.38 MB/sec
           - PeakMemoryUsage: 185.72 KB (190176)
           - RowsSent: 8.18M (8182685)
           - RpcFailure: 0 (0)
           - RpcRetry: 0 (0)
           - SerializeBatchTime: 5s788ms
           - TotalBytesSent: 1.36 GB (1455381679)
           - UncompressedRowBatchSize: 1.93 GB (2076465996)
        HDFS_SCAN_NODE (id=0):(Total: 79.000ms, non-child: 79.000ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:2/467.06 MB 2:2/451.17 MB 
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4 out of 4
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 
          File Formats: PARQUET/SNAPPY:4 
          BytesRead(500.000ms): 623.91 MB, 623.91 MB, 639.91 MB, 655.91 MB, 655.91 MB, 663.91 MB, 703.91 MB, 743.91 MB, 749.38 MB, 749.38 MB, 749.38 MB, 765.38 MB, 781.38 MB, 789.38 MB, 809.73 MB, 851.12 MB, 851.12 MB, 851.12 MB, 867.12 MB, 881.39 MB, 884.72 MB, 888.20 MB, 904.20 MB, 905.20 MB, 905.20 MB, 905.20 MB, 905.20 MB, 918.54 MB, 918.54 MB, 918.54 MB, 918.61 MB, 918.61 MB, 918.61 MB, 918.61 MB
           - FooterProcessingTime: (Avg: 750.007us ; Min: 0.000ns ; Max: 1.000ms ; Number of samples: 4)
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 3.76 
           - BytesRead: 918.61 MB (963230992)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 918.61 MB (963230992)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 918.61 MB (963230992)
           - CachedFileHandlesHitCount: 143 (143)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 1s301ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 10 (10)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 2 (2)
           - NumRowGroups: 4 (4)
           - NumScannerThreadsStarted: 4 (4)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 715.85 MB (750622592)
           - PerReadThreadRawHdfsThroughput: 2.34 GB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 35.000ms
           - RowBatchQueuePutWaitTime: 1m2s
           - RowsRead: 8.18M (8182685)
           - RowsReturned: 8.18M (8182685)
           - RowsReturnedRate: 103.58 M/sec
           - ScanRangesComplete: 4 (4)
           - ScannerThreadsInvoluntaryContextSwitches: 98 (98)
           - ScannerThreadsTotalWallClockTime: 1m4s
             - MaterializeTupleTime(*): 715.007ms
             - ScannerThreadsSysTime: 98.914ms
             - ScannerThreadsUserTime: 2s225ms
           - ScannerThreadsVoluntaryContextSwitches: 7.93K (7928)
           - TotalRawHdfsOpenFileTime(*): 0.000ns
           - TotalRawHdfsReadTime(*): 383.003ms
           - TotalReadThroughput: 54.04 MB/sec
        CodeGen:(Total: 21.000ms, non-child: 21.000ms, % non-child: 100.00%)
           - CodegenTime: 1.000ms
           - CompileTime: 4.000ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 5.000ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 12.000ms
      Instance 7e441f58868a8585:7452021000000005 (host=ip-10-197-10-88.eu-west-1.compute.internal:22000):(Total: 17s105ms, non-child: 19.000ms, % non-child: 0.11%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:3/627.80 MB 1:1/253.66 MB 
        Fragment Instance Lifecycle Event Timeline: 17s105ms
           - Prepare Finished: 51.000ms (51.000ms)
           - Open Finished: 72.000ms (21.000ms)
           - First Batch Produced: 107.000ms (35.000ms)
           - First Batch Sent: 109.000ms (2.000ms)
           - ExecInternal Finished: 17s105ms (16s996ms)
        MemoryUsage(500.000ms): 692.86 MB, 699.27 MB, 700.14 MB, 700.51 MB, 700.56 MB, 700.60 MB, 684.63 MB, 685.08 MB, 685.29 MB, 685.32 MB, 685.18 MB, 677.29 MB, 661.38 MB, 637.53 MB, 613.87 MB, 605.89 MB, 589.97 MB, 524.56 MB, 511.58 MB, 495.58 MB, 484.73 MB, 476.73 MB, 476.90 MB, 476.89 MB, 452.74 MB, 436.96 MB, 404.94 MB, 365.30 MB, 341.49 MB, 309.50 MB, 285.79 MB, 237.92 MB, 142.99 MB, 31.44 MB
        ThreadUsage(500.000ms): 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1
         - AverageThreadTokens: 4.38 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 700.84 MB (734883717)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 704.86 MB (739094888)
         - RowsProduced: 7.89M (7887894)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 5s835ms
         - TotalStorageWaitTime: 183.001ms
         - TotalThreadsInvoluntaryContextSwitches: 231 (231)
         - TotalThreadsTotalWallClockTime: 1m15s
           - TotalThreadsSysTime: 445.228ms
           - TotalThreadsUserTime: 13s057ms
         - TotalThreadsVoluntaryContextSwitches: 18.54K (18544)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 17s033ms
             - ExecTreeExecTime: 86.000ms
           - OpenTime: 21.000ms
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 51.000ms
             - ExecTreePrepareTime: 0.000ns
        KrpcDataStreamSender (dst_id=1):(Total: 16s982ms, non-child: 11s143ms, % non-child: 65.62%)
          BytesSent(500.000ms): 26.93 MB, 66.51 MB, 110.22 MB, 150.25 MB, 194.98 MB, 230.43 MB, 276.49 MB, 317.98 MB, 348.51 MB, 373.38 MB, 406.12 MB, 450.37 MB, 487.00 MB, 525.66 MB, 571.16 MB, 611.32 MB, 648.53 MB, 684.93 MB, 720.33 MB, 759.24 MB, 805.05 MB, 839.81 MB, 871.11 MB, 904.63 MB, 939.44 MB, 970.81 MB, 1006.05 MB, 1.02 GB, 1.08 GB, 1.12 GB, 1.16 GB, 1.20 GB, 1.25 GB, 1.29 GB
           - EosSent: 6 (6)
           - NetworkThroughput: 94.11 MB/sec
           - PeakMemoryUsage: 185.72 KB (190176)
           - RowsSent: 7.89M (7887894)
           - RpcFailure: 0 (0)
           - RpcRetry: 0 (0)
           - SerializeBatchTime: 5s436ms
           - TotalBytesSent: 1.30 GB (1399551209)
           - UncompressedRowBatchSize: 1.86 GB (1996363422)
        HDFS_SCAN_NODE (id=0):(Total: 83.000ms, non-child: 83.000ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:3/627.80 MB 1:1/253.66 MB 
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4 out of 4
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 
          File Formats: PARQUET/SNAPPY:4 
          BytesRead(500.000ms): 602.80 MB, 602.80 MB, 602.80 MB, 618.80 MB, 634.80 MB, 634.80 MB, 655.41 MB, 695.41 MB, 711.41 MB, 711.41 MB, 711.41 MB, 719.41 MB, 735.41 MB, 743.41 MB, 751.41 MB, 775.41 MB, 791.41 MB, 803.16 MB, 803.16 MB, 827.16 MB, 831.51 MB, 836.43 MB, 844.43 MB, 860.43 MB, 860.43 MB, 865.73 MB, 881.73 MB, 881.73 MB, 881.73 MB, 881.84 MB, 881.84 MB, 881.84 MB, 881.84 MB, 881.84 MB
           - FooterProcessingTime: (Avg: 750.006us ; Min: 0.000ns ; Max: 1.000ms ; Number of samples: 4)
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 3.38 
           - BytesRead: 881.84 MB (924678748)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 881.84 MB (924678748)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 881.84 MB (924678748)
           - CachedFileHandlesHitCount: 138 (138)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 1s231ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 10 (10)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 2 (2)
           - NumRowGroups: 4 (4)
           - NumScannerThreadsStarted: 4 (4)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 700.39 MB (734416868)
           - PerReadThreadRawHdfsThroughput: 2.17 GB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 35.000ms
           - RowBatchQueuePutWaitTime: 56s298ms
           - RowsRead: 7.89M (7887894)
           - RowsReturned: 7.89M (7887894)
           - RowsReturnedRate: 95.03 M/sec
           - ScanRangesComplete: 4 (4)
           - ScannerThreadsInvoluntaryContextSwitches: 123 (123)
           - ScannerThreadsTotalWallClockTime: 58s719ms
             - MaterializeTupleTime(*): 816.006ms
             - ScannerThreadsSysTime: 95.015ms
             - ScannerThreadsUserTime: 2s161ms
           - ScannerThreadsVoluntaryContextSwitches: 7.65K (7647)
           - TotalRawHdfsOpenFileTime(*): 0.000ns
           - TotalRawHdfsReadTime(*): 397.002ms
           - TotalReadThroughput: 51.87 MB/sec
        CodeGen:(Total: 21.000ms, non-child: 21.000ms, % non-child: 100.00%)
           - CodegenTime: 1.000ms
           - CompileTime: 4.000ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 5.000ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 12.000ms
      Instance 7e441f58868a8585:7452021000000002 (host=ip-10-197-11-142.eu-west-1.compute.internal:22000):(Total: 16s970ms, non-child: 27.000ms, % non-child: 0.16%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/167.11 MB 2:4/678.05 MB 
        Fragment Instance Lifecycle Event Timeline: 16s971ms
           - Prepare Finished: 51.000ms (51.000ms)
           - Open Finished: 72.000ms (21.000ms)
           - First Batch Produced: 106.000ms (34.000ms)
           - First Batch Sent: 108.000ms (2.000ms)
           - ExecInternal Finished: 16s971ms (16s863ms)
        MemoryUsage(500.000ms): 765.77 MB, 772.47 MB, 772.79 MB, 773.80 MB, 773.89 MB, 774.25 MB, 773.85 MB, 762.22 MB, 754.42 MB, 746.55 MB, 740.69 MB, 732.79 MB, 732.80 MB, 732.67 MB, 724.67 MB, 701.24 MB, 661.22 MB, 629.30 MB, 625.35 MB, 603.35 MB, 571.83 MB, 444.85 MB, 427.24 MB, 403.24 MB, 394.97 MB, 387.30 MB, 367.22 MB, 359.05 MB, 261.78 MB, 245.86 MB, 198.36 MB, 113.10 MB, 81.26 MB, 7.70 MB
        ThreadUsage(500.000ms): 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 2, 2, 1
         - AverageThreadTokens: 4.94 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 774.46 MB (812076675)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 3.40 GB (3650119082)
         - RowsProduced: 7.57M (7568704)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 4s401ms
         - TotalStorageWaitTime: 254.001ms
         - TotalThreadsInvoluntaryContextSwitches: 1.65K (1652)
         - TotalThreadsTotalWallClockTime: 1m24s
           - TotalThreadsSysTime: 758.495ms
           - TotalThreadsUserTime: 14s414ms
         - TotalThreadsVoluntaryContextSwitches: 16.32K (16321)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 16s899ms
             - ExecTreeExecTime: 104.000ms
           - OpenTime: 21.000ms
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 50.000ms
             - ExecTreePrepareTime: 0.000ns
        KrpcDataStreamSender (dst_id=1):(Total: 16s820ms, non-child: 12s417ms, % non-child: 73.82%)
          BytesSent(500.000ms): 24.65 MB, 66.72 MB, 108.74 MB, 148.75 MB, 187.00 MB, 227.95 MB, 271.53 MB, 313.40 MB, 343.59 MB, 369.08 MB, 401.56 MB, 443.03 MB, 485.01 MB, 529.95 MB, 576.96 MB, 617.74 MB, 653.17 MB, 688.82 MB, 724.27 MB, 761.84 MB, 803.51 MB, 838.04 MB, 869.12 MB, 901.81 MB, 936.65 MB, 969.05 MB, 1003.70 MB, 1.01 GB, 1.06 GB, 1.10 GB, 1.14 GB, 1.17 GB, 1.21 GB, 1.25 GB
           - EosSent: 6 (6)
           - NetworkThroughput: 110.72 MB/sec
           - PeakMemoryUsage: 185.72 KB (190176)
           - RowsSent: 7.57M (7568704)
           - RpcFailure: 0 (0)
           - RpcRetry: 0 (0)
           - SerializeBatchTime: 5s908ms
           - TotalBytesSent: 1.25 GB (1340563625)
           - UncompressedRowBatchSize: 1.78 GB (1910432069)
        HDFS_SCAN_NODE (id=0):(Total: 102.000ms, non-child: 102.000ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/167.11 MB 2:4/678.05 MB 
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out of 5
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 
          File Formats: PARQUET/SNAPPY:5 
          BytesRead(500.000ms): 647.74 MB, 647.74 MB, 647.74 MB, 663.74 MB, 663.74 MB, 687.74 MB, 687.74 MB, 695.27 MB, 707.59 MB, 707.59 MB, 741.49 MB, 754.97 MB, 754.97 MB, 762.97 MB, 762.97 MB, 778.97 MB, 778.97 MB, 778.97 MB, 781.91 MB, 790.95 MB, 806.95 MB, 812.83 MB, 812.83 MB, 812.83 MB, 827.11 MB, 827.11 MB, 829.57 MB, 837.57 MB, 837.57 MB, 845.57 MB, 845.57 MB, 845.63 MB, 845.63 MB, 845.63 MB
           - FooterProcessingTime: (Avg: 5.800ms ; Min: 0.000ns ; Max: 26.000ms ; Number of samples: 5)
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 3.94 
           - BytesRead: 845.63 MB (886710526)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 845.63 MB (886710526)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 845.63 MB (886710526)
           - CachedFileHandlesHitCount: 138 (138)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 1s374ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 10 (10)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 2 (2)
           - NumRowGroups: 5 (5)
           - NumScannerThreadsStarted: 5 (5)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 774.11 MB (811708098)
           - PerReadThreadRawHdfsThroughput: 2.24 GB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 33.000ms
           - RowBatchQueuePutWaitTime: 1m4s
           - RowsRead: 7.57M (7568704)
           - RowsReturned: 7.57M (7568704)
           - RowsReturnedRate: 74.20 M/sec
           - ScanRangesComplete: 5 (5)
           - ScannerThreadsInvoluntaryContextSwitches: 135 (135)
           - ScannerThreadsTotalWallClockTime: 1m7s
             - MaterializeTupleTime(*): 882.004ms
             - ScannerThreadsSysTime: 175.091ms
             - ScannerThreadsUserTime: 2s317ms
           - ScannerThreadsVoluntaryContextSwitches: 7.35K (7351)
           - TotalRawHdfsOpenFileTime(*): 0.000ns
           - TotalRawHdfsReadTime(*): 369.002ms
           - TotalReadThroughput: 49.74 MB/sec
        CodeGen:(Total: 21.000ms, non-child: 21.000ms, % non-child: 100.00%)
           - CodegenTime: 1.000ms
           - CompileTime: 3.000ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 6.000ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 12.000ms
      Instance 7e441f58868a8585:7452021000000004 (host=ip-10-197-29-94.eu-west-1.compute.internal:22000):(Total: 13s276ms, non-child: 15.000ms, % non-child: 0.11%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/197.15 MB 2:1/253.70 MB 0:1/212.81 MB 
        Fragment Instance Lifecycle Event Timeline: 13s277ms
           - Prepare Finished: 51.000ms (51.000ms)
           - Open Finished: 74.000ms (23.000ms)
           - First Batch Produced: 111.000ms (37.000ms)
           - First Batch Sent: 114.000ms (3.000ms)
           - ExecInternal Finished: 13s277ms (13s163ms)
        MemoryUsage(500.000ms): 0, 533.62 MB, 535.88 MB, 536.01 MB, 535.96 MB, 536.17 MB, 536.19 MB, 536.50 MB, 536.74 MB, 536.63 MB, 528.69 MB, 505.96 MB, 478.24 MB, 470.17 MB, 470.15 MB, 454.34 MB, 446.31 MB, 434.25 MB, 412.47 MB, 397.38 MB, 389.31 MB, 373.32 MB, 349.35 MB, 277.69 MB, 190.58 MB, 119.54 MB, 79.70 MB
        ThreadUsage(500.000ms): 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 2, 2
         - AverageThreadTokens: 3.70 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 536.93 MB (563013699)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 2.73 GB (2926852205)
         - RowsProduced: 5.87M (5872115)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 3s474ms
         - TotalStorageWaitTime: 33.000ms
         - TotalThreadsInvoluntaryContextSwitches: 2.79K (2792)
         - TotalThreadsTotalWallClockTime: 50s268ms
           - TotalThreadsSysTime: 571.274ms
           - TotalThreadsUserTime: 11s141ms
         - TotalThreadsVoluntaryContextSwitches: 13.29K (13286)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 13s203ms
             - ExecTreeExecTime: 121.000ms
           - OpenTime: 23.000ms
             - ExecTreeOpenTime: 0.000ns
           - PrepareTime: 50.000ms
             - ExecTreePrepareTime: 0.000ns
        KrpcDataStreamSender (dst_id=1):(Total: 13s121ms, non-child: 9s642ms, % non-child: 73.49%)
          BytesSent(500.000ms): 32.44 MB, 72.73 MB, 116.97 MB, 157.51 MB, 196.82 MB, 228.57 MB, 267.48 MB, 310.26 MB, 338.82 MB, 370.78 MB, 413.11 MB, 446.41 MB, 480.28 MB, 516.76 MB, 558.14 MB, 591.15 MB, 625.03 MB, 659.01 MB, 692.28 MB, 731.28 MB, 769.56 MB, 812.39 MB, 852.55 MB, 896.49 MB, 938.39 MB, 981.76 MB
           - EosSent: 6 (6)
           - NetworkThroughput: 103.73 MB/sec
           - PeakMemoryUsage: 185.72 KB (190176)
           - RowsSent: 5.87M (5872115)
           - RpcFailure: 0 (0)
           - RpcRetry: 0 (0)
           - SerializeBatchTime: 4s538ms
           - TotalBytesSent: 1001.82 MB (1050482229)
           - UncompressedRowBatchSize: 1.40 GB (1499344163)
        HDFS_SCAN_NODE (id=0):(Total: 117.000ms, non-child: 117.000ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/197.15 MB 2:1/253.70 MB 0:1/212.81 MB 
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 3 out of 3
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 
          File Formats: PARQUET/SNAPPY:3 
          BytesRead(500.000ms): 0, 458.13 MB, 458.13 MB, 482.13 MB, 482.13 MB, 498.13 MB, 543.55 MB, 551.55 MB, 551.55 MB, 567.55 MB, 575.55 MB, 588.81 MB, 607.23 MB, 623.23 MB, 623.23 MB, 629.39 MB, 645.39 MB, 648.70 MB, 649.72 MB, 650.60 MB, 658.60 MB, 658.60 MB, 658.60 MB, 663.94 MB, 663.94 MB, 663.94 MB, 663.94 MB
           - FooterProcessingTime: (Avg: 666.669us ; Min: 0.000ns ; Max: 1.000ms ; Number of samples: 3)
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 2.81 
           - BytesRead: 663.94 MB (696193384)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 663.94 MB (696193384)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 663.94 MB (696193384)
           - CachedFileHandlesHitCount: 104 (104)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 995.003ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 10 (10)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 3 (3)
           - NumRowGroups: 3 (3)
           - NumScannerThreadsStarted: 3 (3)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 536.78 MB (562855983)
           - PerReadThreadRawHdfsThroughput: 1.98 GB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 37.000ms
           - RowBatchQueuePutWaitTime: 35s136ms
           - RowsRead: 5.87M (5872115)
           - RowsReturned: 5.87M (5872115)
           - RowsReturnedRate: 50.19 M/sec
           - ScanRangesComplete: 3 (3)
           - ScannerThreadsInvoluntaryContextSwitches: 223 (223)
           - ScannerThreadsTotalWallClockTime: 37s042ms
             - MaterializeTupleTime(*): 687.002ms
             - ScannerThreadsSysTime: 118.760ms
             - ScannerThreadsUserTime: 1s825ms
           - ScannerThreadsVoluntaryContextSwitches: 5.67K (5674)
           - TotalRawHdfsOpenFileTime(*): 0.000ns
           - TotalRawHdfsReadTime(*): 327.001ms
           - TotalReadThroughput: 49.18 MB/sec
        CodeGen:(Total: 23.000ms, non-child: 23.000ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 4.000ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 6.000ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 12.000ms

Mime
View raw message