impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeszy <jes...@gmail.com>
Subject Re: Issue in data loading in Impala + Kudu
Date Thu, 10 May 2018 12:34:27 GMT
As suggested over on the Kudu list, this is likely due to key
duplication (which is fine on HDFS, but won't work for Kudu). The
profile has the following error that confirms this:
    Errors: Key already present in Kudu table
'impala::kudu_impala_500.LINEITEM'. (1 of -1831809966 similar)

Raised IMPALA-7007 to address the overflow(?) of the counter.

On 10 May 2018 at 08:00, Mostafa Mokhtar <mmokhtar@cloudera.com> wrote:
> Can you try rerunning the query again against the Kudu database instead of
> default?
>
> select count(*) from kudu_impala_500.LINEITEM;
>
>
> On Wed, May 9, 2018 at 10:13 PM, Geetika Gupta <geetika.gupta@knoldus.in>
> wrote:
>>
>> Please find below the query profile :
>>
>> Query (id=9649f7ab3bcc5fb8:f4d6a60700000000):
>>   Summary:
>>     Session ID: b04a9080d1e1724d:41f2b0d261e8f280
>>     Session Type: HIVESERVER2
>>     HiveServer2 Protocol Version: V6
>>     Start Time: 2018-05-08 17:55:23.181264000
>>     End Time: 2018-05-10 00:34:17.784273000
>>     Query Type: DML
>>     Query State: FINISHED
>>     Query Status: OK
>>     Impala Version: impalad version 3.0.0-SNAPSHOT RELEASE (build
>> b68e06997c1f49f6b723d78e217efddec4f56f3a)
>>     User: root
>>     Connected User: root
>>     Delegated User:
>>     Network Address: ::ffff:46.4.88.233:59862
>>     Default Db: kudu_impala_500
>>     Sql Statement: insert into LINEITEM select * from
>> PARQUETIMPALA500.LINEITEM
>>     Coordinator: slave2:22000
>>     Query Options (set by configuration):
>>     Query Options (set by configuration and planner): MT_DOP=0
>>     Plan:
>> ----------------
>> Max Per-Host Resource Reservation: Memory=0B
>> Per-Host Resource Estimates: Memory=704.00MB
>> WARNING: The following tables are missing relevant table and/or column
>> statistics.
>> parquetimpala500.lineitem
>>
>> F00:PLAN FRAGMENT [RANDOM] hosts=7 instances=7
>> |  Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B
>> INSERT INTO KUDU [kudu_impala_500.lineitem]
>> |  mem-estimate=0B mem-reservation=0B
>> |
>> 00:SCAN HDFS [parquetimpala500.lineitem, RANDOM]
>>    partitions=1/1 files=396 size=97.29GB
>>    stored statistics:
>>      table: rows=unavailable size=unavailable
>>      columns: unavailable
>>    extrapolated-rows=disabled
>>    mem-estimate=704.00MB mem-reservation=0B
>>    tuple-ids=0 row-size=171B cardinality=unavailable
>> ----------------
>>     Estimated Per-Host Mem: 738197504
>>     Tables Missing Stats: parquetimpala500.lineitem
>>     Per Host Min Reservation: slave1:22000(0) slave2:22000(0)
>> slave3:22000(0) slave4:22000(0) slave5:22000(0) slave6:22000(0)
>> slave7:22000(0)
>>     Request Pool: default-pool
>>     Admission result: Admitted immediately
>>     ExecSummary:
>> Operator       #Hosts  Avg Time  Max Time  #Rows  Est. #Rows  Peak Mem
>> Est. Peak Mem  Detail
>>
>> -----------------------------------------------------------------------------------------------------------------
>> 00:SCAN HDFS        7   4s417ms   6s154ms  3.00B          -1   1.39 GB
>> 704.00 MB  parquetimpala500.lineitem
>>     Errors: Key already present in Kudu table
>> 'impala::kudu_impala_500.LINEITEM'. (1 of -1831809966 similar)
>>
>>     Query Compilation: 6s413ms
>>        - Metadata load started: 14.443ms (14.443ms)
>>        - Metadata load finished. loaded-tables=2/2 load-requests=1
>> catalog-updates=7: 6s298ms (6s283ms)
>>        - Analysis finished: 6s301ms (3.659ms)
>>        - Value transfer graph computed: 6s302ms (282.554us)
>>        - Single node plan created: 6s363ms (61.624ms)
>>        - Runtime filters computed: 6s363ms (98.878us)
>>        - Distributed plan created: 6s366ms (2.832ms)
>>        - Planning finished: 6s413ms (46.751ms)
>>     Query Timeline: 30h38m
>>        - Query submitted: 56.829us (56.829us)
>>        - Planning finished: 6s431ms (6s431ms)
>>        - Submit for admission: 6s432ms (821.867us)
>>        - Completed admission: 6s432ms (14.519us)
>>        - Ready to start on 7 backends: 6s432ms (99.515us)
>>        - All 7 execution backends (7 fragment instances) started: 6s535ms
>> (103.320ms)
>>        - Released admission control resources: 30h38m (30h38m)
>>        - DML data written: 30h38m (934.739us)
>>        - DML Metastore update finished: 30h38m (157.938us)
>>        - Request finished: 30h38m (41.379us)
>>        - First row fetched: 30h38m (173.124us)
>>        - First row fetched: 30h38m (1.750ms)
>>        - First row fetched: 30h38m (1.361ms)
>>        - Unregister query: 30h38m (1.456ms)
>>      - ComputeScanRangeAssignmentTimer: 558.440us
>>   ImpalaServer:
>>      - ClientFetchWaitTimer: 4.721ms
>>      - MetastoreUpdateTimer: 191.424us
>>      - RowMaterializationTimer: 0.000ns
>>   Execution Profile 9649f7ab3bcc5fb8:f4d6a60700000000:(Total: 30h38m,
>> non-child: 0.000ns, % non-child: 0.00%)
>>     Number of filters: 0
>>     Filter routing table:
>>  ID  Src. Node  Tgt. Node(s)  Target type  Partition filter  Pending
>> (Expected)  First arrived  Completed   Enabled
>>
>> -------------------------------------------------------------------------------------------------------------------
>>
>>     Backend startup latencies: Count: 7, min / max: 1ms / 103ms, 25th
>> %-ile: 1ms, 50th %-ile: 67ms, 75th %-ile: 67ms, 90th %-ile: 80ms, 95th
>> %-ile: 103ms, 99.9th %-ile: 103ms
>>     DML Stats:
>> Partition: Default
>> NumModifiedRows: 536870912
>> NumRowErrors: 2463157330
>>   BytesWritten: 0
>>     Per Node Peak Memory Usage: slave1:22000(1.40 GB) slave2:22000(1.41
>> GB) slave3:22000(1.39 GB) slave4:22000(1.41 GB) slave5:22000(1.41 GB)
>> slave6:22000(1.40 GB) slave7:22000(1.40 GB)
>>      - FiltersReceived: 0 (0)
>>      - FinalizationTimer: 0.000ns
>>      - NumBackends: 7 (7)
>>      - NumFragmentInstances: 7 (7)
>>      - NumFragments: 1 (1)
>>     Averaged Fragment F00:(Total: 29h51m, non-child: 14s973ms, %
>> non-child: 0.01%)
>>       split sizes:  min: 13.63 GB, max: 14.10 GB, avg: 13.90 GB, stddev:
>> 143.22 MB
>>       completion times: min:28h47m  max:30h38m  mean: 29h51m
>> stddev:41m26s
>>       execution rates: min:129.50 KB/sec  max:140.31 KB/sec  mean:135.67
>> KB/sec  stddev:3.50 KB/sec
>>       num instances: 7
>>        - AverageThreadTokens: 8.89
>>        - BloomFilterBytes: 0
>>        - PeakMemoryUsage: 1.40 GB (1503884428)
>>        - PeakReservation: 0
>>        - PeakUsedReservation: 0
>>        - PerHostPeakMemUsage: 1.40 GB (1503884428)
>>        - RowsProduced: 428.58M (428575463)
>>        - TotalNetworkReceiveTime: 0.000ns
>>        - TotalNetworkSendTime: 0.000ns
>>        - TotalStorageWaitTime: 2m10s
>>        - TotalThreadsInvoluntaryContextSwitches: 369.81K (369812)
>>        - TotalThreadsTotalWallClockTime: 265h20m
>>          - TotalThreadsSysTime: 1m59s
>>          - TotalThreadsUserTime: 1h9m
>>        - TotalThreadsVoluntaryContextSwitches: 433.92K (433917)
>>       Buffer pool:
>>          - AllocTime: 0.000ns
>>          - CumulativeAllocationBytes: 0
>>          - CumulativeAllocations: 0 (0)
>>          - PeakReservation: 0
>>          - PeakUnpinnedBytes: 0
>>          - PeakUsedReservation: 0
>>          - ReadIoBytes: 0
>>          - ReadIoOps: 0 (0)
>>          - ReadIoWaitTime: 0.000ns
>>          - ReservationLimit: 0
>>          - WriteIoBytes: 0
>>          - WriteIoOps: 0 (0)
>>          - WriteIoWaitTime: 0.000ns
>>       Fragment Instance Lifecycle Timings:
>>          - ExecTime: 29h51m
>>            - ExecTreeExecTime: 4s724ms
>>          - OpenTime: 519.095ms
>>            - ExecTreeOpenTime: 36.350us
>>          - PrepareTime: 60.445ms
>>            - ExecTreePrepareTime: 19.494ms
>>       KuduTableSink:(Total: 29h50m, non-child: 29h50m, % non-child:
>> 100.00%)
>>          - KuduApplyTimer: 28h54m
>>          - NumRowErrors: 351.88M (351879618)
>>          - PeakMemoryUsage: 20.00 MB (20971520)
>>          - RowsProcessedRate: 3.99 K/sec
>>          - TotalNumRows: 428.58M (428575463)
>>       HDFS_SCAN_NODE (id=0):(Total: 4s417ms, non-child: 4s417ms, %
>> non-child: 100.00%)
>>          - AverageHdfsReadThreadConcurrency: 0.00
>>          - AverageScannerThreadConcurrency: 7.89
>>          - BytesRead: 13.90 GB (14928723218)
>>          - BytesReadDataNodeCache: 0
>>          - BytesReadLocal: 13.90 GB (14928723218)
>>          - BytesReadRemoteUnexpected: 0
>>          - BytesReadShortCircuit: 0
>>          - CachedFileHandlesHitCount: 2.31K (2305)
>>          - CachedFileHandlesMissCount: 56 (56)
>>          - CollectionItemsRead: 0 (0)
>>          - DecompressionTime: 22s321ms
>>          - MaxCompressedTextFileLength: 0
>>          - NumColumns: 16 (16)
>>          - NumDictFilteredRowGroups: 0 (0)
>>          - NumDisksAccessed: 1 (1)
>>          - NumRowGroups: 56 (56)
>>          - NumScannerThreadsStarted: 8 (8)
>>          - NumScannersWithNoReads: 0 (0)
>>          - NumStatsFilteredRowGroups: 0 (0)
>>          - PeakMemoryUsage: 1.38 GB (1482717084)
>>          - PerReadThreadRawHdfsThroughput: 69.65 MB/sec
>>          - RemoteScanRanges: 0 (0)
>>          - RowBatchQueueGetWaitTime: 1s754ms
>>          - RowBatchQueuePutWaitTime: 235h24m
>>          - RowsRead: 428.58M (428575463)
>>          - RowsReturned: 428.58M (428575463)
>>          - RowsReturnedRate: 99.08 M/sec
>>          - ScanRangesComplete: 56 (56)
>>          - ScannerThreadsInvoluntaryContextSwitches: 12.33K (12332)
>>          - ScannerThreadsTotalWallClockTime: 235h28m
>>            - MaterializeTupleTime(*): 2m3s
>>            - ScannerThreadsSysTime: 3s896ms
>>            - ScannerThreadsUserTime: 2m39s
>>          - ScannerThreadsVoluntaryContextSwitches: 419.69K (419690)
>>          - TotalRawHdfsOpenFileTime(*): 3s995ms
>>          - TotalRawHdfsReadTime(*): 4m2s
>>          - TotalReadThroughput: 135.78 KB/sec
>>       CodeGen:(Total: 436.431ms, non-child: 436.431ms, % non-child:
>> 100.00%)
>>          - CodegenInvoluntaryContextSwitches: 5 (5)
>>          - CodegenTotalWallClockTime: 436.422ms
>>            - CodegenSysTime: 8.585ms
>>            - CodegenUserTime: 41.806ms
>>          - CodegenVoluntaryContextSwitches: 76 (76)
>>          - CompileTime: 179.278ms
>>          - IrGenerationTime: 90.176ms
>>          - LoadTime: 0.000ns
>>          - ModuleBitcodeSize: 2.26 MB (2373832)
>>          - NumFunctions: 19 (19)
>>          - NumInstructions: 706 (706)
>>          - OptimizationTime: 151.857ms
>>          - PeakMemoryUsage: 353.00 KB (361472)
>>          - PrepareTime: 14.499ms
>>     Fragment F00:
>>       Instance 9649f7ab3bcc5fb8:f4d6a60700000000
>> (host=slave3:22000):(Total: 30h38m, non-child: 11s364ms, % non-child: 0.01%)
>>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.63 GB
>>         Fragment Instance Lifecycle Event Timeline: 30h38m
>>            - Prepare Finished: 86.126ms (86.126ms)
>>            - Open Finished: 1s157ms (1s070ms)
>>            - First Batch Produced: 3s031ms (1s874ms)
>>            - First Batch Sent: 3s037ms (5.724ms)
>>            - ExecInternal Finished: 30h38m (30h38m)
>>         MemoryUsage(34m8s): 1.28 GB, 1.34 GB, 1.37 GB, 1.35 GB, 1.21 GB,
>> 925.43 MB, 727.39 MB, 1.17 GB, 1.35 GB, 1.35 GB, 1.35 GB, 1.35 GB, 1.26 GB,
>> 1.06 GB, 836.27 MB, 792.34 MB, 1.26 GB, 1.36 GB, 1.36 GB, 1.30 GB, 1.10 GB,
>> 827.96 MB, 1.01 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.12 GB, 875.56 MB, 979.49
>> MB, 1.34 GB, 1.37 GB, 1.37 GB, 1.28 GB, 1.10 GB, 950.36 MB, 805.45 MB,
>> 963.12 MB, 1.28 GB, 1.34 GB, 1.34 GB, 1.30 GB, 1.04 GB, 808.54 MB, 994.36
>> MB, 1.17 GB, 1.18 GB, 1.18 GB, 1.18 GB, 1.17 GB, 1.13 GB, 1.04 GB, 954.52
>> MB, 811.51 MB
>>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 8, 8, 8, 8, 8, 8, 8, 8, 8
>>          - AverageThreadTokens: 8.80
>>          - BloomFilterBytes: 0
>>          - PeakMemoryUsage: 1.39 GB (1487275684)
>>          - PeakReservation: 0
>>          - PeakUsedReservation: 0
>>          - PerHostPeakMemUsage: 1.39 GB (1487275684)
>>          - RowsProduced: 424.29M (424287956)
>>          - TotalNetworkReceiveTime: 0.000ns
>>          - TotalNetworkSendTime: 0.000ns
>>          - TotalStorageWaitTime: 1m32s
>>          - TotalThreadsInvoluntaryContextSwitches: 349.35K (349347)
>>          - TotalThreadsTotalWallClockTime: 269h37m
>>            - TotalThreadsSysTime: 1m54s
>>            - TotalThreadsUserTime: 1h6m
>>          - TotalThreadsVoluntaryContextSwitches: 438.17K (438165)
>>         Buffer pool:
>>            - AllocTime: 0.000ns
>>            - CumulativeAllocationBytes: 0
>>            - CumulativeAllocations: 0 (0)
>>            - PeakReservation: 0
>>            - PeakUnpinnedBytes: 0
>>            - PeakUsedReservation: 0
>>            - ReadIoBytes: 0
>>            - ReadIoOps: 0 (0)
>>            - ReadIoWaitTime: 0.000ns
>>            - ReservationLimit: 0
>>            - WriteIoBytes: 0
>>            - WriteIoOps: 0 (0)
>>            - WriteIoWaitTime: 0.000ns
>>         Fragment Instance Lifecycle Timings:
>>            - ExecTime: 30h38m
>>              - ExecTreeExecTime: 4s586ms
>>            - OpenTime: 1s070ms
>>              - ExecTreeOpenTime: 32.390us
>>            - PrepareTime: 85.903ms
>>              - ExecTreePrepareTime: 16.283ms
>>         KuduTableSink:(Total: 30h38m, non-child: 30h38m, % non-child:
>> 100.00%)
>>            - KuduApplyTimer: 29h44m
>>            - NumRowErrors: 371.48M (371483541)
>>            - PeakMemoryUsage: 20.00 MB (20971520)
>>            - RowsProcessedRate: 3.85 K/sec
>>            - TotalNumRows: 424.29M (424287956)
>>         HDFS_SCAN_NODE (id=0):(Total: 4s304ms, non-child: 4s304ms, %
>> non-child: 100.00%)
>>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.63 GB
>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of
>> 56
>>           Hdfs Read Thread Concurrency Bucket: 0:99.87% 1:0.1279% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>>           File Formats: PARQUET/SNAPPY:56
>>           BytesRead(34m8s): 1.19 GB, 1.46 GB, 1.65 GB, 1.78 GB, 1.88 GB,
>> 1.97 GB, 2.01 GB, 2.91 GB, 3.27 GB, 3.46 GB, 3.61 GB, 3.72 GB, 3.80 GB, 3.90
>> GB, 3.96 GB, 4.16 GB, 5.05 GB, 5.36 GB, 5.60 GB, 5.76 GB, 5.87 GB, 5.94 GB,
>> 6.57 GB, 7.32 GB, 7.57 GB, 7.75 GB, 7.86 GB, 7.92 GB, 8.43 GB, 9.18 GB, 9.43
>> GB, 9.62 GB, 9.75 GB, 9.84 GB, 9.89 GB, 9.94 GB, 10.41 GB, 11.07 GB, 11.24
>> GB, 11.42 GB, 11.66 GB, 11.82 GB, 12.05 GB, 12.48 GB, 12.96 GB, 13.07 GB,
>> 13.19 GB, 13.31 GB, 13.38 GB, 13.46 GB, 13.51 GB, 13.57 GB, 13.61 GB
>>            - FooterProcessingTime: (Avg: 381.833ms ; Min: 46.831ms ; Max:
>> 4s214ms ; Number of samples: 56)
>>            - AverageHdfsReadThreadConcurrency: 0.00
>>            - AverageScannerThreadConcurrency: 7.80
>>            - BytesRead: 13.63 GB (14635606518)
>>            - BytesReadDataNodeCache: 0
>>            - BytesReadLocal: 13.63 GB (14635606518)
>>            - BytesReadRemoteUnexpected: 0
>>            - BytesReadShortCircuit: 0
>>            - CachedFileHandlesHitCount: 2.27K (2268)
>>            - CachedFileHandlesMissCount: 56 (56)
>>            - CollectionItemsRead: 0 (0)
>>            - DecompressionTime: 19s612ms
>>            - MaxCompressedTextFileLength: 0
>>            - NumColumns: 16 (16)
>>            - NumDictFilteredRowGroups: 0 (0)
>>            - NumDisksAccessed: 1 (1)
>>            - NumRowGroups: 56 (56)
>>            - NumScannerThreadsStarted: 8 (8)
>>            - NumScannersWithNoReads: 0 (0)
>>            - NumStatsFilteredRowGroups: 0 (0)
>>            - PeakMemoryUsage: 1.37 GB (1466047697)
>>            - PerReadThreadRawHdfsThroughput: 100.45 MB/sec
>>            - RemoteScanRanges: 0 (0)
>>            - RowBatchQueueGetWaitTime: 1s872ms
>>            - RowBatchQueuePutWaitTime: 238h55m
>>            - RowsRead: 424.29M (424287956)
>>            - RowsReturned: 424.29M (424287956)
>>            - RowsReturnedRate: 98.57 M/sec
>>            - ScanRangesComplete: 56 (56)
>>            - ScannerThreadsInvoluntaryContextSwitches: 12.67K (12674)
>>            - ScannerThreadsTotalWallClockTime: 238h58m
>>              - MaterializeTupleTime(*): 1m57s
>>              - ScannerThreadsSysTime: 3s421ms
>>              - ScannerThreadsUserTime: 2m31s
>>            - ScannerThreadsVoluntaryContextSwitches: 415.49K (415490)
>>            - TotalRawHdfsOpenFileTime(*): 2s904ms
>>            - TotalRawHdfsReadTime(*): 2m18s
>>            - TotalReadThroughput: 129.60 KB/sec
>>         CodeGen:(Total: 915.511ms, non-child: 915.511ms, % non-child:
>> 100.00%)
>>            - CodegenInvoluntaryContextSwitches: 10 (10)
>>            - CodegenTotalWallClockTime: 915.504ms
>>              - CodegenSysTime: 8.256ms
>>              - CodegenUserTime: 45.032ms
>>            - CodegenVoluntaryContextSwitches: 87 (87)
>>            - CompileTime: 319.788ms
>>            - IrGenerationTime: 276.446ms
>>            - LoadTime: 0.000ns
>>            - ModuleBitcodeSize: 2.26 MB (2373832)
>>            - NumFunctions: 19 (19)
>>            - NumInstructions: 706 (706)
>>            - OptimizationTime: 301.622ms
>>            - PeakMemoryUsage: 353.00 KB (361472)
>>            - PrepareTime: 16.858ms
>>       Instance 9649f7ab3bcc5fb8:f4d6a60700000002
>> (host=slave1:22000):(Total: 30h17m, non-child: 1s508ms, % non-child: 0.00%)
>>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:57/14.10 GB
>>         Fragment Instance Lifecycle Event Timeline: 30h17m
>>            - Prepare Finished: 32.853ms (32.853ms)
>>            - Open Finished: 362.691ms (329.838ms)
>>            - First Batch Produced: 1s725ms (1s362ms)
>>            - First Batch Sent: 1s728ms (3.415ms)
>>            - ExecInternal Finished: 30h17m (30h17m)
>>         MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.27 GB, 1.07 GB,
>> 798.75 MB, 1.36 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.19 GB, 974.24 MB,
>> 800.43 MB, 1.27 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.26 GB, 1010.99 MB, 805.20
>> MB, 1.29 GB, 1.38 GB, 1.38 GB, 1.27 GB, 987.74 MB, 811.33 MB, 1.27 GB, 1.38
>> GB, 1.38 GB, 1.34 GB, 1.18 GB, 922.96 MB, 765.77 MB, 1.13 GB, 1.36 GB, 1.36
>> GB, 1.36 GB, 1.37 GB, 1.30 GB, 1.03 GB, 919.88 MB, 1.38 GB, 1.38 GB, 1.38
>> GB, 1.39 GB, 1.35 GB, 1.24 GB, 1.17 GB, 1005.37 MB, 844.55 MB, 751.70 MB,
>> 518.60 MB
>>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 5
>>          - AverageThreadTokens: 8.91
>>          - BloomFilterBytes: 0
>>          - PeakMemoryUsage: 1.40 GB (1500404835)
>>          - PeakReservation: 0
>>          - PeakUsedReservation: 0
>>          - PerHostPeakMemUsage: 1.40 GB (1500404835)
>>          - RowsProduced: 434.96M (434955041)
>>          - TotalNetworkReceiveTime: 0.000ns
>>          - TotalNetworkSendTime: 0.000ns
>>          - TotalStorageWaitTime: 2m56s
>>          - TotalThreadsInvoluntaryContextSwitches: 370.59K (370587)
>>          - TotalThreadsTotalWallClockTime: 269h58m
>>            - TotalThreadsSysTime: 1m58s
>>            - TotalThreadsUserTime: 1h9m
>>          - TotalThreadsVoluntaryContextSwitches: 437.63K (437630)
>>         Buffer pool:
>>            - AllocTime: 0.000ns
>>            - CumulativeAllocationBytes: 0
>>            - CumulativeAllocations: 0 (0)
>>            - PeakReservation: 0
>>            - PeakUnpinnedBytes: 0
>>            - PeakUsedReservation: 0
>>            - ReadIoBytes: 0
>>            - ReadIoOps: 0 (0)
>>            - ReadIoWaitTime: 0.000ns
>>            - ReservationLimit: 0
>>            - WriteIoBytes: 0
>>            - WriteIoOps: 0 (0)
>>            - WriteIoWaitTime: 0.000ns
>>         Fragment Instance Lifecycle Timings:
>>            - ExecTime: 30h17m
>>              - ExecTreeExecTime: 4s575ms
>>            - OpenTime: 329.834ms
>>              - ExecTreeOpenTime: 45.986us
>>            - PrepareTime: 32.581ms
>>              - ExecTreePrepareTime: 18.647ms
>>         KuduTableSink:(Total: 30h17m, non-child: 30h17m, % non-child:
>> 100.00%)
>>            - KuduApplyTimer: 29h22m
>>            - NumRowErrors: 357.86M (357862755)
>>            - PeakMemoryUsage: 20.00 MB (20971520)
>>            - RowsProcessedRate: 3.99 K/sec
>>            - TotalNumRows: 434.96M (434955041)
>>         HDFS_SCAN_NODE (id=0):(Total: 4s229ms, non-child: 4s229ms, %
>> non-child: 100.00%)
>>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:57/14.10 GB
>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of
>> 57
>>           Hdfs Read Thread Concurrency Bucket: 0:99.63% 1:0.3706% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>>           File Formats: PARQUET/SNAPPY:57
>>           BytesRead(34m8s): 1.32 GB, 1.59 GB, 1.74 GB, 1.85 GB, 1.94 GB,
>> 2.11 GB, 3.21 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB, 3.94 GB, 4.11
>> GB, 5.04 GB, 5.32 GB, 5.50 GB, 5.67 GB, 5.81 GB, 5.92 GB, 6.09 GB, 7.06 GB,
>> 7.40 GB, 7.63 GB, 7.79 GB, 7.90 GB, 8.08 GB, 9.02 GB, 9.35 GB, 9.55 GB, 9.70
>> GB, 9.81 GB, 9.90 GB, 10.01 GB, 10.74 GB, 11.19 GB, 11.34 GB, 11.48 GB,
>> 11.57 GB, 11.69 GB, 11.84 GB, 12.25 GB, 13.25 GB, 13.41 GB, 13.50 GB, 13.60
>> GB, 13.67 GB, 13.73 GB, 13.79 GB, 13.83 GB, 13.86 GB, 13.92 GB, 14.03 GB
>>            - FooterProcessingTime: (Avg: 414.198ms ; Min: 51.291ms ; Max:
>> 2s894ms ; Number of samples: 57)
>>            - AverageHdfsReadThreadConcurrency: 0.00
>>            - AverageScannerThreadConcurrency: 7.91
>>            - BytesRead: 14.11 GB (15148002597)
>>            - BytesReadDataNodeCache: 0
>>            - BytesReadLocal: 14.11 GB (15148002597)
>>            - BytesReadRemoteUnexpected: 0
>>            - BytesReadShortCircuit: 0
>>            - CachedFileHandlesHitCount: 2.33K (2333)
>>            - CachedFileHandlesMissCount: 57 (57)
>>            - CollectionItemsRead: 0 (0)
>>            - DecompressionTime: 23s330ms
>>            - MaxCompressedTextFileLength: 0
>>            - NumColumns: 16 (16)
>>            - NumDictFilteredRowGroups: 0 (0)
>>            - NumDisksAccessed: 1 (1)
>>            - NumRowGroups: 57 (57)
>>            - NumScannerThreadsStarted: 8 (8)
>>            - NumScannersWithNoReads: 0 (0)
>>            - NumStatsFilteredRowGroups: 0 (0)
>>            - PeakMemoryUsage: 1.38 GB (1479367012)
>>            - PerReadThreadRawHdfsThroughput: 36.38 MB/sec
>>            - RemoteScanRanges: 0 (0)
>>            - RowBatchQueueGetWaitTime: 1s354ms
>>            - RowBatchQueuePutWaitTime: 239h35m
>>            - RowsRead: 434.96M (434955041)
>>            - RowsReturned: 434.96M (434955041)
>>            - RowsReturnedRate: 102.83 M/sec
>>            - ScanRangesComplete: 57 (57)
>>            - ScannerThreadsInvoluntaryContextSwitches: 7.23K (7231)
>>            - ScannerThreadsTotalWallClockTime: 239h40m
>>              - MaterializeTupleTime(*): 2m1s
>>              - ScannerThreadsSysTime: 4s044ms
>>              - ScannerThreadsUserTime: 2m42s
>>            - ScannerThreadsVoluntaryContextSwitches: 425.94K (425935)
>>            - TotalRawHdfsOpenFileTime(*): 5s555ms
>>            - TotalRawHdfsReadTime(*): 6m37s
>>            - TotalReadThroughput: 135.68 KB/sec
>>         CodeGen:(Total: 274.452ms, non-child: 274.452ms, % non-child:
>> 100.00%)
>>            - CodegenInvoluntaryContextSwitches: 4 (4)
>>            - CodegenTotalWallClockTime: 274.443ms
>>              - CodegenSysTime: 7.440ms
>>              - CodegenUserTime: 40.295ms
>>            - CodegenVoluntaryContextSwitches: 86 (86)
>>            - CompileTime: 103.404ms
>>            - IrGenerationTime: 32.648ms
>>            - LoadTime: 0.000ns
>>            - ModuleBitcodeSize: 2.26 MB (2373832)
>>            - NumFunctions: 19 (19)
>>            - NumInstructions: 706 (706)
>>            - OptimizationTime: 122.033ms
>>            - PeakMemoryUsage: 353.00 KB (361472)
>>            - PrepareTime: 15.943ms
>>       Instance 9649f7ab3bcc5fb8:f4d6a60700000001
>> (host=slave4:22000):(Total: 30h16m, non-child: 2s019ms, % non-child: 0.00%)
>>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>>         Fragment Instance Lifecycle Event Timeline: 30h16m
>>            - Prepare Finished: 105.379ms (105.379ms)
>>            - Open Finished: 519.128ms (413.748ms)
>>            - First Batch Produced: 1s789ms (1s270ms)
>>            - First Batch Sent: 1s795ms (6.364ms)
>>            - ExecInternal Finished: 30h16m (30h16m)
>>         MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.23 GB,
>> 924.30 MB, 715.63 MB, 1.26 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.30 GB,
>> 1.08 GB, 843.90 MB, 887.06 MB, 1.40 GB, 1.41 GB, 1.39 GB, 1.19 GB, 826.21
>> MB, 1.19 GB, 1.38 GB, 1.38 GB, 1.31 GB, 1.05 GB, 748.49 MB, 1.17 GB, 1.38
>> GB, 1.38 GB, 1.38 GB, 1.24 GB, 999.76 MB, 733.27 MB, 995.28 MB, 1.35 GB,
>> 1.38 GB, 1.38 GB, 1.38 GB, 1.35 GB, 1.10 GB, 861.01 MB, 1.39 GB, 1.40 GB,
>> 1.40 GB, 1.40 GB, 1.39 GB, 1.33 GB, 1.24 GB, 1.13 GB, 1007.59 MB, 820.51 MB,
>> 655.33 MB
>>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
>>          - AverageThreadTokens: 8.97
>>          - BloomFilterBytes: 0
>>          - PeakMemoryUsage: 1.41 GB (1513166691)
>>          - PeakReservation: 0
>>          - PeakUsedReservation: 0
>>          - PerHostPeakMemUsage: 1.41 GB (1513166691)
>>          - RowsProduced: 425.55M (425550934)
>>          - TotalNetworkReceiveTime: 0.000ns
>>          - TotalNetworkSendTime: 0.000ns
>>          - TotalStorageWaitTime: 1m49s
>>          - TotalThreadsInvoluntaryContextSwitches: 364.52K (364523)
>>          - TotalThreadsTotalWallClockTime: 271h36m
>>            - TotalThreadsSysTime: 1m56s
>>            - TotalThreadsUserTime: 1h9m
>>          - TotalThreadsVoluntaryContextSwitches: 431.64K (431643)
>>         Buffer pool:
>>            - AllocTime: 0.000ns
>>            - CumulativeAllocationBytes: 0
>>            - CumulativeAllocations: 0 (0)
>>            - PeakReservation: 0
>>            - PeakUnpinnedBytes: 0
>>            - PeakUsedReservation: 0
>>            - ReadIoBytes: 0
>>            - ReadIoOps: 0 (0)
>>            - ReadIoWaitTime: 0.000ns
>>            - ReservationLimit: 0
>>            - WriteIoBytes: 0
>>            - WriteIoOps: 0 (0)
>>            - WriteIoWaitTime: 0.000ns
>>         Fragment Instance Lifecycle Timings:
>>            - ExecTime: 30h16m
>>              - ExecTreeExecTime: 4s185ms
>>            - OpenTime: 413.744ms
>>              - ExecTreeOpenTime: 31.615us
>>            - PrepareTime: 105.138ms
>>              - ExecTreePrepareTime: 17.849ms
>>         KuduTableSink:(Total: 30h16m, non-child: 30h16m, % non-child:
>> 100.00%)
>>            - KuduApplyTimer: 29h20m
>>            - NumRowErrors: 336.92M (336918666)
>>            - PeakMemoryUsage: 20.00 MB (20971520)
>>            - RowsProcessedRate: 3.90 K/sec
>>            - TotalNumRows: 425.55M (425550934)
>>         HDFS_SCAN_NODE (id=0):(Total: 3s886ms, non-child: 3s886ms, %
>> non-child: 100.00%)
>>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of
>> 56
>>           Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.1368% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>>           File Formats: PARQUET/SNAPPY:56
>>           BytesRead(34m8s): 1.29 GB, 1.51 GB, 1.66 GB, 1.79 GB, 1.88 GB,
>> 1.97 GB, 2.00 GB, 3.01 GB, 3.30 GB, 3.49 GB, 3.63 GB, 3.74 GB, 3.82 GB, 3.92
>> GB, 3.96 GB, 4.29 GB, 5.26 GB, 5.47 GB, 5.71 GB, 5.87 GB, 5.94 GB, 6.88 GB,
>> 7.34 GB, 7.59 GB, 7.76 GB, 7.89 GB, 7.93 GB, 8.83 GB, 9.30 GB, 9.51 GB, 9.67
>> GB, 9.79 GB, 9.89 GB, 9.91 GB, 10.45 GB, 11.14 GB, 11.30 GB, 11.46 GB, 11.55
>> GB, 11.68 GB, 11.83 GB, 12.10 GB, 13.21 GB, 13.35 GB, 13.47 GB, 13.56 GB,
>> 13.64 GB, 13.71 GB, 13.76 GB, 13.81 GB, 13.86 GB, 13.87 GB, 13.88 GB
>>            - FooterProcessingTime: (Avg: 331.439ms ; Min: 46.760ms ; Max:
>> 2s508ms ; Number of samples: 56)
>>            - AverageHdfsReadThreadConcurrency: 0.00
>>            - AverageScannerThreadConcurrency: 7.97
>>            - BytesRead: 13.88 GB (14898393824)
>>            - BytesReadDataNodeCache: 0
>>            - BytesReadLocal: 13.88 GB (14898393824)
>>            - BytesReadRemoteUnexpected: 0
>>            - BytesReadShortCircuit: 0
>>            - CachedFileHandlesHitCount: 2.30K (2296)
>>            - CachedFileHandlesMissCount: 56 (56)
>>            - CollectionItemsRead: 0 (0)
>>            - DecompressionTime: 22s686ms
>>            - MaxCompressedTextFileLength: 0
>>            - NumColumns: 16 (16)
>>            - NumDictFilteredRowGroups: 0 (0)
>>            - NumDisksAccessed: 1 (1)
>>            - NumRowGroups: 56 (56)
>>            - NumScannerThreadsStarted: 8 (8)
>>            - NumScannersWithNoReads: 0 (0)
>>            - NumStatsFilteredRowGroups: 0 (0)
>>            - PeakMemoryUsage: 1.39 GB (1491938707)
>>            - PerReadThreadRawHdfsThroughput: 95.20 MB/sec
>>            - RemoteScanRanges: 0 (0)
>>            - RowBatchQueueGetWaitTime: 1s255ms
>>            - RowBatchQueuePutWaitTime: 241h15m
>>            - RowsRead: 425.55M (425550934)
>>            - RowsReturned: 425.55M (425550934)
>>            - RowsReturnedRate: 109.50 M/sec
>>            - ScanRangesComplete: 56 (56)
>>            - ScannerThreadsInvoluntaryContextSwitches: 12.02K (12019)
>>            - ScannerThreadsTotalWallClockTime: 241h20m
>>              - MaterializeTupleTime(*): 2m6s
>>              - ScannerThreadsSysTime: 3s548ms
>>              - ScannerThreadsUserTime: 2m39s
>>            - ScannerThreadsVoluntaryContextSwitches: 416.75K (416750)
>>            - TotalRawHdfsOpenFileTime(*): 3s230ms
>>            - TotalRawHdfsReadTime(*): 2m29s
>>            - TotalReadThroughput: 133.55 KB/sec
>>         CodeGen:(Total: 315.226ms, non-child: 315.226ms, % non-child:
>> 100.00%)
>>            - CodegenInvoluntaryContextSwitches: 9 (9)
>>            - CodegenTotalWallClockTime: 315.217ms
>>              - CodegenSysTime: 5.142ms
>>              - CodegenUserTime: 43.310ms
>>            - CodegenVoluntaryContextSwitches: 91 (91)
>>            - CompileTime: 129.393ms
>>            - IrGenerationTime: 60.406ms
>>            - LoadTime: 0.000ns
>>            - ModuleBitcodeSize: 2.26 MB (2373832)
>>            - NumFunctions: 19 (19)
>>            - NumInstructions: 706 (706)
>>            - OptimizationTime: 111.534ms
>>            - PeakMemoryUsage: 353.00 KB (361472)
>>            - PrepareTime: 13.425ms
>>       Instance 9649f7ab3bcc5fb8:f4d6a60700000004
>> (host=slave7:22000):(Total: 30h10m, non-child: 16s427ms, % non-child: 0.02%)
>>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:58/14.03 GB
>>         Fragment Instance Lifecycle Event Timeline: 30h10m
>>            - Prepare Finished: 105.361ms (105.361ms)
>>            - Open Finished: 980.610ms (875.249ms)
>>            - First Batch Produced: 4s553ms (3s573ms)
>>            - First Batch Sent: 4s559ms (5.668ms)
>>            - ExecInternal Finished: 30h10m (30h10m)
>>         MemoryUsage(34m8s): 1.38 GB, 1.40 GB, 1.40 GB, 1.28 GB, 1.08 GB,
>> 796.55 MB, 1.34 GB, 1.37 GB, 1.35 GB, 1.34 GB, 1.27 GB, 1.21 GB, 983.86 MB,
>> 1.08 GB, 1.38 GB, 1.36 GB, 1.35 GB, 1.28 GB, 1.17 GB, 921.06 MB, 1.02 GB,
>> 1.35 GB, 1.33 GB, 1.28 GB, 1.15 GB, 905.74 MB, 1.01 GB, 1.36 GB, 1.34 GB,
>> 1.32 GB, 1.21 GB, 1.10 GB, 877.34 MB, 1.04 GB, 1.32 GB, 1.38 GB, 1.36 GB,
>> 1.35 GB, 1.32 GB, 1.21 GB, 1000.76 MB, 1.11 GB, 1.37 GB, 1.36 GB, 1.33 GB,
>> 1.29 GB, 1.20 GB, 1.09 GB, 1.07 GB, 937.32 MB, 814.51 MB, 480.87 MB, 155.08
>> MB
>>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 5, 2
>>          - AverageThreadTokens: 8.80
>>          - BloomFilterBytes: 0
>>          - PeakMemoryUsage: 1.40 GB (1500511831)
>>          - PeakReservation: 0
>>          - PeakUsedReservation: 0
>>          - PerHostPeakMemUsage: 1.40 GB (1500511831)
>>          - RowsProduced: 433.04M (433043954)
>>          - TotalNetworkReceiveTime: 0.000ns
>>          - TotalNetworkSendTime: 0.000ns
>>          - TotalStorageWaitTime: 3m18s
>>          - TotalThreadsInvoluntaryContextSwitches: 374.99K (374993)
>>          - TotalThreadsTotalWallClockTime: 265h25m
>>            - TotalThreadsSysTime: 1m57s
>>            - TotalThreadsUserTime: 1h10m
>>          - TotalThreadsVoluntaryContextSwitches: 435.67K (435665)
>>         Buffer pool:
>>            - AllocTime: 0.000ns
>>            - CumulativeAllocationBytes: 0
>>            - CumulativeAllocations: 0 (0)
>>            - PeakReservation: 0
>>            - PeakUnpinnedBytes: 0
>>            - PeakUsedReservation: 0
>>            - ReadIoBytes: 0
>>            - ReadIoOps: 0 (0)
>>            - ReadIoWaitTime: 0.000ns
>>            - ReservationLimit: 0
>>            - WriteIoBytes: 0
>>            - WriteIoOps: 0 (0)
>>            - WriteIoWaitTime: 0.000ns
>>         Fragment Instance Lifecycle Timings:
>>            - ExecTime: 30h10m
>>              - ExecTreeExecTime: 6s426ms
>>            - OpenTime: 875.240ms
>>              - ExecTreeOpenTime: 41.994us
>>            - PrepareTime: 105.102ms
>>              - ExecTreePrepareTime: 24.800ms
>>         KuduTableSink:(Total: 30h9m, non-child: 30h9m, % non-child:
>> 100.00%)
>>            - KuduApplyTimer: 29h12m
>>            - NumRowErrors: 357.43M (357426783)
>>            - PeakMemoryUsage: 20.00 MB (20971520)
>>            - RowsProcessedRate: 3.99 K/sec
>>            - TotalNumRows: 433.04M (433043954)
>>         HDFS_SCAN_NODE (id=0):(Total: 6s154ms, non-child: 6s154ms, %
>> non-child: 100.00%)
>>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:58/14.03 GB
>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 58 out of
>> 58
>>           Hdfs Read Thread Concurrency Bucket: 0:99.68% 1:0.3169% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>>           File Formats: PARQUET/SNAPPY:58
>>           BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB,
>> 2.10 GB, 3.18 GB, 3.29 GB, 3.44 GB, 3.62 GB, 3.72 GB, 3.95 GB, 4.03 GB, 4.61
>> GB, 5.22 GB, 5.37 GB, 5.52 GB, 5.66 GB, 5.90 GB, 6.01 GB, 6.53 GB, 7.22 GB,
>> 7.43 GB, 7.62 GB, 7.83 GB, 7.99 GB, 8.49 GB, 9.18 GB, 9.38 GB, 9.55 GB, 9.67
>> GB, 9.88 GB, 9.98 GB, 10.53 GB, 11.04 GB, 11.22 GB, 11.39 GB, 11.50 GB,
>> 11.58 GB, 11.70 GB, 11.95 GB, 12.61 GB, 13.30 GB, 13.43 GB, 13.52 GB, 13.59
>> GB, 13.65 GB, 13.72 GB, 13.91 GB, 13.94 GB, 13.96 GB, 13.98 GB, 14.03 GB
>>            - FooterProcessingTime: (Avg: 615.194ms ; Min: 50.813ms ; Max:
>> 6s740ms ; Number of samples: 58)
>>            - AverageHdfsReadThreadConcurrency: 0.00
>>            - AverageScannerThreadConcurrency: 7.80
>>            - BytesRead: 14.04 GB (15072987184)
>>            - BytesReadDataNodeCache: 0
>>            - BytesReadLocal: 14.04 GB (15072987184)
>>            - BytesReadRemoteUnexpected: 0
>>            - BytesReadShortCircuit: 0
>>            - CachedFileHandlesHitCount: 2.34K (2340)
>>            - CachedFileHandlesMissCount: 58 (58)
>>            - CollectionItemsRead: 0 (0)
>>            - DecompressionTime: 21s821ms
>>            - MaxCompressedTextFileLength: 0
>>            - NumColumns: 16 (16)
>>            - NumDictFilteredRowGroups: 0 (0)
>>            - NumDisksAccessed: 1 (1)
>>            - NumRowGroups: 58 (58)
>>            - NumScannerThreadsStarted: 8 (8)
>>            - NumScannersWithNoReads: 0 (0)
>>            - NumStatsFilteredRowGroups: 0 (0)
>>            - PeakMemoryUsage: 1.38 GB (1479283937)
>>            - PerReadThreadRawHdfsThroughput: 41.17 MB/sec
>>            - RemoteScanRanges: 0 (0)
>>            - RowBatchQueueGetWaitTime: 3s555ms
>>            - RowBatchQueuePutWaitTime: 235h9m
>>            - RowsRead: 433.04M (433043954)
>>            - RowsReturned: 433.04M (433043954)
>>            - RowsReturnedRate: 70.36 M/sec
>>            - ScanRangesComplete: 58 (58)
>>            - ScannerThreadsInvoluntaryContextSwitches: 16.45K (16447)
>>            - ScannerThreadsTotalWallClockTime: 235h15m
>>              - MaterializeTupleTime(*): 2m5s
>>              - ScannerThreadsSysTime: 3s924ms
>>              - ScannerThreadsUserTime: 2m40s
>>            - ScannerThreadsVoluntaryContextSwitches: 424.08K (424080)
>>            - TotalRawHdfsOpenFileTime(*): 4s852ms
>>            - TotalRawHdfsReadTime(*): 5m49s
>>            - TotalReadThroughput: 135.58 KB/sec
>>         CodeGen:(Total: 771.273ms, non-child: 771.273ms, % non-child:
>> 100.00%)
>>            - CodegenInvoluntaryContextSwitches: 5 (5)
>>            - CodegenTotalWallClockTime: 771.263ms
>>              - CodegenSysTime: 19.916ms
>>              - CodegenUserTime: 41.600ms
>>            - CodegenVoluntaryContextSwitches: 91 (91)
>>            - CompileTime: 404.632ms
>>            - IrGenerationTime: 153.195ms
>>            - LoadTime: 0.000ns
>>            - ModuleBitcodeSize: 2.26 MB (2373832)
>>            - NumFunctions: 19 (19)
>>            - NumInstructions: 706 (706)
>>            - OptimizationTime: 199.856ms
>>            - PeakMemoryUsage: 353.00 KB (361472)
>>            - PrepareTime: 12.885ms
>>       Instance 9649f7ab3bcc5fb8:f4d6a60700000006
>> (host=slave6:22000):(Total: 29h59m, non-child: 21s225ms, % non-child: 0.02%)
>>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:57/13.92 GB
>>         Fragment Instance Lifecycle Event Timeline: 29h59m
>>            - Prepare Finished: 55.600ms (55.600ms)
>>            - Open Finished: 598.215ms (542.614ms)
>>            - First Batch Produced: 2s350ms (1s752ms)
>>            - First Batch Sent: 2s356ms (5.853ms)
>>            - ExecInternal Finished: 29h59m (29h59m)
>>         MemoryUsage(34m8s): 1.37 GB, 1.37 GB, 1.36 GB, 1.24 GB, 1.04 GB,
>> 864.64 MB, 1.37 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.31 GB, 1.17 GB, 965.56 MB,
>> 876.58 MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.38 GB, 1.23 GB, 995.12 MB, 871.65
>> MB, 1.35 GB, 1.40 GB, 1.38 GB, 1.24 GB, 977.74 MB, 850.26 MB, 1.27 GB, 1.29
>> GB, 1.30 GB, 1.31 GB, 1.16 GB, 954.46 MB, 906.32 MB, 1.24 GB, 1.34 GB, 1.30
>> GB, 1.31 GB, 1.36 GB, 1.29 GB, 1.02 GB, 1.02 GB, 1.32 GB, 1.30 GB, 1.31 GB,
>> 1.38 GB, 1.33 GB, 1.23 GB, 1.14 GB, 1017.39 MB, 822.59 MB, 631.09 MB
>>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 7
>>          - AverageThreadTokens: 8.86
>>          - BloomFilterBytes: 0
>>          - PeakMemoryUsage: 1.40 GB (1499916960)
>>          - PeakReservation: 0
>>          - PeakUsedReservation: 0
>>          - PerHostPeakMemUsage: 1.40 GB (1499916960)
>>          - RowsProduced: 428.25M (428247756)
>>          - TotalNetworkReceiveTime: 0.000ns
>>          - TotalNetworkSendTime: 0.000ns
>>          - TotalStorageWaitTime: 1m52s
>>          - TotalThreadsInvoluntaryContextSwitches: 388.30K (388300)
>>          - TotalThreadsTotalWallClockTime: 265h50m
>>            - TotalThreadsSysTime: 1m58s
>>            - TotalThreadsUserTime: 1h13m
>>          - TotalThreadsVoluntaryContextSwitches: 431.32K (431323)
>>         Buffer pool:
>>            - AllocTime: 0.000ns
>>            - CumulativeAllocationBytes: 0
>>            - CumulativeAllocations: 0 (0)
>>            - PeakReservation: 0
>>            - PeakUnpinnedBytes: 0
>>            - PeakUsedReservation: 0
>>            - ReadIoBytes: 0
>>            - ReadIoOps: 0 (0)
>>            - ReadIoWaitTime: 0.000ns
>>            - ReservationLimit: 0
>>            - WriteIoBytes: 0
>>            - WriteIoOps: 0 (0)
>>            - WriteIoWaitTime: 0.000ns
>>         Fragment Instance Lifecycle Timings:
>>            - ExecTime: 29h59m
>>              - ExecTreeExecTime: 4s703ms
>>            - OpenTime: 542.606ms
>>              - ExecTreeOpenTime: 32.022us
>>            - PrepareTime: 55.377ms
>>              - ExecTreePrepareTime: 20.617ms
>>         KuduTableSink:(Total: 29h59m, non-child: 29h59m, % non-child:
>> 100.00%)
>>            - KuduApplyTimer: 29h
>>            - NumRowErrors: 346.33M (346333617)
>>            - PeakMemoryUsage: 20.00 MB (20971520)
>>            - RowsProcessedRate: 3.97 K/sec
>>            - TotalNumRows: 428.25M (428247756)
>>         HDFS_SCAN_NODE (id=0):(Total: 4s407ms, non-child: 4s407ms, %
>> non-child: 100.00%)
>>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:57/13.92 GB
>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of
>> 57
>>           Hdfs Read Thread Concurrency Bucket: 0:99.81% 1:0.1895% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>>           File Formats: PARQUET/SNAPPY:57
>>           BytesRead(34m8s): 1.29 GB, 1.56 GB, 1.71 GB, 1.81 GB, 1.90 GB,
>> 2.18 GB, 3.19 GB, 3.30 GB, 3.46 GB, 3.65 GB, 3.76 GB, 3.84 GB, 3.91 GB, 4.19
>> GB, 5.09 GB, 5.31 GB, 5.48 GB, 5.64 GB, 5.78 GB, 5.90 GB, 6.16 GB, 7.13 GB,
>> 7.39 GB, 7.61 GB, 7.76 GB, 7.91 GB, 8.15 GB, 9.04 GB, 9.25 GB, 9.49 GB, 9.72
>> GB, 9.83 GB, 9.95 GB, 10.16 GB, 10.86 GB, 11.12 GB, 11.26 GB, 11.41 GB,
>> 11.60 GB, 11.71 GB, 11.87 GB, 12.39 GB, 13.18 GB, 13.32 GB, 13.42 GB, 13.63
>> GB, 13.70 GB, 13.76 GB, 13.82 GB, 13.87 GB, 13.89 GB, 13.90 GB
>>            - FooterProcessingTime: (Avg: 368.827ms ; Min: 61.227ms ; Max:
>> 3s788ms ; Number of samples: 57)
>>            - AverageHdfsReadThreadConcurrency: 0.00
>>            - AverageScannerThreadConcurrency: 7.86
>>            - BytesRead: 13.92 GB (14949018844)
>>            - BytesReadDataNodeCache: 0
>>            - BytesReadLocal: 13.92 GB (14949018844)
>>            - BytesReadRemoteUnexpected: 0
>>            - BytesReadShortCircuit: 0
>>            - CachedFileHandlesHitCount: 2.31K (2315)
>>            - CachedFileHandlesMissCount: 57 (57)
>>            - CollectionItemsRead: 0 (0)
>>            - DecompressionTime: 22s734ms
>>            - MaxCompressedTextFileLength: 0
>>            - NumColumns: 16 (16)
>>            - NumDictFilteredRowGroups: 0 (0)
>>            - NumDisksAccessed: 1 (1)
>>            - NumRowGroups: 57 (57)
>>            - NumScannerThreadsStarted: 8 (8)
>>            - NumScannersWithNoReads: 0 (0)
>>            - NumStatsFilteredRowGroups: 0 (0)
>>            - PeakMemoryUsage: 1.38 GB (1478857724)
>>            - PerReadThreadRawHdfsThroughput: 69.53 MB/sec
>>            - RemoteScanRanges: 0 (0)
>>            - RowBatchQueueGetWaitTime: 1s749ms
>>            - RowBatchQueuePutWaitTime: 235h46m
>>            - RowsRead: 428.25M (428247756)
>>            - RowsReturned: 428.25M (428247756)
>>            - RowsReturnedRate: 97.15 M/sec
>>            - ScanRangesComplete: 57 (57)
>>            - ScannerThreadsInvoluntaryContextSwitches: 15.99K (15992)
>>            - ScannerThreadsTotalWallClockTime: 235h50m
>>              - MaterializeTupleTime(*): 2m7s
>>              - ScannerThreadsSysTime: 3s685ms
>>              - ScannerThreadsUserTime: 2m42s
>>            - ScannerThreadsVoluntaryContextSwitches: 419.35K (419351)
>>            - TotalRawHdfsOpenFileTime(*): 3s380ms
>>            - TotalRawHdfsReadTime(*): 3m25s
>>            - TotalReadThroughput: 135.25 KB/sec
>>         CodeGen:(Total: 405.466ms, non-child: 405.466ms, % non-child:
>> 100.00%)
>>            - CodegenInvoluntaryContextSwitches: 2 (2)
>>            - CodegenTotalWallClockTime: 405.458ms
>>              - CodegenSysTime: 0.000ns
>>              - CodegenUserTime: 49.456ms
>>            - CodegenVoluntaryContextSwitches: 91 (91)
>>            - CompileTime: 170.012ms
>>            - IrGenerationTime: 52.154ms
>>            - LoadTime: 0.000ns
>>            - ModuleBitcodeSize: 2.26 MB (2373832)
>>            - NumFunctions: 19 (19)
>>            - NumInstructions: 706 (706)
>>            - OptimizationTime: 168.310ms
>>            - PeakMemoryUsage: 353.00 KB (361472)
>>            - PrepareTime: 14.493ms
>>       Instance 9649f7ab3bcc5fb8:f4d6a60700000003
>> (host=slave2:22000):(Total: 28h48m, non-child: 21s449ms, % non-child: 0.02%)
>>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>>         Fragment Instance Lifecycle Event Timeline: 28h48m
>>            - Prepare Finished: 13.348ms (13.348ms)
>>            - Open Finished: 348.443ms (335.094ms)
>>            - First Batch Produced: 1s505ms (1s157ms)
>>            - First Batch Sent: 1s508ms (3.334ms)
>>            - ExecInternal Finished: 28h48m (28h48m)
>>         MemoryUsage(34m8s): 1.38 GB, 1.38 GB, 1.39 GB, 1.26 GB, 1.06 GB,
>> 777.91 MB, 1.37 GB, 1.38 GB, 1.38 GB, 1.39 GB, 1.33 GB, 1.20 GB, 985.85 MB,
>> 781.77 MB, 1.25 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.25 GB, 1015.91 MB, 756.94
>> MB, 1.27 GB, 1.37 GB, 1.37 GB, 1.27 GB, 1012.70 MB, 775.19 MB, 1.25 GB, 1.41
>> GB, 1.41 GB, 1.30 GB, 1.01 GB, 871.44 MB, 1.34 GB, 1.38 GB, 1.38 GB, 1.38
>> GB, 1.37 GB, 1.31 GB, 1.14 GB, 825.57 MB, 1.32 GB, 1.40 GB, 1.40 GB, 1.37
>> GB, 1.28 GB, 1.17 GB, 1.02 GB, 856.05 MB, 694.00 MB
>>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
>>          - AverageThreadTokens: 8.94
>>          - BloomFilterBytes: 0
>>          - PeakMemoryUsage: 1.41 GB (1512855058)
>>          - PeakReservation: 0
>>          - PeakUsedReservation: 0
>>          - PerHostPeakMemUsage: 1.41 GB (1512855058)
>>          - RowsProduced: 427.11M (427110203)
>>          - TotalNetworkReceiveTime: 0.000ns
>>          - TotalNetworkSendTime: 0.000ns
>>          - TotalStorageWaitTime: 2m27s
>>          - TotalThreadsInvoluntaryContextSwitches: 375.92K (375918)
>>          - TotalThreadsTotalWallClockTime: 257h33m
>>            - TotalThreadsSysTime: 2m12s
>>            - TotalThreadsUserTime: 1h9m
>>          - TotalThreadsVoluntaryContextSwitches: 431.39K (431386)
>>         Buffer pool:
>>            - AllocTime: 0.000ns
>>            - CumulativeAllocationBytes: 0
>>            - CumulativeAllocations: 0 (0)
>>            - PeakReservation: 0
>>            - PeakUnpinnedBytes: 0
>>            - PeakUsedReservation: 0
>>            - ReadIoBytes: 0
>>            - ReadIoOps: 0 (0)
>>            - ReadIoWaitTime: 0.000ns
>>            - ReservationLimit: 0
>>            - WriteIoBytes: 0
>>            - WriteIoOps: 0 (0)
>>            - WriteIoWaitTime: 0.000ns
>>         Fragment Instance Lifecycle Timings:
>>            - ExecTime: 28h48m
>>              - ExecTreeExecTime: 4s366ms
>>            - OpenTime: 335.089ms
>>              - ExecTreeOpenTime: 40.091us
>>            - PrepareTime: 13.257ms
>>              - ExecTreePrepareTime: 12.776ms
>>         KuduTableSink:(Total: 28h48m, non-child: 28h48m, % non-child:
>> 100.00%)
>>            - KuduApplyTimer: 27h51m
>>            - NumRowErrors: 347.46M (347457539)
>>            - PeakMemoryUsage: 20.00 MB (20971520)
>>            - RowsProcessedRate: 4.12 K/sec
>>            - TotalNumRows: 427.11M (427110203)
>>         HDFS_SCAN_NODE (id=0):(Total: 4s001ms, non-child: 4s001ms, %
>> non-child: 100.00%)
>>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of
>> 56
>>           Hdfs Read Thread Concurrency Bucket: 0:99.69% 1:0.3106% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>>           File Formats: PARQUET/SNAPPY:56
>>           BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB,
>> 2.08 GB, 3.22 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB, 3.94 GB, 4.08
>> GB, 5.02 GB, 5.30 GB, 5.48 GB, 5.66 GB, 5.80 GB, 5.91 GB, 6.00 GB, 7.04 GB,
>> 7.37 GB, 7.61 GB, 7.77 GB, 7.90 GB, 8.01 GB, 8.95 GB, 9.39 GB, 9.61 GB, 9.77
>> GB, 9.88 GB, 10.17 GB, 11.12 GB, 11.26 GB, 11.40 GB, 11.56 GB, 11.66 GB,
>> 11.73 GB, 11.82 GB, 11.97 GB, 13.08 GB, 13.48 GB, 13.60 GB, 13.68 GB, 13.74
>> GB, 13.80 GB, 13.85 GB, 13.87 GB, 13.88 GB
>>            - FooterProcessingTime: (Avg: 339.758ms ; Min: 55.620ms ; Max:
>> 2s575ms ; Number of samples: 56)
>>            - AverageHdfsReadThreadConcurrency: 0.00
>>            - AverageScannerThreadConcurrency: 7.94
>>            - BytesRead: 13.88 GB (14898474484)
>>            - BytesReadDataNodeCache: 0
>>            - BytesReadLocal: 13.88 GB (14898474484)
>>            - BytesReadRemoteUnexpected: 0
>>            - BytesReadShortCircuit: 0
>>            - CachedFileHandlesHitCount: 2.29K (2294)
>>            - CachedFileHandlesMissCount: 56 (56)
>>            - CollectionItemsRead: 0 (0)
>>            - DecompressionTime: 23s940ms
>>            - MaxCompressedTextFileLength: 0
>>            - NumColumns: 16 (16)
>>            - NumDictFilteredRowGroups: 0 (0)
>>            - NumDisksAccessed: 1 (1)
>>            - NumRowGroups: 56 (56)
>>            - NumScannerThreadsStarted: 8 (8)
>>            - NumScannersWithNoReads: 0 (0)
>>            - NumStatsFilteredRowGroups: 0 (0)
>>            - PeakMemoryUsage: 1.39 GB (1491692569)
>>            - PerReadThreadRawHdfsThroughput: 44.62 MB/sec
>>            - RemoteScanRanges: 0 (0)
>>            - RowBatchQueueGetWaitTime: 1s156ms
>>            - RowBatchQueuePutWaitTime: 228h40m
>>            - RowsRead: 427.11M (427110203)
>>            - RowsReturned: 427.11M (427110203)
>>            - RowsReturnedRate: 106.74 M/sec
>>            - ScanRangesComplete: 56 (56)
>>            - ScannerThreadsInvoluntaryContextSwitches: 7.41K (7412)
>>            - ScannerThreadsTotalWallClockTime: 228h45m
>>              - MaterializeTupleTime(*): 2m
>>              - ScannerThreadsSysTime: 4s574ms
>>              - ScannerThreadsUserTime: 2m39s
>>            - ScannerThreadsVoluntaryContextSwitches: 418.25K (418255)
>>            - TotalRawHdfsOpenFileTime(*): 4s930ms
>>            - TotalRawHdfsReadTime(*): 5m18s
>>            - TotalReadThroughput: 140.35 KB/sec
>>         CodeGen:(Total: 316.986ms, non-child: 316.986ms, % non-child:
>> 100.00%)
>>            - CodegenInvoluntaryContextSwitches: 7 (7)
>>            - CodegenTotalWallClockTime: 316.976ms
>>              - CodegenSysTime: 19.346ms
>>              - CodegenUserTime: 35.434ms
>>            - CodegenVoluntaryContextSwitches: 87 (87)
>>            - CompileTime: 104.224ms
>>            - IrGenerationTime: 54.920ms
>>            - LoadTime: 0.000ns
>>            - ModuleBitcodeSize: 2.26 MB (2373832)
>>            - NumFunctions: 19 (19)
>>            - NumInstructions: 706 (706)
>>            - OptimizationTime: 143.706ms
>>            - PeakMemoryUsage: 353.00 KB (361472)
>>            - PrepareTime: 13.284ms
>>       Instance 9649f7ab3bcc5fb8:f4d6a60700000005
>> (host=slave5:22000):(Total: 28h47m, non-child: 30s821ms, % non-child: 0.03%)
>>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>>         Fragment Instance Lifecycle Event Timeline: 28h47m
>>            - Prepare Finished: 25.961ms (25.961ms)
>>            - Open Finished: 92.181ms (66.220ms)
>>            - First Batch Produced: 1s428ms (1s336ms)
>>            - First Batch Sent: 1s432ms (3.458ms)
>>            - ExecInternal Finished: 28h47m (28h47m)
>>         MemoryUsage(34m8s): 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB, 1.07 GB,
>> 786.07 MB, 1.37 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.31 GB, 1.19 GB, 973.49 MB,
>> 778.55 MB, 1.26 GB, 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB, 1.00 GB, 812.17 MB,
>> 1.29 GB, 1.41 GB, 1.39 GB, 1.14 GB, 881.39 MB, 1.30 GB, 1.37 GB, 1.37 GB,
>> 1.33 GB, 1.16 GB, 910.68 MB, 880.78 MB, 1.32 GB, 1.38 GB, 1.38 GB, 1.38 GB,
>> 1.37 GB, 1.29 GB, 1.13 GB, 843.04 MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.35 GB,
>> 1.27 GB, 1.16 GB, 1.00 GB, 848.32 MB, 709.30 MB
>>         ThreadUsage(34m8s): 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
>>          - AverageThreadTokens: 8.94
>>          - BloomFilterBytes: 0
>>          - PeakMemoryUsage: 1.41 GB (1513059939)
>>          - PeakReservation: 0
>>          - PeakUsedReservation: 0
>>          - PerHostPeakMemUsage: 1.41 GB (1513059939)
>>          - RowsProduced: 426.83M (426832398)
>>          - TotalNetworkReceiveTime: 0.000ns
>>          - TotalNetworkSendTime: 0.000ns
>>          - TotalStorageWaitTime: 1m16s
>>          - TotalThreadsInvoluntaryContextSwitches: 365.02K (365022)
>>          - TotalThreadsTotalWallClockTime: 257h18m
>>            - TotalThreadsSysTime: 1m56s
>>            - TotalThreadsUserTime: 1h9m
>>          - TotalThreadsVoluntaryContextSwitches: 431.61K (431609)
>>         Buffer pool:
>>            - AllocTime: 0.000ns
>>            - CumulativeAllocationBytes: 0
>>            - CumulativeAllocations: 0 (0)
>>            - PeakReservation: 0
>>            - PeakUnpinnedBytes: 0
>>            - PeakUsedReservation: 0
>>            - ReadIoBytes: 0
>>            - ReadIoOps: 0 (0)
>>            - ReadIoWaitTime: 0.000ns
>>            - ReservationLimit: 0
>>            - WriteIoBytes: 0
>>            - WriteIoOps: 0 (0)
>>            - WriteIoWaitTime: 0.000ns
>>         Fragment Instance Lifecycle Timings:
>>            - ExecTime: 28h47m
>>              - ExecTreeExecTime: 4s226ms
>>            - OpenTime: 66.213ms
>>              - ExecTreeOpenTime: 30.358us
>>            - PrepareTime: 25.760ms
>>              - ExecTreePrepareTime: 25.484ms
>>         KuduTableSink:(Total: 28h46m, non-child: 28h46m, % non-child:
>> 100.00%)
>>            - KuduApplyTimer: 27h51m
>>            - NumRowErrors: 345.67M (345674429)
>>            - PeakMemoryUsage: 20.00 MB (20971520)
>>            - RowsProcessedRate: 4.12 K/sec
>>            - TotalNumRows: 426.83M (426832398)
>>         HDFS_SCAN_NODE (id=0):(Total: 3s937ms, non-child: 3s937ms, %
>> non-child: 100.00%)
>>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of
>> 56
>>           Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.138% 2:0% 3:0%
>> 4:0% 5:0% 6:0%
>>           File Formats: PARQUET/SNAPPY:56
>>           BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB,
>> 2.08 GB, 3.23 GB, 3.30 GB, 3.48 GB, 3.67 GB, 3.78 GB, 3.87 GB, 3.94 GB, 4.09
>> GB, 5.01 GB, 5.31 GB, 5.49 GB, 5.66 GB, 5.81 GB, 5.92 GB, 6.09 GB, 7.05 GB,
>> 7.45 GB, 7.69 GB, 7.86 GB, 8.14 GB, 9.07 GB, 9.35 GB, 9.56 GB, 9.70 GB, 9.81
>> GB, 9.90 GB, 10.23 GB, 11.09 GB, 11.27 GB, 11.41 GB, 11.56 GB, 11.66 GB,
>> 11.73 GB, 11.82 GB, 12.00 GB, 13.08 GB, 13.48 GB, 13.61 GB, 13.68 GB, 13.74
>> GB, 13.80 GB, 13.85 GB, 13.87 GB, 13.88 GB
>>            - FooterProcessingTime: (Avg: 256.473ms ; Min: 49.349ms ; Max:
>> 2s343ms ; Number of samples: 56)
>>            - AverageHdfsReadThreadConcurrency: 0.00
>>            - AverageScannerThreadConcurrency: 7.94
>>            - BytesRead: 13.88 GB (14898579078)
>>            - BytesReadDataNodeCache: 0
>>            - BytesReadLocal: 13.88 GB (14898579078)
>>            - BytesReadRemoteUnexpected: 0
>>            - BytesReadShortCircuit: 0
>>            - CachedFileHandlesHitCount: 2.29K (2293)
>>            - CachedFileHandlesMissCount: 56 (56)
>>            - CollectionItemsRead: 0 (0)
>>            - DecompressionTime: 22s127ms
>>            - MaxCompressedTextFileLength: 0
>>            - NumColumns: 16 (16)
>>            - NumDictFilteredRowGroups: 0 (0)
>>            - NumDisksAccessed: 1 (1)
>>            - NumRowGroups: 56 (56)
>>            - NumScannerThreadsStarted: 8 (8)
>>            - NumScannersWithNoReads: 0 (0)
>>            - NumStatsFilteredRowGroups: 0 (0)
>>            - PeakMemoryUsage: 1.39 GB (1491831945)
>>            - PerReadThreadRawHdfsThroughput: 100.21 MB/sec
>>            - RemoteScanRanges: 0 (0)
>>            - RowBatchQueueGetWaitTime: 1s334ms
>>            - RowBatchQueuePutWaitTime: 228h26m
>>            - RowsRead: 426.83M (426832398)
>>            - RowsReturned: 426.83M (426832398)
>>            - RowsReturnedRate: 108.39 M/sec
>>            - ScanRangesComplete: 56 (56)
>>            - ScannerThreadsInvoluntaryContextSwitches: 14.55K (14553)
>>            - ScannerThreadsTotalWallClockTime: 228h30m
>>              - MaterializeTupleTime(*): 2m3s
>>              - ScannerThreadsSysTime: 4s079ms
>>              - ScannerThreadsUserTime: 2m39s
>>            - ScannerThreadsVoluntaryContextSwitches: 417.98K (417975)
>>            - TotalRawHdfsOpenFileTime(*): 3s114ms
>>            - TotalRawHdfsReadTime(*): 2m21s
>>            - TotalReadThroughput: 140.45 KB/sec
>>         CodeGen:(Total: 56.104ms, non-child: 56.104ms, % non-child:
>> 100.00%)
>>            - CodegenInvoluntaryContextSwitches: 3 (3)
>>            - CodegenTotalWallClockTime: 56.095ms
>>              - CodegenSysTime: 0.000ns
>>              - CodegenUserTime: 37.520ms
>>            - CodegenVoluntaryContextSwitches: 1 (1)
>>            - CompileTime: 23.495ms
>>            - IrGenerationTime: 1.462ms
>>            - LoadTime: 0.000ns
>>            - ModuleBitcodeSize: 2.26 MB (2373832)
>>            - NumFunctions: 19 (19)
>>            - NumInstructions: 706 (706)
>>            - OptimizationTime: 15.936ms
>>            - PeakMemoryUsage: 353.00 KB (361472)
>>            - PrepareTime: 14.604ms
>>
>>
>>
>> On Thu, May 10, 2018 at 10:33 AM, Mostafa Mokhtar <mmokhtar@cloudera.com>
>> wrote:
>>>
>>> Can you share the query profile for the successful insert query?
>>>
>>> Thanks
>>> Mostafa
>>>
>>> On May 9, 2018, at 9:55 PM, Geetika Gupta <geetika.gupta@knoldus.in>
>>> wrote:
>>>
>>> Thanks, Jeszy.
>>>
>>> We build impala again with --release flag and data load was successful
>>> after that.
>>>
>>> But now we are facing another issue. The table in which we loaded the
>>> data has less number of rows. We executed the following command:
>>>
>>> insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
>>>
>>> This query was successful, but when we tried the count(*) on both the
>>> tables, row count was different:
>>>
>>> 0: jdbc:hive2://slave2:21050/default> select count(*) from lineitem
>>> . . . . . . . . . . . . . . . . . . > ;
>>> 536870912
>>>
>>> 0: jdbc:hive2://slave2:21050/default> select count(*) from
>>> parquetimpala500.lineitem;
>>> 3000028242
>>>
>>> Do you have any idea about this issue.
>>>
>>>
>>> On Mon, May 7, 2018 at 12:06 PM, Jeszy <jeszyb@gmail.com> wrote:
>>>>
>>>> Impala doesn't store the data itself, so you can switch versions
>>>> without rewriting data. But you don't have to do that, you would just
>>>> have to build impala using the -release flag (of buildall.sh) and run
>>>> it using the release binaries (versus the debug ones). If you would be
>>>> looking at performance, using the release version is highly
>>>> recommended anyway.
>>>>
>>>> On 7 May 2018 at 08:30, Geetika Gupta <geetika.gupta@knoldus.in> wrote:
>>>> > Hi Jeszy,
>>>> >
>>>> > Currently, we are using the apache impala's Github master branch code.
>>>> > We
>>>> > tried using the released version but we encountered some errors
>>>> > related to
>>>> > downloading of dependencies and could not complete the installation.
>>>> >
>>>> > The current version of impala we are using: 2.12
>>>> >
>>>> > We can't try with the new release as we have already loaded 500GB of
>>>> > TPCH
>>>> > data on our cluster.
>>>> >
>>>> > On Mon, May 7, 2018 at 11:43 AM, Jeszy <jeszyb@gmail.com> wrote:
>>>> >>
>>>> >> What version of Impala are you using?
>>>> >> DCHECKs won't be triggered if you run a release build. Looking at the
>>>> >> code, it should work with bad values if not for the DCHECK. Can you
>>>> >> try using a release build?
>>>> >>
>>>> >> On 7 May 2018 at 08:04, Geetika Gupta <geetika.gupta@knoldus.in>
>>>> >> wrote:
>>>> >> > Hi community,
>>>> >> >
>>>> >> > I was trying to load 500GB of TPCH data into kudu table using the
>>>> >> > following
>>>> >> > query:
>>>> >> >
>>>> >> > insert into lineitem select * from PARQUETIMPALA500.LINEITEM
>>>> >> >
>>>> >> > While executing the query for around 17 hrs it got cancelled as the
>>>> >> > impalad
>>>> >> > process of that machine got aborted. Here are the logs of the
>>>> >> > impalad
>>>> >> > process.
>>>> >> >
>>>> >> > impalad.ERROR
>>>> >> >
>>>> >> > Log file created at: 2018/05/06 13:40:34
>>>> >> > Running on machine: slave2
>>>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>> >> > E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged
>>>> >> > to
>>>> >> > this
>>>> >> > file.
>>>> >> > SLF4J: Class path contains multiple SLF4J bindings.
>>>> >> > SLF4J: Found binding in
>>>> >> >
>>>> >> >
>>>> >> > [jar:file:/root/softwares/impala/fe/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>> >> > SLF4J: Found binding in
>>>> >> >
>>>> >> >
>>>> >> > [jar:file:/root/softwares/impala/testdata/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>> >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>>> >> > explanation.
>>>> >> > SLF4J: Actual binding is of type
>>>> >> > [org.slf4j.impl.Log4jLoggerFactory]
>>>> >> > 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load
>>>> >> > native-hadoop
>>>> >> > library for your platform... using builtin-java classes where
>>>> >> > applicable
>>>> >> > 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
>>>> >> > short-circuit
>>>> >> > local reads feature cannot be used because libhadoop cannot be
>>>> >> > loaded.
>>>> >> > tcmalloc: large alloc 1073741824 bytes == 0x484434000 @  0x4135176
>>>> >> > 0x7fd9e9fc3929
>>>> >> > tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @
>>>> >> > 0x4135176
>>>> >> > 0x7fd9e9fc3929
>>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>>> >> > log_entry.count
>>>> >> >> 0 (-1831809966 vs. 0)
>>>> >> > *** Check failure stack trace: ***
>>>> >> >     @          0x3fc0c0d  google::LogMessage::Fail()
>>>> >> >     @          0x3fc24b2  google::LogMessage::SendToLog()
>>>> >> >     @          0x3fc05e7  google::LogMessage::Flush()
>>>> >> >     @          0x3fc3bae
>>>> >> > google::LogMessageFatal::~LogMessageFatal()
>>>> >> >     @          0x1bbcb31  impala::PrintErrorMap()
>>>> >> >     @          0x1bbcd07  impala::PrintErrorMapToString()
>>>> >> >     @          0x2decbd7  impala::Coordinator::GetErrorLog()
>>>> >> >     @          0x1a8d634  impala::ImpalaServer::UnregisterQuery()
>>>> >> >     @          0x1b29264  impala::ImpalaServer::CloseOperation()
>>>> >> >     @          0x2c5ce86
>>>> >> >
>>>> >> >
>>>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseOperation()
>>>> >> >     @          0x2c56b8c
>>>> >> >
>>>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
>>>> >> >     @          0x2c2fcb1
>>>> >> > impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
>>>> >> >     @          0x16fdb20
>>>> >> > apache::thrift::TDispatchProcessor::process()
>>>> >> >     @          0x18ea6b3
>>>> >> > apache::thrift::server::TAcceptQueueServer::Task::run()
>>>> >> >     @          0x18e2181  impala::ThriftThread::RunRunnable()
>>>> >> >     @          0x18e3885  boost::_mfi::mf2<>::operator()()
>>>> >> >     @          0x18e371b  boost::_bi::list3<>::operator()<>()
>>>> >> >     @          0x18e3467  boost::_bi::bind_t<>::operator()()
>>>> >> >     @          0x18e337a
>>>> >> > boost::detail::function::void_function_obj_invoker0<>::invoke()
>>>> >> >     @          0x192761c  boost::function0<>::operator()()
>>>> >> >     @          0x1c3ebf7  impala::Thread::SuperviseThread()
>>>> >> >     @          0x1c470cd  boost::_bi::list5<>::operator()<>()
>>>> >> >     @          0x1c46ff1  boost::_bi::bind_t<>::operator()()
>>>> >> >     @          0x1c46fb4  boost::detail::thread_data<>::run()
>>>> >> >     @          0x2eedb4a  thread_proxy
>>>> >> >     @     0x7fda1dbb16ba  start_thread
>>>> >> >     @     0x7fda1d8e741d  clone
>>>> >> > Wrote minidump to
>>>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>>>> >> >
>>>> >> > impalad.FATAL
>>>> >> >
>>>> >> > Log file created at: 2018/05/07 09:46:12
>>>> >> > Running on machine: slave2
>>>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>>> >> > log_entry.count
>>>> >> >> 0 (-1831809966 vs. 0)
>>>> >> >
>>>> >> > Impalad.INFO
>>>> >> > edentials={real_user=root}} blocked reactor thread for 34288.6us
>>>> >> > I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for
>>>> >> > RPC
>>>> >> > call
>>>> >> > kudu.tserver.TabletServerService.Write ->
>>>> >> > {remote=136.243.74.42:7050
>>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>>>> >> > for
>>>> >> > 35859.8us
>>>> >> > I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for
>>>> >> > RPC
>>>> >> > call
>>>> >> > kudu.tserver.TabletServerService.Write ->
>>>> >> > {remote=136.243.74.42:7050
>>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>>>> >> > for
>>>> >> > 40664.9us
>>>> >> > I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for
>>>> >> > RPC
>>>> >> > call
>>>> >> > kudu.tserver.TabletServerService.Write ->
>>>> >> > {remote=136.243.74.42:7050
>>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>>>> >> > for
>>>> >> > 49514.6us
>>>> >> > I0507 09:46:12.664149  4507 coordinator.cc:783] Release admission
>>>> >> > control
>>>> >> > resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
>>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>>> >> > log_entry.count
>>>> >> >> 0 (-1831809966 vs. 0)
>>>> >> > Wrote minidump to
>>>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>>>> >> >
>>>> >> > Note:
>>>> >> > We are executing the queries on 8 node cluster with the following
>>>> >> > configuration
>>>> >> > Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk
>>>> >> > each,
>>>> >> > Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>>>> >> >
>>>> >> >
>>>> >> > --
>>>> >> > Regards,
>>>> >> > Geetika Gupta
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Regards,
>>>> > Geetika Gupta
>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Geetika Gupta
>>
>>
>>
>>
>> --
>> Regards,
>> Geetika Gupta
>
>

Mime
View raw message