impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mostafa Mokhtar <mmokh...@cloudera.com>
Subject Re: Issue in data loading in Impala + Kudu
Date Thu, 10 May 2018 06:00:55 GMT
Can you try rerunning the query again against the Kudu database instead of
default?

select count(*) from kudu_impala_500.LINEITEM;


On Wed, May 9, 2018 at 10:13 PM, Geetika Gupta <geetika.gupta@knoldus.in>
wrote:

> Please find below the query profile :
>
> Query (id=9649f7ab3bcc5fb8:f4d6a60700000000):
>   Summary:
>     Session ID: b04a9080d1e1724d:41f2b0d261e8f280
>     Session Type: HIVESERVER2
>     HiveServer2 Protocol Version: V6
>     Start Time: 2018-05-08 17:55:23.181264000
>     End Time: 2018-05-10 00:34:17.784273000
>     Query Type: DML
>     Query State: FINISHED
>     Query Status: OK
>     Impala Version: impalad version 3.0.0-SNAPSHOT RELEASE (build b68e06997c1f49f6b723d78e217efddec4f56f3a)
>     User: root
>     Connected User: root
>     Delegated User:
>     Network Address: ::ffff:46.4.88.233:59862
>     Default Db: kudu_impala_500
>     Sql Statement: insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
>     Coordinator: slave2:22000
>     Query Options (set by configuration):
>     Query Options (set by configuration and planner): MT_DOP=0
>     Plan:
> ----------------
> Max Per-Host Resource Reservation: Memory=0B
> Per-Host Resource Estimates: Memory=704.00MB
> WARNING: The following tables are missing relevant table and/or column statistics.
> parquetimpala500.lineitem
>
> F00:PLAN FRAGMENT [RANDOM] hosts=7 instances=7
> |  Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B
> INSERT INTO KUDU [kudu_impala_500.lineitem]
> |  mem-estimate=0B mem-reservation=0B
> |
> 00:SCAN HDFS [parquetimpala500.lineitem, RANDOM]
>    partitions=1/1 files=396 size=97.29GB
>    stored statistics:
>      table: rows=unavailable size=unavailable
>      columns: unavailable
>    extrapolated-rows=disabled
>    mem-estimate=704.00MB mem-reservation=0B
>    tuple-ids=0 row-size=171B cardinality=unavailable
> ----------------
>     Estimated Per-Host Mem: 738197504
>     Tables Missing Stats: parquetimpala500.lineitem
>     Per Host Min Reservation: slave1:22000(0) slave2:22000(0) slave3:22000(0) slave4:22000(0) slave5:22000(0) slave6:22000(0) slave7:22000(0)
>     Request Pool: default-pool
>     Admission result: Admitted immediately
>     ExecSummary:
> Operator       #Hosts  Avg Time  Max Time  #Rows  Est. #Rows  Peak Mem  Est. Peak Mem  Detail
> -----------------------------------------------------------------------------------------------------------------
> 00:SCAN HDFS        7   4s417ms   6s154ms  3.00B          -1   1.39 GB      704.00 MB  parquetimpala500.lineitem
>     Errors: Key already present in Kudu table 'impala::kudu_impala_500.LINEITEM'. (1 of -1831809966 similar)
>
>     Query Compilation: 6s413ms
>        - Metadata load started: 14.443ms (14.443ms)
>        - Metadata load finished. loaded-tables=2/2 load-requests=1 catalog-updates=7: 6s298ms (6s283ms)
>        - Analysis finished: 6s301ms (3.659ms)
>        - Value transfer graph computed: 6s302ms (282.554us)
>        - Single node plan created: 6s363ms (61.624ms)
>        - Runtime filters computed: 6s363ms (98.878us)
>        - Distributed plan created: 6s366ms (2.832ms)
>        - Planning finished: 6s413ms (46.751ms)
>     Query Timeline: 30h38m
>        - Query submitted: 56.829us (56.829us)
>        - Planning finished: 6s431ms (6s431ms)
>        - Submit for admission: 6s432ms (821.867us)
>        - Completed admission: 6s432ms (14.519us)
>        - Ready to start on 7 backends: 6s432ms (99.515us)
>        - All 7 execution backends (7 fragment instances) started: 6s535ms (103.320ms)
>        - Released admission control resources: 30h38m (30h38m)
>        - DML data written: 30h38m (934.739us)
>        - DML Metastore update finished: 30h38m (157.938us)
>        - Request finished: 30h38m (41.379us)
>        - First row fetched: 30h38m (173.124us)
>        - First row fetched: 30h38m (1.750ms)
>        - First row fetched: 30h38m (1.361ms)
>        - Unregister query: 30h38m (1.456ms)
>      - ComputeScanRangeAssignmentTimer: 558.440us
>   ImpalaServer:
>      - ClientFetchWaitTimer: 4.721ms
>      - MetastoreUpdateTimer: 191.424us
>      - RowMaterializationTimer: 0.000ns
>   Execution Profile 9649f7ab3bcc5fb8:f4d6a60700000000:(Total: 30h38m, non-child: 0.000ns, % non-child: 0.00%)
>     Number of filters: 0
>     Filter routing table:
>  ID  Src. Node  Tgt. Node(s)  Target type  Partition filter  Pending (Expected)  First arrived  Completed   Enabled
> -------------------------------------------------------------------------------------------------------------------
>
>     Backend startup latencies: Count: 7, min / max: 1ms / 103ms, 25th %-ile: 1ms, 50th %-ile: 67ms, 75th %-ile: 67ms, 90th %-ile: 80ms, 95th %-ile: 103ms, 99.9th %-ile: 103ms
>     DML Stats:
> Partition: Default
> NumModifiedRows: 536870912
> NumRowErrors: 2463157330
>   BytesWritten: 0
>     Per Node Peak Memory Usage: slave1:22000(1.40 GB) slave2:22000(1.41 GB) slave3:22000(1.39 GB) slave4:22000(1.41 GB) slave5:22000(1.41 GB) slave6:22000(1.40 GB) slave7:22000(1.40 GB)
>      - FiltersReceived: 0 (0)
>      - FinalizationTimer: 0.000ns
>      - NumBackends: 7 (7)
>      - NumFragmentInstances: 7 (7)
>      - NumFragments: 1 (1)
>     Averaged Fragment F00:(Total: 29h51m, non-child: 14s973ms, % non-child: 0.01%)
>       split sizes:  min: 13.63 GB, max: 14.10 GB, avg: 13.90 GB, stddev: 143.22 MB
>       completion times: min:28h47m  max:30h38m  mean: 29h51m  stddev:41m26s
>       execution rates: min:129.50 KB/sec  max:140.31 KB/sec  mean:135.67 KB/sec  stddev:3.50 KB/sec
>       num instances: 7
>        - AverageThreadTokens: 8.89
>        - BloomFilterBytes: 0
>        - PeakMemoryUsage: 1.40 GB (1503884428)
>        - PeakReservation: 0
>        - PeakUsedReservation: 0
>        - PerHostPeakMemUsage: 1.40 GB (1503884428)
>        - RowsProduced: 428.58M (428575463)
>        - TotalNetworkReceiveTime: 0.000ns
>        - TotalNetworkSendTime: 0.000ns
>        - TotalStorageWaitTime: 2m10s
>        - TotalThreadsInvoluntaryContextSwitches: 369.81K (369812)
>        - TotalThreadsTotalWallClockTime: 265h20m
>          - TotalThreadsSysTime: 1m59s
>          - TotalThreadsUserTime: 1h9m
>        - TotalThreadsVoluntaryContextSwitches: 433.92K (433917)
>       Buffer pool:
>          - AllocTime: 0.000ns
>          - CumulativeAllocationBytes: 0
>          - CumulativeAllocations: 0 (0)
>          - PeakReservation: 0
>          - PeakUnpinnedBytes: 0
>          - PeakUsedReservation: 0
>          - ReadIoBytes: 0
>          - ReadIoOps: 0 (0)
>          - ReadIoWaitTime: 0.000ns
>          - ReservationLimit: 0
>          - WriteIoBytes: 0
>          - WriteIoOps: 0 (0)
>          - WriteIoWaitTime: 0.000ns
>       Fragment Instance Lifecycle Timings:
>          - ExecTime: 29h51m
>            - ExecTreeExecTime: 4s724ms
>          - OpenTime: 519.095ms
>            - ExecTreeOpenTime: 36.350us
>          - PrepareTime: 60.445ms
>            - ExecTreePrepareTime: 19.494ms
>       KuduTableSink:(Total: 29h50m, non-child: 29h50m, % non-child: 100.00%)
>          - KuduApplyTimer: 28h54m
>          - NumRowErrors: 351.88M (351879618)
>          - PeakMemoryUsage: 20.00 MB (20971520)
>          - RowsProcessedRate: 3.99 K/sec
>          - TotalNumRows: 428.58M (428575463)
>       HDFS_SCAN_NODE (id=0):(Total: 4s417ms, non-child: 4s417ms, % non-child: 100.00%)
>          - AverageHdfsReadThreadConcurrency: 0.00
>          - AverageScannerThreadConcurrency: 7.89
>          - BytesRead: 13.90 GB (14928723218)
>          - BytesReadDataNodeCache: 0
>          - BytesReadLocal: 13.90 GB (14928723218)
>          - BytesReadRemoteUnexpected: 0
>          - BytesReadShortCircuit: 0
>          - CachedFileHandlesHitCount: 2.31K (2305)
>          - CachedFileHandlesMissCount: 56 (56)
>          - CollectionItemsRead: 0 (0)
>          - DecompressionTime: 22s321ms
>          - MaxCompressedTextFileLength: 0
>          - NumColumns: 16 (16)
>          - NumDictFilteredRowGroups: 0 (0)
>          - NumDisksAccessed: 1 (1)
>          - NumRowGroups: 56 (56)
>          - NumScannerThreadsStarted: 8 (8)
>          - NumScannersWithNoReads: 0 (0)
>          - NumStatsFilteredRowGroups: 0 (0)
>          - PeakMemoryUsage: 1.38 GB (1482717084)
>          - PerReadThreadRawHdfsThroughput: 69.65 MB/sec
>          - RemoteScanRanges: 0 (0)
>          - RowBatchQueueGetWaitTime: 1s754ms
>          - RowBatchQueuePutWaitTime: 235h24m
>          - RowsRead: 428.58M (428575463)
>          - RowsReturned: 428.58M (428575463)
>          - RowsReturnedRate: 99.08 M/sec
>          - ScanRangesComplete: 56 (56)
>          - ScannerThreadsInvoluntaryContextSwitches: 12.33K (12332)
>          - ScannerThreadsTotalWallClockTime: 235h28m
>            - MaterializeTupleTime(*): 2m3s
>            - ScannerThreadsSysTime: 3s896ms
>            - ScannerThreadsUserTime: 2m39s
>          - ScannerThreadsVoluntaryContextSwitches: 419.69K (419690)
>          - TotalRawHdfsOpenFileTime(*): 3s995ms
>          - TotalRawHdfsReadTime(*): 4m2s
>          - TotalReadThroughput: 135.78 KB/sec
>       CodeGen:(Total: 436.431ms, non-child: 436.431ms, % non-child: 100.00%)
>          - CodegenInvoluntaryContextSwitches: 5 (5)
>          - CodegenTotalWallClockTime: 436.422ms
>            - CodegenSysTime: 8.585ms
>            - CodegenUserTime: 41.806ms
>          - CodegenVoluntaryContextSwitches: 76 (76)
>          - CompileTime: 179.278ms
>          - IrGenerationTime: 90.176ms
>          - LoadTime: 0.000ns
>          - ModuleBitcodeSize: 2.26 MB (2373832)
>          - NumFunctions: 19 (19)
>          - NumInstructions: 706 (706)
>          - OptimizationTime: 151.857ms
>          - PeakMemoryUsage: 353.00 KB (361472)
>          - PrepareTime: 14.499ms
>     Fragment F00:
>       Instance 9649f7ab3bcc5fb8:f4d6a60700000000 (host=slave3:22000):(Total: 30h38m, non-child: 11s364ms, % non-child: 0.01%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.63 GB
>         Fragment Instance Lifecycle Event Timeline: 30h38m
>            - Prepare Finished: 86.126ms (86.126ms)
>            - Open Finished: 1s157ms (1s070ms)
>            - First Batch Produced: 3s031ms (1s874ms)
>            - First Batch Sent: 3s037ms (5.724ms)
>            - ExecInternal Finished: 30h38m (30h38m)
>         MemoryUsage(34m8s): 1.28 GB, 1.34 GB, 1.37 GB, 1.35 GB, 1.21 GB, 925.43 MB, 727.39 MB, 1.17 GB, 1.35 GB, 1.35 GB, 1.35 GB, 1.35 GB, 1.26 GB, 1.06 GB, 836.27 MB, 792.34 MB, 1.26 GB, 1.36 GB, 1.36 GB, 1.30 GB, 1.10 GB, 827.96 MB, 1.01 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.12 GB, 875.56 MB, 979.49 MB, 1.34 GB, 1.37 GB, 1.37 GB, 1.28 GB, 1.10 GB, 950.36 MB, 805.45 MB, 963.12 MB, 1.28 GB, 1.34 GB, 1.34 GB, 1.30 GB, 1.04 GB, 808.54 MB, 994.36 MB, 1.17 GB, 1.18 GB, 1.18 GB, 1.18 GB, 1.17 GB, 1.13 GB, 1.04 GB, 954.52 MB, 811.51 MB
>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 8, 8, 8, 8, 8, 8, 8, 8
>          - AverageThreadTokens: 8.80
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 1.39 GB (1487275684)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 1.39 GB (1487275684)
>          - RowsProduced: 424.29M (424287956)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 0.000ns
>          - TotalStorageWaitTime: 1m32s
>          - TotalThreadsInvoluntaryContextSwitches: 349.35K (349347)
>          - TotalThreadsTotalWallClockTime: 269h37m
>            - TotalThreadsSysTime: 1m54s
>            - TotalThreadsUserTime: 1h6m
>          - TotalThreadsVoluntaryContextSwitches: 438.17K (438165)
>         Buffer pool:
>            - AllocTime: 0.000ns
>            - CumulativeAllocationBytes: 0
>            - CumulativeAllocations: 0 (0)
>            - PeakReservation: 0
>            - PeakUnpinnedBytes: 0
>            - PeakUsedReservation: 0
>            - ReadIoBytes: 0
>            - ReadIoOps: 0 (0)
>            - ReadIoWaitTime: 0.000ns
>            - ReservationLimit: 0
>            - WriteIoBytes: 0
>            - WriteIoOps: 0 (0)
>            - WriteIoWaitTime: 0.000ns
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 30h38m
>              - ExecTreeExecTime: 4s586ms
>            - OpenTime: 1s070ms
>              - ExecTreeOpenTime: 32.390us
>            - PrepareTime: 85.903ms
>              - ExecTreePrepareTime: 16.283ms
>         KuduTableSink:(Total: 30h38m, non-child: 30h38m, % non-child: 100.00%)
>            - KuduApplyTimer: 29h44m
>            - NumRowErrors: 371.48M (371483541)
>            - PeakMemoryUsage: 20.00 MB (20971520)
>            - RowsProcessedRate: 3.85 K/sec
>            - TotalNumRows: 424.29M (424287956)
>         HDFS_SCAN_NODE (id=0):(Total: 4s304ms, non-child: 4s304ms, % non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.63 GB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
>           Hdfs Read Thread Concurrency Bucket: 0:99.87% 1:0.1279% 2:0% 3:0% 4:0% 5:0% 6:0%
>           File Formats: PARQUET/SNAPPY:56
>           BytesRead(34m8s): 1.19 GB, 1.46 GB, 1.65 GB, 1.78 GB, 1.88 GB, 1.97 GB, 2.01 GB, 2.91 GB, 3.27 GB, 3.46 GB, 3.61 GB, 3.72 GB, 3.80 GB, 3.90 GB, 3.96 GB, 4.16 GB, 5.05 GB, 5.36 GB, 5.60 GB, 5.76 GB, 5.87 GB, 5.94 GB, 6.57 GB, 7.32 GB, 7.57 GB, 7.75 GB, 7.86 GB, 7.92 GB, 8.43 GB, 9.18 GB, 9.43 GB, 9.62 GB, 9.75 GB, 9.84 GB, 9.89 GB, 9.94 GB, 10.41 GB, 11.07 GB, 11.24 GB, 11.42 GB, 11.66 GB, 11.82 GB, 12.05 GB, 12.48 GB, 12.96 GB, 13.07 GB, 13.19 GB, 13.31 GB, 13.38 GB, 13.46 GB, 13.51 GB, 13.57 GB, 13.61 GB
>            - FooterProcessingTime: (Avg: 381.833ms ; Min: 46.831ms ; Max: 4s214ms ; Number of samples: 56)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 7.80
>            - BytesRead: 13.63 GB (14635606518)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 13.63 GB (14635606518)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 0
>            - CachedFileHandlesHitCount: 2.27K (2268)
>            - CachedFileHandlesMissCount: 56 (56)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 19s612ms
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 16 (16)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 1 (1)
>            - NumRowGroups: 56 (56)
>            - NumScannerThreadsStarted: 8 (8)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 1.37 GB (1466047697)
>            - PerReadThreadRawHdfsThroughput: 100.45 MB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 1s872ms
>            - RowBatchQueuePutWaitTime: 238h55m
>            - RowsRead: 424.29M (424287956)
>            - RowsReturned: 424.29M (424287956)
>            - RowsReturnedRate: 98.57 M/sec
>            - ScanRangesComplete: 56 (56)
>            - ScannerThreadsInvoluntaryContextSwitches: 12.67K (12674)
>            - ScannerThreadsTotalWallClockTime: 238h58m
>              - MaterializeTupleTime(*): 1m57s
>              - ScannerThreadsSysTime: 3s421ms
>              - ScannerThreadsUserTime: 2m31s
>            - ScannerThreadsVoluntaryContextSwitches: 415.49K (415490)
>            - TotalRawHdfsOpenFileTime(*): 2s904ms
>            - TotalRawHdfsReadTime(*): 2m18s
>            - TotalReadThroughput: 129.60 KB/sec
>         CodeGen:(Total: 915.511ms, non-child: 915.511ms, % non-child: 100.00%)
>            - CodegenInvoluntaryContextSwitches: 10 (10)
>            - CodegenTotalWallClockTime: 915.504ms
>              - CodegenSysTime: 8.256ms
>              - CodegenUserTime: 45.032ms
>            - CodegenVoluntaryContextSwitches: 87 (87)
>            - CompileTime: 319.788ms
>            - IrGenerationTime: 276.446ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 2.26 MB (2373832)
>            - NumFunctions: 19 (19)
>            - NumInstructions: 706 (706)
>            - OptimizationTime: 301.622ms
>            - PeakMemoryUsage: 353.00 KB (361472)
>            - PrepareTime: 16.858ms
>       Instance 9649f7ab3bcc5fb8:f4d6a60700000002 (host=slave1:22000):(Total: 30h17m, non-child: 1s508ms, % non-child: 0.00%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:57/14.10 GB
>         Fragment Instance Lifecycle Event Timeline: 30h17m
>            - Prepare Finished: 32.853ms (32.853ms)
>            - Open Finished: 362.691ms (329.838ms)
>            - First Batch Produced: 1s725ms (1s362ms)
>            - First Batch Sent: 1s728ms (3.415ms)
>            - ExecInternal Finished: 30h17m (30h17m)
>         MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.27 GB, 1.07 GB, 798.75 MB, 1.36 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.19 GB, 974.24 MB, 800.43 MB, 1.27 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.26 GB, 1010.99 MB, 805.20 MB, 1.29 GB, 1.38 GB, 1.38 GB, 1.27 GB, 987.74 MB, 811.33 MB, 1.27 GB, 1.38 GB, 1.38 GB, 1.34 GB, 1.18 GB, 922.96 MB, 765.77 MB, 1.13 GB, 1.36 GB, 1.36 GB, 1.36 GB, 1.37 GB, 1.30 GB, 1.03 GB, 919.88 MB, 1.38 GB, 1.38 GB, 1.38 GB, 1.39 GB, 1.35 GB, 1.24 GB, 1.17 GB, 1005.37 MB, 844.55 MB, 751.70 MB, 518.60 MB
>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 5
>          - AverageThreadTokens: 8.91
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 1.40 GB (1500404835)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 1.40 GB (1500404835)
>          - RowsProduced: 434.96M (434955041)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 0.000ns
>          - TotalStorageWaitTime: 2m56s
>          - TotalThreadsInvoluntaryContextSwitches: 370.59K (370587)
>          - TotalThreadsTotalWallClockTime: 269h58m
>            - TotalThreadsSysTime: 1m58s
>            - TotalThreadsUserTime: 1h9m
>          - TotalThreadsVoluntaryContextSwitches: 437.63K (437630)
>         Buffer pool:
>            - AllocTime: 0.000ns
>            - CumulativeAllocationBytes: 0
>            - CumulativeAllocations: 0 (0)
>            - PeakReservation: 0
>            - PeakUnpinnedBytes: 0
>            - PeakUsedReservation: 0
>            - ReadIoBytes: 0
>            - ReadIoOps: 0 (0)
>            - ReadIoWaitTime: 0.000ns
>            - ReservationLimit: 0
>            - WriteIoBytes: 0
>            - WriteIoOps: 0 (0)
>            - WriteIoWaitTime: 0.000ns
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 30h17m
>              - ExecTreeExecTime: 4s575ms
>            - OpenTime: 329.834ms
>              - ExecTreeOpenTime: 45.986us
>            - PrepareTime: 32.581ms
>              - ExecTreePrepareTime: 18.647ms
>         KuduTableSink:(Total: 30h17m, non-child: 30h17m, % non-child: 100.00%)
>            - KuduApplyTimer: 29h22m
>            - NumRowErrors: 357.86M (357862755)
>            - PeakMemoryUsage: 20.00 MB (20971520)
>            - RowsProcessedRate: 3.99 K/sec
>            - TotalNumRows: 434.96M (434955041)
>         HDFS_SCAN_NODE (id=0):(Total: 4s229ms, non-child: 4s229ms, % non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:57/14.10 GB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of 57
>           Hdfs Read Thread Concurrency Bucket: 0:99.63% 1:0.3706% 2:0% 3:0% 4:0% 5:0% 6:0%
>           File Formats: PARQUET/SNAPPY:57
>           BytesRead(34m8s): 1.32 GB, 1.59 GB, 1.74 GB, 1.85 GB, 1.94 GB, 2.11 GB, 3.21 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB, 3.94 GB, 4.11 GB, 5.04 GB, 5.32 GB, 5.50 GB, 5.67 GB, 5.81 GB, 5.92 GB, 6.09 GB, 7.06 GB, 7.40 GB, 7.63 GB, 7.79 GB, 7.90 GB, 8.08 GB, 9.02 GB, 9.35 GB, 9.55 GB, 9.70 GB, 9.81 GB, 9.90 GB, 10.01 GB, 10.74 GB, 11.19 GB, 11.34 GB, 11.48 GB, 11.57 GB, 11.69 GB, 11.84 GB, 12.25 GB, 13.25 GB, 13.41 GB, 13.50 GB, 13.60 GB, 13.67 GB, 13.73 GB, 13.79 GB, 13.83 GB, 13.86 GB, 13.92 GB, 14.03 GB
>            - FooterProcessingTime: (Avg: 414.198ms ; Min: 51.291ms ; Max: 2s894ms ; Number of samples: 57)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 7.91
>            - BytesRead: 14.11 GB (15148002597)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 14.11 GB (15148002597)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 0
>            - CachedFileHandlesHitCount: 2.33K (2333)
>            - CachedFileHandlesMissCount: 57 (57)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 23s330ms
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 16 (16)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 1 (1)
>            - NumRowGroups: 57 (57)
>            - NumScannerThreadsStarted: 8 (8)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 1.38 GB (1479367012)
>            - PerReadThreadRawHdfsThroughput: 36.38 MB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 1s354ms
>            - RowBatchQueuePutWaitTime: 239h35m
>            - RowsRead: 434.96M (434955041)
>            - RowsReturned: 434.96M (434955041)
>            - RowsReturnedRate: 102.83 M/sec
>            - ScanRangesComplete: 57 (57)
>            - ScannerThreadsInvoluntaryContextSwitches: 7.23K (7231)
>            - ScannerThreadsTotalWallClockTime: 239h40m
>              - MaterializeTupleTime(*): 2m1s
>              - ScannerThreadsSysTime: 4s044ms
>              - ScannerThreadsUserTime: 2m42s
>            - ScannerThreadsVoluntaryContextSwitches: 425.94K (425935)
>            - TotalRawHdfsOpenFileTime(*): 5s555ms
>            - TotalRawHdfsReadTime(*): 6m37s
>            - TotalReadThroughput: 135.68 KB/sec
>         CodeGen:(Total: 274.452ms, non-child: 274.452ms, % non-child: 100.00%)
>            - CodegenInvoluntaryContextSwitches: 4 (4)
>            - CodegenTotalWallClockTime: 274.443ms
>              - CodegenSysTime: 7.440ms
>              - CodegenUserTime: 40.295ms
>            - CodegenVoluntaryContextSwitches: 86 (86)
>            - CompileTime: 103.404ms
>            - IrGenerationTime: 32.648ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 2.26 MB (2373832)
>            - NumFunctions: 19 (19)
>            - NumInstructions: 706 (706)
>            - OptimizationTime: 122.033ms
>            - PeakMemoryUsage: 353.00 KB (361472)
>            - PrepareTime: 15.943ms
>       Instance 9649f7ab3bcc5fb8:f4d6a60700000001 (host=slave4:22000):(Total: 30h16m, non-child: 2s019ms, % non-child: 0.00%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
>         Fragment Instance Lifecycle Event Timeline: 30h16m
>            - Prepare Finished: 105.379ms (105.379ms)
>            - Open Finished: 519.128ms (413.748ms)
>            - First Batch Produced: 1s789ms (1s270ms)
>            - First Batch Sent: 1s795ms (6.364ms)
>            - ExecInternal Finished: 30h16m (30h16m)
>         MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.23 GB, 924.30 MB, 715.63 MB, 1.26 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.30 GB, 1.08 GB, 843.90 MB, 887.06 MB, 1.40 GB, 1.41 GB, 1.39 GB, 1.19 GB, 826.21 MB, 1.19 GB, 1.38 GB, 1.38 GB, 1.31 GB, 1.05 GB, 748.49 MB, 1.17 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.24 GB, 999.76 MB, 733.27 MB, 995.28 MB, 1.35 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.35 GB, 1.10 GB, 861.01 MB, 1.39 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.39 GB, 1.33 GB, 1.24 GB, 1.13 GB, 1007.59 MB, 820.51 MB, 655.33 MB
>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
>          - AverageThreadTokens: 8.97
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 1.41 GB (1513166691)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 1.41 GB (1513166691)
>          - RowsProduced: 425.55M (425550934)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 0.000ns
>          - TotalStorageWaitTime: 1m49s
>          - TotalThreadsInvoluntaryContextSwitches: 364.52K (364523)
>          - TotalThreadsTotalWallClockTime: 271h36m
>            - TotalThreadsSysTime: 1m56s
>            - TotalThreadsUserTime: 1h9m
>          - TotalThreadsVoluntaryContextSwitches: 431.64K (431643)
>         Buffer pool:
>            - AllocTime: 0.000ns
>            - CumulativeAllocationBytes: 0
>            - CumulativeAllocations: 0 (0)
>            - PeakReservation: 0
>            - PeakUnpinnedBytes: 0
>            - PeakUsedReservation: 0
>            - ReadIoBytes: 0
>            - ReadIoOps: 0 (0)
>            - ReadIoWaitTime: 0.000ns
>            - ReservationLimit: 0
>            - WriteIoBytes: 0
>            - WriteIoOps: 0 (0)
>            - WriteIoWaitTime: 0.000ns
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 30h16m
>              - ExecTreeExecTime: 4s185ms
>            - OpenTime: 413.744ms
>              - ExecTreeOpenTime: 31.615us
>            - PrepareTime: 105.138ms
>              - ExecTreePrepareTime: 17.849ms
>         KuduTableSink:(Total: 30h16m, non-child: 30h16m, % non-child: 100.00%)
>            - KuduApplyTimer: 29h20m
>            - NumRowErrors: 336.92M (336918666)
>            - PeakMemoryUsage: 20.00 MB (20971520)
>            - RowsProcessedRate: 3.90 K/sec
>            - TotalNumRows: 425.55M (425550934)
>         HDFS_SCAN_NODE (id=0):(Total: 3s886ms, non-child: 3s886ms, % non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
>           Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.1368% 2:0% 3:0% 4:0% 5:0% 6:0%
>           File Formats: PARQUET/SNAPPY:56
>           BytesRead(34m8s): 1.29 GB, 1.51 GB, 1.66 GB, 1.79 GB, 1.88 GB, 1.97 GB, 2.00 GB, 3.01 GB, 3.30 GB, 3.49 GB, 3.63 GB, 3.74 GB, 3.82 GB, 3.92 GB, 3.96 GB, 4.29 GB, 5.26 GB, 5.47 GB, 5.71 GB, 5.87 GB, 5.94 GB, 6.88 GB, 7.34 GB, 7.59 GB, 7.76 GB, 7.89 GB, 7.93 GB, 8.83 GB, 9.30 GB, 9.51 GB, 9.67 GB, 9.79 GB, 9.89 GB, 9.91 GB, 10.45 GB, 11.14 GB, 11.30 GB, 11.46 GB, 11.55 GB, 11.68 GB, 11.83 GB, 12.10 GB, 13.21 GB, 13.35 GB, 13.47 GB, 13.56 GB, 13.64 GB, 13.71 GB, 13.76 GB, 13.81 GB, 13.86 GB, 13.87 GB, 13.88 GB
>            - FooterProcessingTime: (Avg: 331.439ms ; Min: 46.760ms ; Max: 2s508ms ; Number of samples: 56)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 7.97
>            - BytesRead: 13.88 GB (14898393824)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 13.88 GB (14898393824)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 0
>            - CachedFileHandlesHitCount: 2.30K (2296)
>            - CachedFileHandlesMissCount: 56 (56)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 22s686ms
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 16 (16)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 1 (1)
>            - NumRowGroups: 56 (56)
>            - NumScannerThreadsStarted: 8 (8)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 1.39 GB (1491938707)
>            - PerReadThreadRawHdfsThroughput: 95.20 MB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 1s255ms
>            - RowBatchQueuePutWaitTime: 241h15m
>            - RowsRead: 425.55M (425550934)
>            - RowsReturned: 425.55M (425550934)
>            - RowsReturnedRate: 109.50 M/sec
>            - ScanRangesComplete: 56 (56)
>            - ScannerThreadsInvoluntaryContextSwitches: 12.02K (12019)
>            - ScannerThreadsTotalWallClockTime: 241h20m
>              - MaterializeTupleTime(*): 2m6s
>              - ScannerThreadsSysTime: 3s548ms
>              - ScannerThreadsUserTime: 2m39s
>            - ScannerThreadsVoluntaryContextSwitches: 416.75K (416750)
>            - TotalRawHdfsOpenFileTime(*): 3s230ms
>            - TotalRawHdfsReadTime(*): 2m29s
>            - TotalReadThroughput: 133.55 KB/sec
>         CodeGen:(Total: 315.226ms, non-child: 315.226ms, % non-child: 100.00%)
>            - CodegenInvoluntaryContextSwitches: 9 (9)
>            - CodegenTotalWallClockTime: 315.217ms
>              - CodegenSysTime: 5.142ms
>              - CodegenUserTime: 43.310ms
>            - CodegenVoluntaryContextSwitches: 91 (91)
>            - CompileTime: 129.393ms
>            - IrGenerationTime: 60.406ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 2.26 MB (2373832)
>            - NumFunctions: 19 (19)
>            - NumInstructions: 706 (706)
>            - OptimizationTime: 111.534ms
>            - PeakMemoryUsage: 353.00 KB (361472)
>            - PrepareTime: 13.425ms
>       Instance 9649f7ab3bcc5fb8:f4d6a60700000004 (host=slave7:22000):(Total: 30h10m, non-child: 16s427ms, % non-child: 0.02%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:58/14.03 GB
>         Fragment Instance Lifecycle Event Timeline: 30h10m
>            - Prepare Finished: 105.361ms (105.361ms)
>            - Open Finished: 980.610ms (875.249ms)
>            - First Batch Produced: 4s553ms (3s573ms)
>            - First Batch Sent: 4s559ms (5.668ms)
>            - ExecInternal Finished: 30h10m (30h10m)
>         MemoryUsage(34m8s): 1.38 GB, 1.40 GB, 1.40 GB, 1.28 GB, 1.08 GB, 796.55 MB, 1.34 GB, 1.37 GB, 1.35 GB, 1.34 GB, 1.27 GB, 1.21 GB, 983.86 MB, 1.08 GB, 1.38 GB, 1.36 GB, 1.35 GB, 1.28 GB, 1.17 GB, 921.06 MB, 1.02 GB, 1.35 GB, 1.33 GB, 1.28 GB, 1.15 GB, 905.74 MB, 1.01 GB, 1.36 GB, 1.34 GB, 1.32 GB, 1.21 GB, 1.10 GB, 877.34 MB, 1.04 GB, 1.32 GB, 1.38 GB, 1.36 GB, 1.35 GB, 1.32 GB, 1.21 GB, 1000.76 MB, 1.11 GB, 1.37 GB, 1.36 GB, 1.33 GB, 1.29 GB, 1.20 GB, 1.09 GB, 1.07 GB, 937.32 MB, 814.51 MB, 480.87 MB, 155.08 MB
>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 5, 2
>          - AverageThreadTokens: 8.80
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 1.40 GB (1500511831)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 1.40 GB (1500511831)
>          - RowsProduced: 433.04M (433043954)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 0.000ns
>          - TotalStorageWaitTime: 3m18s
>          - TotalThreadsInvoluntaryContextSwitches: 374.99K (374993)
>          - TotalThreadsTotalWallClockTime: 265h25m
>            - TotalThreadsSysTime: 1m57s
>            - TotalThreadsUserTime: 1h10m
>          - TotalThreadsVoluntaryContextSwitches: 435.67K (435665)
>         Buffer pool:
>            - AllocTime: 0.000ns
>            - CumulativeAllocationBytes: 0
>            - CumulativeAllocations: 0 (0)
>            - PeakReservation: 0
>            - PeakUnpinnedBytes: 0
>            - PeakUsedReservation: 0
>            - ReadIoBytes: 0
>            - ReadIoOps: 0 (0)
>            - ReadIoWaitTime: 0.000ns
>            - ReservationLimit: 0
>            - WriteIoBytes: 0
>            - WriteIoOps: 0 (0)
>            - WriteIoWaitTime: 0.000ns
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 30h10m
>              - ExecTreeExecTime: 6s426ms
>            - OpenTime: 875.240ms
>              - ExecTreeOpenTime: 41.994us
>            - PrepareTime: 105.102ms
>              - ExecTreePrepareTime: 24.800ms
>         KuduTableSink:(Total: 30h9m, non-child: 30h9m, % non-child: 100.00%)
>            - KuduApplyTimer: 29h12m
>            - NumRowErrors: 357.43M (357426783)
>            - PeakMemoryUsage: 20.00 MB (20971520)
>            - RowsProcessedRate: 3.99 K/sec
>            - TotalNumRows: 433.04M (433043954)
>         HDFS_SCAN_NODE (id=0):(Total: 6s154ms, non-child: 6s154ms, % non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:58/14.03 GB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 58 out of 58
>           Hdfs Read Thread Concurrency Bucket: 0:99.68% 1:0.3169% 2:0% 3:0% 4:0% 5:0% 6:0%
>           File Formats: PARQUET/SNAPPY:58
>           BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB, 2.10 GB, 3.18 GB, 3.29 GB, 3.44 GB, 3.62 GB, 3.72 GB, 3.95 GB, 4.03 GB, 4.61 GB, 5.22 GB, 5.37 GB, 5.52 GB, 5.66 GB, 5.90 GB, 6.01 GB, 6.53 GB, 7.22 GB, 7.43 GB, 7.62 GB, 7.83 GB, 7.99 GB, 8.49 GB, 9.18 GB, 9.38 GB, 9.55 GB, 9.67 GB, 9.88 GB, 9.98 GB, 10.53 GB, 11.04 GB, 11.22 GB, 11.39 GB, 11.50 GB, 11.58 GB, 11.70 GB, 11.95 GB, 12.61 GB, 13.30 GB, 13.43 GB, 13.52 GB, 13.59 GB, 13.65 GB, 13.72 GB, 13.91 GB, 13.94 GB, 13.96 GB, 13.98 GB, 14.03 GB
>            - FooterProcessingTime: (Avg: 615.194ms ; Min: 50.813ms ; Max: 6s740ms ; Number of samples: 58)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 7.80
>            - BytesRead: 14.04 GB (15072987184)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 14.04 GB (15072987184)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 0
>            - CachedFileHandlesHitCount: 2.34K (2340)
>            - CachedFileHandlesMissCount: 58 (58)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 21s821ms
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 16 (16)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 1 (1)
>            - NumRowGroups: 58 (58)
>            - NumScannerThreadsStarted: 8 (8)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 1.38 GB (1479283937)
>            - PerReadThreadRawHdfsThroughput: 41.17 MB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 3s555ms
>            - RowBatchQueuePutWaitTime: 235h9m
>            - RowsRead: 433.04M (433043954)
>            - RowsReturned: 433.04M (433043954)
>            - RowsReturnedRate: 70.36 M/sec
>            - ScanRangesComplete: 58 (58)
>            - ScannerThreadsInvoluntaryContextSwitches: 16.45K (16447)
>            - ScannerThreadsTotalWallClockTime: 235h15m
>              - MaterializeTupleTime(*): 2m5s
>              - ScannerThreadsSysTime: 3s924ms
>              - ScannerThreadsUserTime: 2m40s
>            - ScannerThreadsVoluntaryContextSwitches: 424.08K (424080)
>            - TotalRawHdfsOpenFileTime(*): 4s852ms
>            - TotalRawHdfsReadTime(*): 5m49s
>            - TotalReadThroughput: 135.58 KB/sec
>         CodeGen:(Total: 771.273ms, non-child: 771.273ms, % non-child: 100.00%)
>            - CodegenInvoluntaryContextSwitches: 5 (5)
>            - CodegenTotalWallClockTime: 771.263ms
>              - CodegenSysTime: 19.916ms
>              - CodegenUserTime: 41.600ms
>            - CodegenVoluntaryContextSwitches: 91 (91)
>            - CompileTime: 404.632ms
>            - IrGenerationTime: 153.195ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 2.26 MB (2373832)
>            - NumFunctions: 19 (19)
>            - NumInstructions: 706 (706)
>            - OptimizationTime: 199.856ms
>            - PeakMemoryUsage: 353.00 KB (361472)
>            - PrepareTime: 12.885ms
>       Instance 9649f7ab3bcc5fb8:f4d6a60700000006 (host=slave6:22000):(Total: 29h59m, non-child: 21s225ms, % non-child: 0.02%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:57/13.92 GB
>         Fragment Instance Lifecycle Event Timeline: 29h59m
>            - Prepare Finished: 55.600ms (55.600ms)
>            - Open Finished: 598.215ms (542.614ms)
>            - First Batch Produced: 2s350ms (1s752ms)
>            - First Batch Sent: 2s356ms (5.853ms)
>            - ExecInternal Finished: 29h59m (29h59m)
>         MemoryUsage(34m8s): 1.37 GB, 1.37 GB, 1.36 GB, 1.24 GB, 1.04 GB, 864.64 MB, 1.37 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.31 GB, 1.17 GB, 965.56 MB, 876.58 MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.38 GB, 1.23 GB, 995.12 MB, 871.65 MB, 1.35 GB, 1.40 GB, 1.38 GB, 1.24 GB, 977.74 MB, 850.26 MB, 1.27 GB, 1.29 GB, 1.30 GB, 1.31 GB, 1.16 GB, 954.46 MB, 906.32 MB, 1.24 GB, 1.34 GB, 1.30 GB, 1.31 GB, 1.36 GB, 1.29 GB, 1.02 GB, 1.02 GB, 1.32 GB, 1.30 GB, 1.31 GB, 1.38 GB, 1.33 GB, 1.23 GB, 1.14 GB, 1017.39 MB, 822.59 MB, 631.09 MB
>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 7
>          - AverageThreadTokens: 8.86
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 1.40 GB (1499916960)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 1.40 GB (1499916960)
>          - RowsProduced: 428.25M (428247756)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 0.000ns
>          - TotalStorageWaitTime: 1m52s
>          - TotalThreadsInvoluntaryContextSwitches: 388.30K (388300)
>          - TotalThreadsTotalWallClockTime: 265h50m
>            - TotalThreadsSysTime: 1m58s
>            - TotalThreadsUserTime: 1h13m
>          - TotalThreadsVoluntaryContextSwitches: 431.32K (431323)
>         Buffer pool:
>            - AllocTime: 0.000ns
>            - CumulativeAllocationBytes: 0
>            - CumulativeAllocations: 0 (0)
>            - PeakReservation: 0
>            - PeakUnpinnedBytes: 0
>            - PeakUsedReservation: 0
>            - ReadIoBytes: 0
>            - ReadIoOps: 0 (0)
>            - ReadIoWaitTime: 0.000ns
>            - ReservationLimit: 0
>            - WriteIoBytes: 0
>            - WriteIoOps: 0 (0)
>            - WriteIoWaitTime: 0.000ns
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 29h59m
>              - ExecTreeExecTime: 4s703ms
>            - OpenTime: 542.606ms
>              - ExecTreeOpenTime: 32.022us
>            - PrepareTime: 55.377ms
>              - ExecTreePrepareTime: 20.617ms
>         KuduTableSink:(Total: 29h59m, non-child: 29h59m, % non-child: 100.00%)
>            - KuduApplyTimer: 29h
>            - NumRowErrors: 346.33M (346333617)
>            - PeakMemoryUsage: 20.00 MB (20971520)
>            - RowsProcessedRate: 3.97 K/sec
>            - TotalNumRows: 428.25M (428247756)
>         HDFS_SCAN_NODE (id=0):(Total: 4s407ms, non-child: 4s407ms, % non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:57/13.92 GB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of 57
>           Hdfs Read Thread Concurrency Bucket: 0:99.81% 1:0.1895% 2:0% 3:0% 4:0% 5:0% 6:0%
>           File Formats: PARQUET/SNAPPY:57
>           BytesRead(34m8s): 1.29 GB, 1.56 GB, 1.71 GB, 1.81 GB, 1.90 GB, 2.18 GB, 3.19 GB, 3.30 GB, 3.46 GB, 3.65 GB, 3.76 GB, 3.84 GB, 3.91 GB, 4.19 GB, 5.09 GB, 5.31 GB, 5.48 GB, 5.64 GB, 5.78 GB, 5.90 GB, 6.16 GB, 7.13 GB, 7.39 GB, 7.61 GB, 7.76 GB, 7.91 GB, 8.15 GB, 9.04 GB, 9.25 GB, 9.49 GB, 9.72 GB, 9.83 GB, 9.95 GB, 10.16 GB, 10.86 GB, 11.12 GB, 11.26 GB, 11.41 GB, 11.60 GB, 11.71 GB, 11.87 GB, 12.39 GB, 13.18 GB, 13.32 GB, 13.42 GB, 13.63 GB, 13.70 GB, 13.76 GB, 13.82 GB, 13.87 GB, 13.89 GB, 13.90 GB
>            - FooterProcessingTime: (Avg: 368.827ms ; Min: 61.227ms ; Max: 3s788ms ; Number of samples: 57)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 7.86
>            - BytesRead: 13.92 GB (14949018844)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 13.92 GB (14949018844)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 0
>            - CachedFileHandlesHitCount: 2.31K (2315)
>            - CachedFileHandlesMissCount: 57 (57)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 22s734ms
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 16 (16)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 1 (1)
>            - NumRowGroups: 57 (57)
>            - NumScannerThreadsStarted: 8 (8)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 1.38 GB (1478857724)
>            - PerReadThreadRawHdfsThroughput: 69.53 MB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 1s749ms
>            - RowBatchQueuePutWaitTime: 235h46m
>            - RowsRead: 428.25M (428247756)
>            - RowsReturned: 428.25M (428247756)
>            - RowsReturnedRate: 97.15 M/sec
>            - ScanRangesComplete: 57 (57)
>            - ScannerThreadsInvoluntaryContextSwitches: 15.99K (15992)
>            - ScannerThreadsTotalWallClockTime: 235h50m
>              - MaterializeTupleTime(*): 2m7s
>              - ScannerThreadsSysTime: 3s685ms
>              - ScannerThreadsUserTime: 2m42s
>            - ScannerThreadsVoluntaryContextSwitches: 419.35K (419351)
>            - TotalRawHdfsOpenFileTime(*): 3s380ms
>            - TotalRawHdfsReadTime(*): 3m25s
>            - TotalReadThroughput: 135.25 KB/sec
>         CodeGen:(Total: 405.466ms, non-child: 405.466ms, % non-child: 100.00%)
>            - CodegenInvoluntaryContextSwitches: 2 (2)
>            - CodegenTotalWallClockTime: 405.458ms
>              - CodegenSysTime: 0.000ns
>              - CodegenUserTime: 49.456ms
>            - CodegenVoluntaryContextSwitches: 91 (91)
>            - CompileTime: 170.012ms
>            - IrGenerationTime: 52.154ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 2.26 MB (2373832)
>            - NumFunctions: 19 (19)
>            - NumInstructions: 706 (706)
>            - OptimizationTime: 168.310ms
>            - PeakMemoryUsage: 353.00 KB (361472)
>            - PrepareTime: 14.493ms
>       Instance 9649f7ab3bcc5fb8:f4d6a60700000003 (host=slave2:22000):(Total: 28h48m, non-child: 21s449ms, % non-child: 0.02%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
>         Fragment Instance Lifecycle Event Timeline: 28h48m
>            - Prepare Finished: 13.348ms (13.348ms)
>            - Open Finished: 348.443ms (335.094ms)
>            - First Batch Produced: 1s505ms (1s157ms)
>            - First Batch Sent: 1s508ms (3.334ms)
>            - ExecInternal Finished: 28h48m (28h48m)
>         MemoryUsage(34m8s): 1.38 GB, 1.38 GB, 1.39 GB, 1.26 GB, 1.06 GB, 777.91 MB, 1.37 GB, 1.38 GB, 1.38 GB, 1.39 GB, 1.33 GB, 1.20 GB, 985.85 MB, 781.77 MB, 1.25 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.25 GB, 1015.91 MB, 756.94 MB, 1.27 GB, 1.37 GB, 1.37 GB, 1.27 GB, 1012.70 MB, 775.19 MB, 1.25 GB, 1.41 GB, 1.41 GB, 1.30 GB, 1.01 GB, 871.44 MB, 1.34 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.37 GB, 1.31 GB, 1.14 GB, 825.57 MB, 1.32 GB, 1.40 GB, 1.40 GB, 1.37 GB, 1.28 GB, 1.17 GB, 1.02 GB, 856.05 MB, 694.00 MB
>         ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
>          - AverageThreadTokens: 8.94
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 1.41 GB (1512855058)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 1.41 GB (1512855058)
>          - RowsProduced: 427.11M (427110203)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 0.000ns
>          - TotalStorageWaitTime: 2m27s
>          - TotalThreadsInvoluntaryContextSwitches: 375.92K (375918)
>          - TotalThreadsTotalWallClockTime: 257h33m
>            - TotalThreadsSysTime: 2m12s
>            - TotalThreadsUserTime: 1h9m
>          - TotalThreadsVoluntaryContextSwitches: 431.39K (431386)
>         Buffer pool:
>            - AllocTime: 0.000ns
>            - CumulativeAllocationBytes: 0
>            - CumulativeAllocations: 0 (0)
>            - PeakReservation: 0
>            - PeakUnpinnedBytes: 0
>            - PeakUsedReservation: 0
>            - ReadIoBytes: 0
>            - ReadIoOps: 0 (0)
>            - ReadIoWaitTime: 0.000ns
>            - ReservationLimit: 0
>            - WriteIoBytes: 0
>            - WriteIoOps: 0 (0)
>            - WriteIoWaitTime: 0.000ns
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 28h48m
>              - ExecTreeExecTime: 4s366ms
>            - OpenTime: 335.089ms
>              - ExecTreeOpenTime: 40.091us
>            - PrepareTime: 13.257ms
>              - ExecTreePrepareTime: 12.776ms
>         KuduTableSink:(Total: 28h48m, non-child: 28h48m, % non-child: 100.00%)
>            - KuduApplyTimer: 27h51m
>            - NumRowErrors: 347.46M (347457539)
>            - PeakMemoryUsage: 20.00 MB (20971520)
>            - RowsProcessedRate: 4.12 K/sec
>            - TotalNumRows: 427.11M (427110203)
>         HDFS_SCAN_NODE (id=0):(Total: 4s001ms, non-child: 4s001ms, % non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
>           Hdfs Read Thread Concurrency Bucket: 0:99.69% 1:0.3106% 2:0% 3:0% 4:0% 5:0% 6:0%
>           File Formats: PARQUET/SNAPPY:56
>           BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB, 2.08 GB, 3.22 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB, 3.94 GB, 4.08 GB, 5.02 GB, 5.30 GB, 5.48 GB, 5.66 GB, 5.80 GB, 5.91 GB, 6.00 GB, 7.04 GB, 7.37 GB, 7.61 GB, 7.77 GB, 7.90 GB, 8.01 GB, 8.95 GB, 9.39 GB, 9.61 GB, 9.77 GB, 9.88 GB, 10.17 GB, 11.12 GB, 11.26 GB, 11.40 GB, 11.56 GB, 11.66 GB, 11.73 GB, 11.82 GB, 11.97 GB, 13.08 GB, 13.48 GB, 13.60 GB, 13.68 GB, 13.74 GB, 13.80 GB, 13.85 GB, 13.87 GB, 13.88 GB
>            - FooterProcessingTime: (Avg: 339.758ms ; Min: 55.620ms ; Max: 2s575ms ; Number of samples: 56)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 7.94
>            - BytesRead: 13.88 GB (14898474484)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 13.88 GB (14898474484)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 0
>            - CachedFileHandlesHitCount: 2.29K (2294)
>            - CachedFileHandlesMissCount: 56 (56)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 23s940ms
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 16 (16)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 1 (1)
>            - NumRowGroups: 56 (56)
>            - NumScannerThreadsStarted: 8 (8)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 1.39 GB (1491692569)
>            - PerReadThreadRawHdfsThroughput: 44.62 MB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 1s156ms
>            - RowBatchQueuePutWaitTime: 228h40m
>            - RowsRead: 427.11M (427110203)
>            - RowsReturned: 427.11M (427110203)
>            - RowsReturnedRate: 106.74 M/sec
>            - ScanRangesComplete: 56 (56)
>            - ScannerThreadsInvoluntaryContextSwitches: 7.41K (7412)
>            - ScannerThreadsTotalWallClockTime: 228h45m
>              - MaterializeTupleTime(*): 2m
>              - ScannerThreadsSysTime: 4s574ms
>              - ScannerThreadsUserTime: 2m39s
>            - ScannerThreadsVoluntaryContextSwitches: 418.25K (418255)
>            - TotalRawHdfsOpenFileTime(*): 4s930ms
>            - TotalRawHdfsReadTime(*): 5m18s
>            - TotalReadThroughput: 140.35 KB/sec
>         CodeGen:(Total: 316.986ms, non-child: 316.986ms, % non-child: 100.00%)
>            - CodegenInvoluntaryContextSwitches: 7 (7)
>            - CodegenTotalWallClockTime: 316.976ms
>              - CodegenSysTime: 19.346ms
>              - CodegenUserTime: 35.434ms
>            - CodegenVoluntaryContextSwitches: 87 (87)
>            - CompileTime: 104.224ms
>            - IrGenerationTime: 54.920ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 2.26 MB (2373832)
>            - NumFunctions: 19 (19)
>            - NumInstructions: 706 (706)
>            - OptimizationTime: 143.706ms
>            - PeakMemoryUsage: 353.00 KB (361472)
>            - PrepareTime: 13.284ms
>       Instance 9649f7ab3bcc5fb8:f4d6a60700000005 (host=slave5:22000):(Total: 28h47m, non-child: 30s821ms, % non-child: 0.03%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
>         Fragment Instance Lifecycle Event Timeline: 28h47m
>            - Prepare Finished: 25.961ms (25.961ms)
>            - Open Finished: 92.181ms (66.220ms)
>            - First Batch Produced: 1s428ms (1s336ms)
>            - First Batch Sent: 1s432ms (3.458ms)
>            - ExecInternal Finished: 28h47m (28h47m)
>         MemoryUsage(34m8s): 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB, 1.07 GB, 786.07 MB, 1.37 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.31 GB, 1.19 GB, 973.49 MB, 778.55 MB, 1.26 GB, 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB, 1.00 GB, 812.17 MB, 1.29 GB, 1.41 GB, 1.39 GB, 1.14 GB, 881.39 MB, 1.30 GB, 1.37 GB, 1.37 GB, 1.33 GB, 1.16 GB, 910.68 MB, 880.78 MB, 1.32 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.37 GB, 1.29 GB, 1.13 GB, 843.04 MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.35 GB, 1.27 GB, 1.16 GB, 1.00 GB, 848.32 MB, 709.30 MB
>         ThreadUsage(34m8s): 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
>          - AverageThreadTokens: 8.94
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 1.41 GB (1513059939)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 1.41 GB (1513059939)
>          - RowsProduced: 426.83M (426832398)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 0.000ns
>          - TotalStorageWaitTime: 1m16s
>          - TotalThreadsInvoluntaryContextSwitches: 365.02K (365022)
>          - TotalThreadsTotalWallClockTime: 257h18m
>            - TotalThreadsSysTime: 1m56s
>            - TotalThreadsUserTime: 1h9m
>          - TotalThreadsVoluntaryContextSwitches: 431.61K (431609)
>         Buffer pool:
>            - AllocTime: 0.000ns
>            - CumulativeAllocationBytes: 0
>            - CumulativeAllocations: 0 (0)
>            - PeakReservation: 0
>            - PeakUnpinnedBytes: 0
>            - PeakUsedReservation: 0
>            - ReadIoBytes: 0
>            - ReadIoOps: 0 (0)
>            - ReadIoWaitTime: 0.000ns
>            - ReservationLimit: 0
>            - WriteIoBytes: 0
>            - WriteIoOps: 0 (0)
>            - WriteIoWaitTime: 0.000ns
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 28h47m
>              - ExecTreeExecTime: 4s226ms
>            - OpenTime: 66.213ms
>              - ExecTreeOpenTime: 30.358us
>            - PrepareTime: 25.760ms
>              - ExecTreePrepareTime: 25.484ms
>         KuduTableSink:(Total: 28h46m, non-child: 28h46m, % non-child: 100.00%)
>            - KuduApplyTimer: 27h51m
>            - NumRowErrors: 345.67M (345674429)
>            - PeakMemoryUsage: 20.00 MB (20971520)
>            - RowsProcessedRate: 4.12 K/sec
>            - TotalNumRows: 426.83M (426832398)
>         HDFS_SCAN_NODE (id=0):(Total: 3s937ms, non-child: 3s937ms, % non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
>           Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.138% 2:0% 3:0% 4:0% 5:0% 6:0%
>           File Formats: PARQUET/SNAPPY:56
>           BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB, 2.08 GB, 3.23 GB, 3.30 GB, 3.48 GB, 3.67 GB, 3.78 GB, 3.87 GB, 3.94 GB, 4.09 GB, 5.01 GB, 5.31 GB, 5.49 GB, 5.66 GB, 5.81 GB, 5.92 GB, 6.09 GB, 7.05 GB, 7.45 GB, 7.69 GB, 7.86 GB, 8.14 GB, 9.07 GB, 9.35 GB, 9.56 GB, 9.70 GB, 9.81 GB, 9.90 GB, 10.23 GB, 11.09 GB, 11.27 GB, 11.41 GB, 11.56 GB, 11.66 GB, 11.73 GB, 11.82 GB, 12.00 GB, 13.08 GB, 13.48 GB, 13.61 GB, 13.68 GB, 13.74 GB, 13.80 GB, 13.85 GB, 13.87 GB, 13.88 GB
>            - FooterProcessingTime: (Avg: 256.473ms ; Min: 49.349ms ; Max: 2s343ms ; Number of samples: 56)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 7.94
>            - BytesRead: 13.88 GB (14898579078)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 13.88 GB (14898579078)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 0
>            - CachedFileHandlesHitCount: 2.29K (2293)
>            - CachedFileHandlesMissCount: 56 (56)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 22s127ms
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 16 (16)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 1 (1)
>            - NumRowGroups: 56 (56)
>            - NumScannerThreadsStarted: 8 (8)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 1.39 GB (1491831945)
>            - PerReadThreadRawHdfsThroughput: 100.21 MB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 1s334ms
>            - RowBatchQueuePutWaitTime: 228h26m
>            - RowsRead: 426.83M (426832398)
>            - RowsReturned: 426.83M (426832398)
>            - RowsReturnedRate: 108.39 M/sec
>            - ScanRangesComplete: 56 (56)
>            - ScannerThreadsInvoluntaryContextSwitches: 14.55K (14553)
>            - ScannerThreadsTotalWallClockTime: 228h30m
>              - MaterializeTupleTime(*): 2m3s
>              - ScannerThreadsSysTime: 4s079ms
>              - ScannerThreadsUserTime: 2m39s
>            - ScannerThreadsVoluntaryContextSwitches: 417.98K (417975)
>            - TotalRawHdfsOpenFileTime(*): 3s114ms
>            - TotalRawHdfsReadTime(*): 2m21s
>            - TotalReadThroughput: 140.45 KB/sec
>         CodeGen:(Total: 56.104ms, non-child: 56.104ms, % non-child: 100.00%)
>            - CodegenInvoluntaryContextSwitches: 3 (3)
>            - CodegenTotalWallClockTime: 56.095ms
>              - CodegenSysTime: 0.000ns
>              - CodegenUserTime: 37.520ms
>            - CodegenVoluntaryContextSwitches: 1 (1)
>            - CompileTime: 23.495ms
>            - IrGenerationTime: 1.462ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 2.26 MB (2373832)
>            - NumFunctions: 19 (19)
>            - NumInstructions: 706 (706)
>            - OptimizationTime: 15.936ms
>            - PeakMemoryUsage: 353.00 KB (361472)
>            - PrepareTime: 14.604ms
>
>
>
> On Thu, May 10, 2018 at 10:33 AM, Mostafa Mokhtar <mmokhtar@cloudera.com>
> wrote:
>
>> Can you share the query profile for the successful insert query?
>>
>> Thanks
>> Mostafa
>>
>> On May 9, 2018, at 9:55 PM, Geetika Gupta <geetika.gupta@knoldus.in>
>> wrote:
>>
>> Thanks, Jeszy.
>>
>> We build impala again with --release flag and data load was successful
>> after that.
>>
>> But now we are facing another issue. The table in which we loaded the
>> data has less number of rows. We executed the following command:
>>
>> insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
>>
>> This query was successful, but when we tried the count(*) on both the
>> tables, row count was different:
>>
>> 0: jdbc:hive2://slave2:21050/default> select count(*) from lineitem
>> . . . . . . . . . . . . . . . . . . > ;
>> 536870912
>>
>> 0: jdbc:hive2://slave2:21050/default> select count(*) from
>> parquetimpala500.lineitem;
>> 3000028242
>>
>> Do you have any idea about this issue.
>>
>>
>> On Mon, May 7, 2018 at 12:06 PM, Jeszy <jeszyb@gmail.com> wrote:
>>
>>> Impala doesn't store the data itself, so you can switch versions
>>> without rewriting data. But you don't have to do that, you would just
>>> have to build impala using the -release flag (of buildall.sh) and run
>>> it using the release binaries (versus the debug ones). If you would be
>>> looking at performance, using the release version is highly
>>> recommended anyway.
>>>
>>> On 7 May 2018 at 08:30, Geetika Gupta <geetika.gupta@knoldus.in> wrote:
>>> > Hi Jeszy,
>>> >
>>> > Currently, we are using the apache impala's Github master branch code.
>>> We
>>> > tried using the released version but we encountered some errors
>>> related to
>>> > downloading of dependencies and could not complete the installation.
>>> >
>>> > The current version of impala we are using: 2.12
>>> >
>>> > We can't try with the new release as we have already loaded 500GB of
>>> TPCH
>>> > data on our cluster.
>>> >
>>> > On Mon, May 7, 2018 at 11:43 AM, Jeszy <jeszyb@gmail.com> wrote:
>>> >>
>>> >> What version of Impala are you using?
>>> >> DCHECKs won't be triggered if you run a release build. Looking at the
>>> >> code, it should work with bad values if not for the DCHECK. Can you
>>> >> try using a release build?
>>> >>
>>> >> On 7 May 2018 at 08:04, Geetika Gupta <geetika.gupta@knoldus.in>
>>> wrote:
>>> >> > Hi community,
>>> >> >
>>> >> > I was trying to load 500GB of TPCH data into kudu table using the
>>> >> > following
>>> >> > query:
>>> >> >
>>> >> > insert into lineitem select * from PARQUETIMPALA500.LINEITEM
>>> >> >
>>> >> > While executing the query for around 17 hrs it got cancelled as the
>>> >> > impalad
>>> >> > process of that machine got aborted. Here are the logs of the
>>> impalad
>>> >> > process.
>>> >> >
>>> >> > impalad.ERROR
>>> >> >
>>> >> > Log file created at: 2018/05/06 13:40:34
>>> >> > Running on machine: slave2
>>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>> >> > E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged
>>> to
>>> >> > this
>>> >> > file.
>>> >> > SLF4J: Class path contains multiple SLF4J bindings.
>>> >> > SLF4J: Found binding in
>>> >> >
>>> >> > [jar:file:/root/softwares/impala/fe/target/dependency/slf4j-
>>> log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> >> > SLF4J: Found binding in
>>> >> >
>>> >> > [jar:file:/root/softwares/impala/testdata/target/dependency/
>>> slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>> >> > explanation.
>>> >> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFac
>>> tory]
>>> >> > 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load
>>> >> > native-hadoop
>>> >> > library for your platform... using builtin-java classes where
>>> applicable
>>> >> > 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
>>> >> > short-circuit
>>> >> > local reads feature cannot be used because libhadoop cannot be
>>> loaded.
>>> >> > tcmalloc: large alloc 1073741824 bytes == 0x484434000 @  0x4135176
>>> >> > 0x7fd9e9fc3929
>>> >> > tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @
>>> 0x4135176
>>> >> > 0x7fd9e9fc3929
>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>> >> > log_entry.count
>>> >> >> 0 (-1831809966 vs. 0)
>>> >> > *** Check failure stack trace: ***
>>> >> >     @          0x3fc0c0d  google::LogMessage::Fail()
>>> >> >     @          0x3fc24b2  google::LogMessage::SendToLog()
>>> >> >     @          0x3fc05e7  google::LogMessage::Flush()
>>> >> >     @          0x3fc3bae  google::LogMessageFatal::~LogM
>>> essageFatal()
>>> >> >     @          0x1bbcb31  impala::PrintErrorMap()
>>> >> >     @          0x1bbcd07  impala::PrintErrorMapToString()
>>> >> >     @          0x2decbd7  impala::Coordinator::GetErrorLog()
>>> >> >     @          0x1a8d634  impala::ImpalaServer::UnregisterQuery()
>>> >> >     @          0x1b29264  impala::ImpalaServer::CloseOperation()
>>> >> >     @          0x2c5ce86
>>> >> >
>>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::pr
>>> ocess_CloseOperation()
>>> >> >     @          0x2c56b8c
>>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::di
>>> spatchCall()
>>> >> >     @          0x2c2fcb1
>>> >> > impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
>>> >> >     @          0x16fdb20  apache::thrift::TDispatchProce
>>> ssor::process()
>>> >> >     @          0x18ea6b3
>>> >> > apache::thrift::server::TAcceptQueueServer::Task::run()
>>> >> >     @          0x18e2181  impala::ThriftThread::RunRunnable()
>>> >> >     @          0x18e3885  boost::_mfi::mf2<>::operator()()
>>> >> >     @          0x18e371b  boost::_bi::list3<>::operator()<>()
>>> >> >     @          0x18e3467  boost::_bi::bind_t<>::operator()()
>>> >> >     @          0x18e337a
>>> >> > boost::detail::function::void_function_obj_invoker0<>::invoke()
>>> >> >     @          0x192761c  boost::function0<>::operator()()
>>> >> >     @          0x1c3ebf7  impala::Thread::SuperviseThread()
>>> >> >     @          0x1c470cd  boost::_bi::list5<>::operator()<>()
>>> >> >     @          0x1c46ff1  boost::_bi::bind_t<>::operator()()
>>> >> >     @          0x1c46fb4  boost::detail::thread_data<>::run()
>>> >> >     @          0x2eedb4a  thread_proxy
>>> >> >     @     0x7fda1dbb16ba  start_thread
>>> >> >     @     0x7fda1d8e741d  clone
>>> >> > Wrote minidump to
>>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>>> >> >
>>> >> > impalad.FATAL
>>> >> >
>>> >> > Log file created at: 2018/05/07 09:46:12
>>> >> > Running on machine: slave2
>>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>> >> > log_entry.count
>>> >> >> 0 (-1831809966 vs. 0)
>>> >> >
>>> >> > Impalad.INFO
>>> >> > edentials={real_user=root}} blocked reactor thread for 34288.6us
>>> >> > I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for
>>> RPC
>>> >> > call
>>> >> > kudu.tserver.TabletServerService.Write -> {remote=
>>> 136.243.74.42:7050
>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor
>>> thread for
>>> >> > 35859.8us
>>> >> > I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for
>>> RPC
>>> >> > call
>>> >> > kudu.tserver.TabletServerService.Write -> {remote=
>>> 136.243.74.42:7050
>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor
>>> thread for
>>> >> > 40664.9us
>>> >> > I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for
>>> RPC
>>> >> > call
>>> >> > kudu.tserver.TabletServerService.Write -> {remote=
>>> 136.243.74.42:7050
>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor
>>> thread for
>>> >> > 49514.6us
>>> >> > I0507 09:46:12.664149  4507 coordinator.cc:783] Release admission
>>> >> > control
>>> >> > resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>> >> > log_entry.count
>>> >> >> 0 (-1831809966 vs. 0)
>>> >> > Wrote minidump to
>>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>>> >> >
>>> >> > Note:
>>> >> > We are executing the queries on 8 node cluster with the following
>>> >> > configuration
>>> >> > Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk
>>> >> > each,
>>> >> > Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Regards,
>>> >> > Geetika Gupta
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Regards,
>>> > Geetika Gupta
>>>
>>
>>
>>
>> --
>> Regards,
>> Geetika Gupta
>>
>>
>
>
> --
> Regards,
> Geetika Gupta
>

Mime
View raw message