impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geetika Gupta <geetika.gu...@knoldus.in>
Subject Re: Issue in data loading in Impala + Kudu
Date Thu, 10 May 2018 05:13:16 GMT
Please find below the query profile :

Query (id=9649f7ab3bcc5fb8:f4d6a60700000000):
  Summary:
    Session ID: b04a9080d1e1724d:41f2b0d261e8f280
    Session Type: HIVESERVER2
    HiveServer2 Protocol Version: V6
    Start Time: 2018-05-08 17:55:23.181264000
    End Time: 2018-05-10 00:34:17.784273000
    Query Type: DML
    Query State: FINISHED
    Query Status: OK
    Impala Version: impalad version 3.0.0-SNAPSHOT RELEASE (build
b68e06997c1f49f6b723d78e217efddec4f56f3a)
    User: root
    Connected User: root
    Delegated User:
    Network Address: ::ffff:46.4.88.233:59862
    Default Db: kudu_impala_500
    Sql Statement: insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
    Coordinator: slave2:22000
    Query Options (set by configuration):
    Query Options (set by configuration and planner): MT_DOP=0
    Plan:
----------------
Max Per-Host Resource Reservation: Memory=0B
Per-Host Resource Estimates: Memory=704.00MB
WARNING: The following tables are missing relevant table and/or column
statistics.
parquetimpala500.lineitem

F00:PLAN FRAGMENT [RANDOM] hosts=7 instances=7
|  Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B
INSERT INTO KUDU [kudu_impala_500.lineitem]
|  mem-estimate=0B mem-reservation=0B
|
00:SCAN HDFS [parquetimpala500.lineitem, RANDOM]
   partitions=1/1 files=396 size=97.29GB
   stored statistics:
     table: rows=unavailable size=unavailable
     columns: unavailable
   extrapolated-rows=disabled
   mem-estimate=704.00MB mem-reservation=0B
   tuple-ids=0 row-size=171B cardinality=unavailable
----------------
    Estimated Per-Host Mem: 738197504
    Tables Missing Stats: parquetimpala500.lineitem
    Per Host Min Reservation: slave1:22000(0) slave2:22000(0)
slave3:22000(0) slave4:22000(0) slave5:22000(0) slave6:22000(0)
slave7:22000(0)
    Request Pool: default-pool
    Admission result: Admitted immediately
    ExecSummary:
Operator       #Hosts  Avg Time  Max Time  #Rows  Est. #Rows  Peak Mem
 Est. Peak Mem  Detail
-----------------------------------------------------------------------------------------------------------------
00:SCAN HDFS        7   4s417ms   6s154ms  3.00B          -1   1.39 GB
     704.00 MB  parquetimpala500.lineitem
    Errors: Key already present in Kudu table
'impala::kudu_impala_500.LINEITEM'. (1 of -1831809966 similar)

    Query Compilation: 6s413ms
       - Metadata load started: 14.443ms (14.443ms)
       - Metadata load finished. loaded-tables=2/2 load-requests=1
catalog-updates=7: 6s298ms (6s283ms)
       - Analysis finished: 6s301ms (3.659ms)
       - Value transfer graph computed: 6s302ms (282.554us)
       - Single node plan created: 6s363ms (61.624ms)
       - Runtime filters computed: 6s363ms (98.878us)
       - Distributed plan created: 6s366ms (2.832ms)
       - Planning finished: 6s413ms (46.751ms)
    Query Timeline: 30h38m
       - Query submitted: 56.829us (56.829us)
       - Planning finished: 6s431ms (6s431ms)
       - Submit for admission: 6s432ms (821.867us)
       - Completed admission: 6s432ms (14.519us)
       - Ready to start on 7 backends: 6s432ms (99.515us)
       - All 7 execution backends (7 fragment instances) started:
6s535ms (103.320ms)
       - Released admission control resources: 30h38m (30h38m)
       - DML data written: 30h38m (934.739us)
       - DML Metastore update finished: 30h38m (157.938us)
       - Request finished: 30h38m (41.379us)
       - First row fetched: 30h38m (173.124us)
       - First row fetched: 30h38m (1.750ms)
       - First row fetched: 30h38m (1.361ms)
       - Unregister query: 30h38m (1.456ms)
     - ComputeScanRangeAssignmentTimer: 558.440us
  ImpalaServer:
     - ClientFetchWaitTimer: 4.721ms
     - MetastoreUpdateTimer: 191.424us
     - RowMaterializationTimer: 0.000ns
  Execution Profile 9649f7ab3bcc5fb8:f4d6a60700000000:(Total: 30h38m,
non-child: 0.000ns, % non-child: 0.00%)
    Number of filters: 0
    Filter routing table:
 ID  Src. Node  Tgt. Node(s)  Target type  Partition filter  Pending
(Expected)  First arrived  Completed   Enabled
-------------------------------------------------------------------------------------------------------------------

    Backend startup latencies: Count: 7, min / max: 1ms / 103ms, 25th
%-ile: 1ms, 50th %-ile: 67ms, 75th %-ile: 67ms, 90th %-ile: 80ms, 95th
%-ile: 103ms, 99.9th %-ile: 103ms
    DML Stats:
Partition: Default
NumModifiedRows: 536870912
NumRowErrors: 2463157330
  BytesWritten: 0
    Per Node Peak Memory Usage: slave1:22000(1.40 GB)
slave2:22000(1.41 GB) slave3:22000(1.39 GB) slave4:22000(1.41 GB)
slave5:22000(1.41 GB) slave6:22000(1.40 GB) slave7:22000(1.40 GB)
     - FiltersReceived: 0 (0)
     - FinalizationTimer: 0.000ns
     - NumBackends: 7 (7)
     - NumFragmentInstances: 7 (7)
     - NumFragments: 1 (1)
    Averaged Fragment F00:(Total: 29h51m, non-child: 14s973ms, %
non-child: 0.01%)
      split sizes:  min: 13.63 GB, max: 14.10 GB, avg: 13.90 GB,
stddev: 143.22 MB
      completion times: min:28h47m  max:30h38m  mean: 29h51m  stddev:41m26s
      execution rates: min:129.50 KB/sec  max:140.31 KB/sec
mean:135.67 KB/sec  stddev:3.50 KB/sec
      num instances: 7
       - AverageThreadTokens: 8.89
       - BloomFilterBytes: 0
       - PeakMemoryUsage: 1.40 GB (1503884428)
       - PeakReservation: 0
       - PeakUsedReservation: 0
       - PerHostPeakMemUsage: 1.40 GB (1503884428)
       - RowsProduced: 428.58M (428575463)
       - TotalNetworkReceiveTime: 0.000ns
       - TotalNetworkSendTime: 0.000ns
       - TotalStorageWaitTime: 2m10s
       - TotalThreadsInvoluntaryContextSwitches: 369.81K (369812)
       - TotalThreadsTotalWallClockTime: 265h20m
         - TotalThreadsSysTime: 1m59s
         - TotalThreadsUserTime: 1h9m
       - TotalThreadsVoluntaryContextSwitches: 433.92K (433917)
      Buffer pool:
         - AllocTime: 0.000ns
         - CumulativeAllocationBytes: 0
         - CumulativeAllocations: 0 (0)
         - PeakReservation: 0
         - PeakUnpinnedBytes: 0
         - PeakUsedReservation: 0
         - ReadIoBytes: 0
         - ReadIoOps: 0 (0)
         - ReadIoWaitTime: 0.000ns
         - ReservationLimit: 0
         - WriteIoBytes: 0
         - WriteIoOps: 0 (0)
         - WriteIoWaitTime: 0.000ns
      Fragment Instance Lifecycle Timings:
         - ExecTime: 29h51m
           - ExecTreeExecTime: 4s724ms
         - OpenTime: 519.095ms
           - ExecTreeOpenTime: 36.350us
         - PrepareTime: 60.445ms
           - ExecTreePrepareTime: 19.494ms
      KuduTableSink:(Total: 29h50m, non-child: 29h50m, % non-child: 100.00%)
         - KuduApplyTimer: 28h54m
         - NumRowErrors: 351.88M (351879618)
         - PeakMemoryUsage: 20.00 MB (20971520)
         - RowsProcessedRate: 3.99 K/sec
         - TotalNumRows: 428.58M (428575463)
      HDFS_SCAN_NODE (id=0):(Total: 4s417ms, non-child: 4s417ms, %
non-child: 100.00%)
         - AverageHdfsReadThreadConcurrency: 0.00
         - AverageScannerThreadConcurrency: 7.89
         - BytesRead: 13.90 GB (14928723218)
         - BytesReadDataNodeCache: 0
         - BytesReadLocal: 13.90 GB (14928723218)
         - BytesReadRemoteUnexpected: 0
         - BytesReadShortCircuit: 0
         - CachedFileHandlesHitCount: 2.31K (2305)
         - CachedFileHandlesMissCount: 56 (56)
         - CollectionItemsRead: 0 (0)
         - DecompressionTime: 22s321ms
         - MaxCompressedTextFileLength: 0
         - NumColumns: 16 (16)
         - NumDictFilteredRowGroups: 0 (0)
         - NumDisksAccessed: 1 (1)
         - NumRowGroups: 56 (56)
         - NumScannerThreadsStarted: 8 (8)
         - NumScannersWithNoReads: 0 (0)
         - NumStatsFilteredRowGroups: 0 (0)
         - PeakMemoryUsage: 1.38 GB (1482717084)
         - PerReadThreadRawHdfsThroughput: 69.65 MB/sec
         - RemoteScanRanges: 0 (0)
         - RowBatchQueueGetWaitTime: 1s754ms
         - RowBatchQueuePutWaitTime: 235h24m
         - RowsRead: 428.58M (428575463)
         - RowsReturned: 428.58M (428575463)
         - RowsReturnedRate: 99.08 M/sec
         - ScanRangesComplete: 56 (56)
         - ScannerThreadsInvoluntaryContextSwitches: 12.33K (12332)
         - ScannerThreadsTotalWallClockTime: 235h28m
           - MaterializeTupleTime(*): 2m3s
           - ScannerThreadsSysTime: 3s896ms
           - ScannerThreadsUserTime: 2m39s
         - ScannerThreadsVoluntaryContextSwitches: 419.69K (419690)
         - TotalRawHdfsOpenFileTime(*): 3s995ms
         - TotalRawHdfsReadTime(*): 4m2s
         - TotalReadThroughput: 135.78 KB/sec
      CodeGen:(Total: 436.431ms, non-child: 436.431ms, % non-child: 100.00%)
         - CodegenInvoluntaryContextSwitches: 5 (5)
         - CodegenTotalWallClockTime: 436.422ms
           - CodegenSysTime: 8.585ms
           - CodegenUserTime: 41.806ms
         - CodegenVoluntaryContextSwitches: 76 (76)
         - CompileTime: 179.278ms
         - IrGenerationTime: 90.176ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 2.26 MB (2373832)
         - NumFunctions: 19 (19)
         - NumInstructions: 706 (706)
         - OptimizationTime: 151.857ms
         - PeakMemoryUsage: 353.00 KB (361472)
         - PrepareTime: 14.499ms
    Fragment F00:
      Instance 9649f7ab3bcc5fb8:f4d6a60700000000
(host=slave3:22000):(Total: 30h38m, non-child: 11s364ms, % non-child:
0.01%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.63 GB
        Fragment Instance Lifecycle Event Timeline: 30h38m
           - Prepare Finished: 86.126ms (86.126ms)
           - Open Finished: 1s157ms (1s070ms)
           - First Batch Produced: 3s031ms (1s874ms)
           - First Batch Sent: 3s037ms (5.724ms)
           - ExecInternal Finished: 30h38m (30h38m)
        MemoryUsage(34m8s): 1.28 GB, 1.34 GB, 1.37 GB, 1.35 GB, 1.21
GB, 925.43 MB, 727.39 MB, 1.17 GB, 1.35 GB, 1.35 GB, 1.35 GB, 1.35 GB,
1.26 GB, 1.06 GB, 836.27 MB, 792.34 MB, 1.26 GB, 1.36 GB, 1.36 GB,
1.30 GB, 1.10 GB, 827.96 MB, 1.01 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.12
GB, 875.56 MB, 979.49 MB, 1.34 GB, 1.37 GB, 1.37 GB, 1.28 GB, 1.10 GB,
950.36 MB, 805.45 MB, 963.12 MB, 1.28 GB, 1.34 GB, 1.34 GB, 1.30 GB,
1.04 GB, 808.54 MB, 994.36 MB, 1.17 GB, 1.18 GB, 1.18 GB, 1.18 GB,
1.17 GB, 1.13 GB, 1.04 GB, 954.52 MB, 811.51 MB
        ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 8, 8, 8, 8, 8, 8, 8, 8, 8
         - AverageThreadTokens: 8.80
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 1.39 GB (1487275684)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 1.39 GB (1487275684)
         - RowsProduced: 424.29M (424287956)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 1m32s
         - TotalThreadsInvoluntaryContextSwitches: 349.35K (349347)
         - TotalThreadsTotalWallClockTime: 269h37m
           - TotalThreadsSysTime: 1m54s
           - TotalThreadsUserTime: 1h6m
         - TotalThreadsVoluntaryContextSwitches: 438.17K (438165)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 30h38m
             - ExecTreeExecTime: 4s586ms
           - OpenTime: 1s070ms
             - ExecTreeOpenTime: 32.390us
           - PrepareTime: 85.903ms
             - ExecTreePrepareTime: 16.283ms
        KuduTableSink:(Total: 30h38m, non-child: 30h38m, % non-child: 100.00%)
           - KuduApplyTimer: 29h44m
           - NumRowErrors: 371.48M (371483541)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 3.85 K/sec
           - TotalNumRows: 424.29M (424287956)
        HDFS_SCAN_NODE (id=0):(Total: 4s304ms, non-child: 4s304ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.63 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
          Hdfs Read Thread Concurrency Bucket: 0:99.87% 1:0.1279% 2:0%
3:0% 4:0% 5:0% 6:0%
          File Formats: PARQUET/SNAPPY:56
          BytesRead(34m8s): 1.19 GB, 1.46 GB, 1.65 GB, 1.78 GB, 1.88
GB, 1.97 GB, 2.01 GB, 2.91 GB, 3.27 GB, 3.46 GB, 3.61 GB, 3.72 GB,
3.80 GB, 3.90 GB, 3.96 GB, 4.16 GB, 5.05 GB, 5.36 GB, 5.60 GB, 5.76
GB, 5.87 GB, 5.94 GB, 6.57 GB, 7.32 GB, 7.57 GB, 7.75 GB, 7.86 GB,
7.92 GB, 8.43 GB, 9.18 GB, 9.43 GB, 9.62 GB, 9.75 GB, 9.84 GB, 9.89
GB, 9.94 GB, 10.41 GB, 11.07 GB, 11.24 GB, 11.42 GB, 11.66 GB, 11.82
GB, 12.05 GB, 12.48 GB, 12.96 GB, 13.07 GB, 13.19 GB, 13.31 GB, 13.38
GB, 13.46 GB, 13.51 GB, 13.57 GB, 13.61 GB
           - FooterProcessingTime: (Avg: 381.833ms ; Min: 46.831ms ;
Max: 4s214ms ; Number of samples: 56)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 7.80
           - BytesRead: 13.63 GB (14635606518)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 13.63 GB (14635606518)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 2.27K (2268)
           - CachedFileHandlesMissCount: 56 (56)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 19s612ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 16 (16)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 56 (56)
           - NumScannerThreadsStarted: 8 (8)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 1.37 GB (1466047697)
           - PerReadThreadRawHdfsThroughput: 100.45 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 1s872ms
           - RowBatchQueuePutWaitTime: 238h55m
           - RowsRead: 424.29M (424287956)
           - RowsReturned: 424.29M (424287956)
           - RowsReturnedRate: 98.57 M/sec
           - ScanRangesComplete: 56 (56)
           - ScannerThreadsInvoluntaryContextSwitches: 12.67K (12674)
           - ScannerThreadsTotalWallClockTime: 238h58m
             - MaterializeTupleTime(*): 1m57s
             - ScannerThreadsSysTime: 3s421ms
             - ScannerThreadsUserTime: 2m31s
           - ScannerThreadsVoluntaryContextSwitches: 415.49K (415490)
           - TotalRawHdfsOpenFileTime(*): 2s904ms
           - TotalRawHdfsReadTime(*): 2m18s
           - TotalReadThroughput: 129.60 KB/sec
        CodeGen:(Total: 915.511ms, non-child: 915.511ms, % non-child: 100.00%)
           - CodegenInvoluntaryContextSwitches: 10 (10)
           - CodegenTotalWallClockTime: 915.504ms
             - CodegenSysTime: 8.256ms
             - CodegenUserTime: 45.032ms
           - CodegenVoluntaryContextSwitches: 87 (87)
           - CompileTime: 319.788ms
           - IrGenerationTime: 276.446ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 2.26 MB (2373832)
           - NumFunctions: 19 (19)
           - NumInstructions: 706 (706)
           - OptimizationTime: 301.622ms
           - PeakMemoryUsage: 353.00 KB (361472)
           - PrepareTime: 16.858ms
      Instance 9649f7ab3bcc5fb8:f4d6a60700000002
(host=slave1:22000):(Total: 30h17m, non-child: 1s508ms, % non-child:
0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:57/14.10 GB
        Fragment Instance Lifecycle Event Timeline: 30h17m
           - Prepare Finished: 32.853ms (32.853ms)
           - Open Finished: 362.691ms (329.838ms)
           - First Batch Produced: 1s725ms (1s362ms)
           - First Batch Sent: 1s728ms (3.415ms)
           - ExecInternal Finished: 30h17m (30h17m)
        MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.27 GB, 1.07
GB, 798.75 MB, 1.36 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.19 GB,
974.24 MB, 800.43 MB, 1.27 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.26 GB,
1010.99 MB, 805.20 MB, 1.29 GB, 1.38 GB, 1.38 GB, 1.27 GB, 987.74 MB,
811.33 MB, 1.27 GB, 1.38 GB, 1.38 GB, 1.34 GB, 1.18 GB, 922.96 MB,
765.77 MB, 1.13 GB, 1.36 GB, 1.36 GB, 1.36 GB, 1.37 GB, 1.30 GB, 1.03
GB, 919.88 MB, 1.38 GB, 1.38 GB, 1.38 GB, 1.39 GB, 1.35 GB, 1.24 GB,
1.17 GB, 1005.37 MB, 844.55 MB, 751.70 MB, 518.60 MB
        ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 5
         - AverageThreadTokens: 8.91
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 1.40 GB (1500404835)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 1.40 GB (1500404835)
         - RowsProduced: 434.96M (434955041)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 2m56s
         - TotalThreadsInvoluntaryContextSwitches: 370.59K (370587)
         - TotalThreadsTotalWallClockTime: 269h58m
           - TotalThreadsSysTime: 1m58s
           - TotalThreadsUserTime: 1h9m
         - TotalThreadsVoluntaryContextSwitches: 437.63K (437630)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 30h17m
             - ExecTreeExecTime: 4s575ms
           - OpenTime: 329.834ms
             - ExecTreeOpenTime: 45.986us
           - PrepareTime: 32.581ms
             - ExecTreePrepareTime: 18.647ms
        KuduTableSink:(Total: 30h17m, non-child: 30h17m, % non-child: 100.00%)
           - KuduApplyTimer: 29h22m
           - NumRowErrors: 357.86M (357862755)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 3.99 K/sec
           - TotalNumRows: 434.96M (434955041)
        HDFS_SCAN_NODE (id=0):(Total: 4s229ms, non-child: 4s229ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:57/14.10 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of 57
          Hdfs Read Thread Concurrency Bucket: 0:99.63% 1:0.3706% 2:0%
3:0% 4:0% 5:0% 6:0%
          File Formats: PARQUET/SNAPPY:57
          BytesRead(34m8s): 1.32 GB, 1.59 GB, 1.74 GB, 1.85 GB, 1.94
GB, 2.11 GB, 3.21 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB,
3.94 GB, 4.11 GB, 5.04 GB, 5.32 GB, 5.50 GB, 5.67 GB, 5.81 GB, 5.92
GB, 6.09 GB, 7.06 GB, 7.40 GB, 7.63 GB, 7.79 GB, 7.90 GB, 8.08 GB,
9.02 GB, 9.35 GB, 9.55 GB, 9.70 GB, 9.81 GB, 9.90 GB, 10.01 GB, 10.74
GB, 11.19 GB, 11.34 GB, 11.48 GB, 11.57 GB, 11.69 GB, 11.84 GB, 12.25
GB, 13.25 GB, 13.41 GB, 13.50 GB, 13.60 GB, 13.67 GB, 13.73 GB, 13.79
GB, 13.83 GB, 13.86 GB, 13.92 GB, 14.03 GB
           - FooterProcessingTime: (Avg: 414.198ms ; Min: 51.291ms ;
Max: 2s894ms ; Number of samples: 57)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 7.91
           - BytesRead: 14.11 GB (15148002597)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 14.11 GB (15148002597)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 2.33K (2333)
           - CachedFileHandlesMissCount: 57 (57)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 23s330ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 16 (16)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 57 (57)
           - NumScannerThreadsStarted: 8 (8)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 1.38 GB (1479367012)
           - PerReadThreadRawHdfsThroughput: 36.38 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 1s354ms
           - RowBatchQueuePutWaitTime: 239h35m
           - RowsRead: 434.96M (434955041)
           - RowsReturned: 434.96M (434955041)
           - RowsReturnedRate: 102.83 M/sec
           - ScanRangesComplete: 57 (57)
           - ScannerThreadsInvoluntaryContextSwitches: 7.23K (7231)
           - ScannerThreadsTotalWallClockTime: 239h40m
             - MaterializeTupleTime(*): 2m1s
             - ScannerThreadsSysTime: 4s044ms
             - ScannerThreadsUserTime: 2m42s
           - ScannerThreadsVoluntaryContextSwitches: 425.94K (425935)
           - TotalRawHdfsOpenFileTime(*): 5s555ms
           - TotalRawHdfsReadTime(*): 6m37s
           - TotalReadThroughput: 135.68 KB/sec
        CodeGen:(Total: 274.452ms, non-child: 274.452ms, % non-child: 100.00%)
           - CodegenInvoluntaryContextSwitches: 4 (4)
           - CodegenTotalWallClockTime: 274.443ms
             - CodegenSysTime: 7.440ms
             - CodegenUserTime: 40.295ms
           - CodegenVoluntaryContextSwitches: 86 (86)
           - CompileTime: 103.404ms
           - IrGenerationTime: 32.648ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 2.26 MB (2373832)
           - NumFunctions: 19 (19)
           - NumInstructions: 706 (706)
           - OptimizationTime: 122.033ms
           - PeakMemoryUsage: 353.00 KB (361472)
           - PrepareTime: 15.943ms
      Instance 9649f7ab3bcc5fb8:f4d6a60700000001
(host=slave4:22000):(Total: 30h16m, non-child: 2s019ms, % non-child:
0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
        Fragment Instance Lifecycle Event Timeline: 30h16m
           - Prepare Finished: 105.379ms (105.379ms)
           - Open Finished: 519.128ms (413.748ms)
           - First Batch Produced: 1s789ms (1s270ms)
           - First Batch Sent: 1s795ms (6.364ms)
           - ExecInternal Finished: 30h16m (30h16m)
        MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.23
GB, 924.30 MB, 715.63 MB, 1.26 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.40 GB,
1.30 GB, 1.08 GB, 843.90 MB, 887.06 MB, 1.40 GB, 1.41 GB, 1.39 GB,
1.19 GB, 826.21 MB, 1.19 GB, 1.38 GB, 1.38 GB, 1.31 GB, 1.05 GB,
748.49 MB, 1.17 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.24 GB, 999.76 MB,
733.27 MB, 995.28 MB, 1.35 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.35 GB,
1.10 GB, 861.01 MB, 1.39 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.39 GB, 1.33
GB, 1.24 GB, 1.13 GB, 1007.59 MB, 820.51 MB, 655.33 MB
        ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
         - AverageThreadTokens: 8.97
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 1.41 GB (1513166691)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 1.41 GB (1513166691)
         - RowsProduced: 425.55M (425550934)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 1m49s
         - TotalThreadsInvoluntaryContextSwitches: 364.52K (364523)
         - TotalThreadsTotalWallClockTime: 271h36m
           - TotalThreadsSysTime: 1m56s
           - TotalThreadsUserTime: 1h9m
         - TotalThreadsVoluntaryContextSwitches: 431.64K (431643)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 30h16m
             - ExecTreeExecTime: 4s185ms
           - OpenTime: 413.744ms
             - ExecTreeOpenTime: 31.615us
           - PrepareTime: 105.138ms
             - ExecTreePrepareTime: 17.849ms
        KuduTableSink:(Total: 30h16m, non-child: 30h16m, % non-child: 100.00%)
           - KuduApplyTimer: 29h20m
           - NumRowErrors: 336.92M (336918666)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 3.90 K/sec
           - TotalNumRows: 425.55M (425550934)
        HDFS_SCAN_NODE (id=0):(Total: 3s886ms, non-child: 3s886ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
          Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.1368% 2:0%
3:0% 4:0% 5:0% 6:0%
          File Formats: PARQUET/SNAPPY:56
          BytesRead(34m8s): 1.29 GB, 1.51 GB, 1.66 GB, 1.79 GB, 1.88
GB, 1.97 GB, 2.00 GB, 3.01 GB, 3.30 GB, 3.49 GB, 3.63 GB, 3.74 GB,
3.82 GB, 3.92 GB, 3.96 GB, 4.29 GB, 5.26 GB, 5.47 GB, 5.71 GB, 5.87
GB, 5.94 GB, 6.88 GB, 7.34 GB, 7.59 GB, 7.76 GB, 7.89 GB, 7.93 GB,
8.83 GB, 9.30 GB, 9.51 GB, 9.67 GB, 9.79 GB, 9.89 GB, 9.91 GB, 10.45
GB, 11.14 GB, 11.30 GB, 11.46 GB, 11.55 GB, 11.68 GB, 11.83 GB, 12.10
GB, 13.21 GB, 13.35 GB, 13.47 GB, 13.56 GB, 13.64 GB, 13.71 GB, 13.76
GB, 13.81 GB, 13.86 GB, 13.87 GB, 13.88 GB
           - FooterProcessingTime: (Avg: 331.439ms ; Min: 46.760ms ;
Max: 2s508ms ; Number of samples: 56)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 7.97
           - BytesRead: 13.88 GB (14898393824)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 13.88 GB (14898393824)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 2.30K (2296)
           - CachedFileHandlesMissCount: 56 (56)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 22s686ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 16 (16)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 56 (56)
           - NumScannerThreadsStarted: 8 (8)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 1.39 GB (1491938707)
           - PerReadThreadRawHdfsThroughput: 95.20 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 1s255ms
           - RowBatchQueuePutWaitTime: 241h15m
           - RowsRead: 425.55M (425550934)
           - RowsReturned: 425.55M (425550934)
           - RowsReturnedRate: 109.50 M/sec
           - ScanRangesComplete: 56 (56)
           - ScannerThreadsInvoluntaryContextSwitches: 12.02K (12019)
           - ScannerThreadsTotalWallClockTime: 241h20m
             - MaterializeTupleTime(*): 2m6s
             - ScannerThreadsSysTime: 3s548ms
             - ScannerThreadsUserTime: 2m39s
           - ScannerThreadsVoluntaryContextSwitches: 416.75K (416750)
           - TotalRawHdfsOpenFileTime(*): 3s230ms
           - TotalRawHdfsReadTime(*): 2m29s
           - TotalReadThroughput: 133.55 KB/sec
        CodeGen:(Total: 315.226ms, non-child: 315.226ms, % non-child: 100.00%)
           - CodegenInvoluntaryContextSwitches: 9 (9)
           - CodegenTotalWallClockTime: 315.217ms
             - CodegenSysTime: 5.142ms
             - CodegenUserTime: 43.310ms
           - CodegenVoluntaryContextSwitches: 91 (91)
           - CompileTime: 129.393ms
           - IrGenerationTime: 60.406ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 2.26 MB (2373832)
           - NumFunctions: 19 (19)
           - NumInstructions: 706 (706)
           - OptimizationTime: 111.534ms
           - PeakMemoryUsage: 353.00 KB (361472)
           - PrepareTime: 13.425ms
      Instance 9649f7ab3bcc5fb8:f4d6a60700000004
(host=slave7:22000):(Total: 30h10m, non-child: 16s427ms, % non-child:
0.02%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:58/14.03 GB
        Fragment Instance Lifecycle Event Timeline: 30h10m
           - Prepare Finished: 105.361ms (105.361ms)
           - Open Finished: 980.610ms (875.249ms)
           - First Batch Produced: 4s553ms (3s573ms)
           - First Batch Sent: 4s559ms (5.668ms)
           - ExecInternal Finished: 30h10m (30h10m)
        MemoryUsage(34m8s): 1.38 GB, 1.40 GB, 1.40 GB, 1.28 GB, 1.08
GB, 796.55 MB, 1.34 GB, 1.37 GB, 1.35 GB, 1.34 GB, 1.27 GB, 1.21 GB,
983.86 MB, 1.08 GB, 1.38 GB, 1.36 GB, 1.35 GB, 1.28 GB, 1.17 GB,
921.06 MB, 1.02 GB, 1.35 GB, 1.33 GB, 1.28 GB, 1.15 GB, 905.74 MB,
1.01 GB, 1.36 GB, 1.34 GB, 1.32 GB, 1.21 GB, 1.10 GB, 877.34 MB, 1.04
GB, 1.32 GB, 1.38 GB, 1.36 GB, 1.35 GB, 1.32 GB, 1.21 GB, 1000.76 MB,
1.11 GB, 1.37 GB, 1.36 GB, 1.33 GB, 1.29 GB, 1.20 GB, 1.09 GB, 1.07
GB, 937.32 MB, 814.51 MB, 480.87 MB, 155.08 MB
        ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 5, 2
         - AverageThreadTokens: 8.80
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 1.40 GB (1500511831)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 1.40 GB (1500511831)
         - RowsProduced: 433.04M (433043954)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 3m18s
         - TotalThreadsInvoluntaryContextSwitches: 374.99K (374993)
         - TotalThreadsTotalWallClockTime: 265h25m
           - TotalThreadsSysTime: 1m57s
           - TotalThreadsUserTime: 1h10m
         - TotalThreadsVoluntaryContextSwitches: 435.67K (435665)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 30h10m
             - ExecTreeExecTime: 6s426ms
           - OpenTime: 875.240ms
             - ExecTreeOpenTime: 41.994us
           - PrepareTime: 105.102ms
             - ExecTreePrepareTime: 24.800ms
        KuduTableSink:(Total: 30h9m, non-child: 30h9m, % non-child: 100.00%)
           - KuduApplyTimer: 29h12m
           - NumRowErrors: 357.43M (357426783)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 3.99 K/sec
           - TotalNumRows: 433.04M (433043954)
        HDFS_SCAN_NODE (id=0):(Total: 6s154ms, non-child: 6s154ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:58/14.03 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 58 out of 58
          Hdfs Read Thread Concurrency Bucket: 0:99.68% 1:0.3169% 2:0%
3:0% 4:0% 5:0% 6:0%
          File Formats: PARQUET/SNAPPY:58
          BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94
GB, 2.10 GB, 3.18 GB, 3.29 GB, 3.44 GB, 3.62 GB, 3.72 GB, 3.95 GB,
4.03 GB, 4.61 GB, 5.22 GB, 5.37 GB, 5.52 GB, 5.66 GB, 5.90 GB, 6.01
GB, 6.53 GB, 7.22 GB, 7.43 GB, 7.62 GB, 7.83 GB, 7.99 GB, 8.49 GB,
9.18 GB, 9.38 GB, 9.55 GB, 9.67 GB, 9.88 GB, 9.98 GB, 10.53 GB, 11.04
GB, 11.22 GB, 11.39 GB, 11.50 GB, 11.58 GB, 11.70 GB, 11.95 GB, 12.61
GB, 13.30 GB, 13.43 GB, 13.52 GB, 13.59 GB, 13.65 GB, 13.72 GB, 13.91
GB, 13.94 GB, 13.96 GB, 13.98 GB, 14.03 GB
           - FooterProcessingTime: (Avg: 615.194ms ; Min: 50.813ms ;
Max: 6s740ms ; Number of samples: 58)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 7.80
           - BytesRead: 14.04 GB (15072987184)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 14.04 GB (15072987184)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 2.34K (2340)
           - CachedFileHandlesMissCount: 58 (58)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 21s821ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 16 (16)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 58 (58)
           - NumScannerThreadsStarted: 8 (8)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 1.38 GB (1479283937)
           - PerReadThreadRawHdfsThroughput: 41.17 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 3s555ms
           - RowBatchQueuePutWaitTime: 235h9m
           - RowsRead: 433.04M (433043954)
           - RowsReturned: 433.04M (433043954)
           - RowsReturnedRate: 70.36 M/sec
           - ScanRangesComplete: 58 (58)
           - ScannerThreadsInvoluntaryContextSwitches: 16.45K (16447)
           - ScannerThreadsTotalWallClockTime: 235h15m
             - MaterializeTupleTime(*): 2m5s
             - ScannerThreadsSysTime: 3s924ms
             - ScannerThreadsUserTime: 2m40s
           - ScannerThreadsVoluntaryContextSwitches: 424.08K (424080)
           - TotalRawHdfsOpenFileTime(*): 4s852ms
           - TotalRawHdfsReadTime(*): 5m49s
           - TotalReadThroughput: 135.58 KB/sec
        CodeGen:(Total: 771.273ms, non-child: 771.273ms, % non-child: 100.00%)
           - CodegenInvoluntaryContextSwitches: 5 (5)
           - CodegenTotalWallClockTime: 771.263ms
             - CodegenSysTime: 19.916ms
             - CodegenUserTime: 41.600ms
           - CodegenVoluntaryContextSwitches: 91 (91)
           - CompileTime: 404.632ms
           - IrGenerationTime: 153.195ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 2.26 MB (2373832)
           - NumFunctions: 19 (19)
           - NumInstructions: 706 (706)
           - OptimizationTime: 199.856ms
           - PeakMemoryUsage: 353.00 KB (361472)
           - PrepareTime: 12.885ms
      Instance 9649f7ab3bcc5fb8:f4d6a60700000006
(host=slave6:22000):(Total: 29h59m, non-child: 21s225ms, % non-child:
0.02%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:57/13.92 GB
        Fragment Instance Lifecycle Event Timeline: 29h59m
           - Prepare Finished: 55.600ms (55.600ms)
           - Open Finished: 598.215ms (542.614ms)
           - First Batch Produced: 2s350ms (1s752ms)
           - First Batch Sent: 2s356ms (5.853ms)
           - ExecInternal Finished: 29h59m (29h59m)
        MemoryUsage(34m8s): 1.37 GB, 1.37 GB, 1.36 GB, 1.24 GB, 1.04
GB, 864.64 MB, 1.37 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.31 GB, 1.17 GB,
965.56 MB, 876.58 MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.38 GB, 1.23 GB,
995.12 MB, 871.65 MB, 1.35 GB, 1.40 GB, 1.38 GB, 1.24 GB, 977.74 MB,
850.26 MB, 1.27 GB, 1.29 GB, 1.30 GB, 1.31 GB, 1.16 GB, 954.46 MB,
906.32 MB, 1.24 GB, 1.34 GB, 1.30 GB, 1.31 GB, 1.36 GB, 1.29 GB, 1.02
GB, 1.02 GB, 1.32 GB, 1.30 GB, 1.31 GB, 1.38 GB, 1.33 GB, 1.23 GB,
1.14 GB, 1017.39 MB, 822.59 MB, 631.09 MB
        ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 7
         - AverageThreadTokens: 8.86
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 1.40 GB (1499916960)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 1.40 GB (1499916960)
         - RowsProduced: 428.25M (428247756)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 1m52s
         - TotalThreadsInvoluntaryContextSwitches: 388.30K (388300)
         - TotalThreadsTotalWallClockTime: 265h50m
           - TotalThreadsSysTime: 1m58s
           - TotalThreadsUserTime: 1h13m
         - TotalThreadsVoluntaryContextSwitches: 431.32K (431323)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 29h59m
             - ExecTreeExecTime: 4s703ms
           - OpenTime: 542.606ms
             - ExecTreeOpenTime: 32.022us
           - PrepareTime: 55.377ms
             - ExecTreePrepareTime: 20.617ms
        KuduTableSink:(Total: 29h59m, non-child: 29h59m, % non-child: 100.00%)
           - KuduApplyTimer: 29h
           - NumRowErrors: 346.33M (346333617)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 3.97 K/sec
           - TotalNumRows: 428.25M (428247756)
        HDFS_SCAN_NODE (id=0):(Total: 4s407ms, non-child: 4s407ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:57/13.92 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of 57
          Hdfs Read Thread Concurrency Bucket: 0:99.81% 1:0.1895% 2:0%
3:0% 4:0% 5:0% 6:0%
          File Formats: PARQUET/SNAPPY:57
          BytesRead(34m8s): 1.29 GB, 1.56 GB, 1.71 GB, 1.81 GB, 1.90
GB, 2.18 GB, 3.19 GB, 3.30 GB, 3.46 GB, 3.65 GB, 3.76 GB, 3.84 GB,
3.91 GB, 4.19 GB, 5.09 GB, 5.31 GB, 5.48 GB, 5.64 GB, 5.78 GB, 5.90
GB, 6.16 GB, 7.13 GB, 7.39 GB, 7.61 GB, 7.76 GB, 7.91 GB, 8.15 GB,
9.04 GB, 9.25 GB, 9.49 GB, 9.72 GB, 9.83 GB, 9.95 GB, 10.16 GB, 10.86
GB, 11.12 GB, 11.26 GB, 11.41 GB, 11.60 GB, 11.71 GB, 11.87 GB, 12.39
GB, 13.18 GB, 13.32 GB, 13.42 GB, 13.63 GB, 13.70 GB, 13.76 GB, 13.82
GB, 13.87 GB, 13.89 GB, 13.90 GB
           - FooterProcessingTime: (Avg: 368.827ms ; Min: 61.227ms ;
Max: 3s788ms ; Number of samples: 57)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 7.86
           - BytesRead: 13.92 GB (14949018844)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 13.92 GB (14949018844)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 2.31K (2315)
           - CachedFileHandlesMissCount: 57 (57)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 22s734ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 16 (16)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 57 (57)
           - NumScannerThreadsStarted: 8 (8)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 1.38 GB (1478857724)
           - PerReadThreadRawHdfsThroughput: 69.53 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 1s749ms
           - RowBatchQueuePutWaitTime: 235h46m
           - RowsRead: 428.25M (428247756)
           - RowsReturned: 428.25M (428247756)
           - RowsReturnedRate: 97.15 M/sec
           - ScanRangesComplete: 57 (57)
           - ScannerThreadsInvoluntaryContextSwitches: 15.99K (15992)
           - ScannerThreadsTotalWallClockTime: 235h50m
             - MaterializeTupleTime(*): 2m7s
             - ScannerThreadsSysTime: 3s685ms
             - ScannerThreadsUserTime: 2m42s
           - ScannerThreadsVoluntaryContextSwitches: 419.35K (419351)
           - TotalRawHdfsOpenFileTime(*): 3s380ms
           - TotalRawHdfsReadTime(*): 3m25s
           - TotalReadThroughput: 135.25 KB/sec
        CodeGen:(Total: 405.466ms, non-child: 405.466ms, % non-child: 100.00%)
           - CodegenInvoluntaryContextSwitches: 2 (2)
           - CodegenTotalWallClockTime: 405.458ms
             - CodegenSysTime: 0.000ns
             - CodegenUserTime: 49.456ms
           - CodegenVoluntaryContextSwitches: 91 (91)
           - CompileTime: 170.012ms
           - IrGenerationTime: 52.154ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 2.26 MB (2373832)
           - NumFunctions: 19 (19)
           - NumInstructions: 706 (706)
           - OptimizationTime: 168.310ms
           - PeakMemoryUsage: 353.00 KB (361472)
           - PrepareTime: 14.493ms
      Instance 9649f7ab3bcc5fb8:f4d6a60700000003
(host=slave2:22000):(Total: 28h48m, non-child: 21s449ms, % non-child:
0.02%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
        Fragment Instance Lifecycle Event Timeline: 28h48m
           - Prepare Finished: 13.348ms (13.348ms)
           - Open Finished: 348.443ms (335.094ms)
           - First Batch Produced: 1s505ms (1s157ms)
           - First Batch Sent: 1s508ms (3.334ms)
           - ExecInternal Finished: 28h48m (28h48m)
        MemoryUsage(34m8s): 1.38 GB, 1.38 GB, 1.39 GB, 1.26 GB, 1.06
GB, 777.91 MB, 1.37 GB, 1.38 GB, 1.38 GB, 1.39 GB, 1.33 GB, 1.20 GB,
985.85 MB, 781.77 MB, 1.25 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.25 GB,
1015.91 MB, 756.94 MB, 1.27 GB, 1.37 GB, 1.37 GB, 1.27 GB, 1012.70 MB,
775.19 MB, 1.25 GB, 1.41 GB, 1.41 GB, 1.30 GB, 1.01 GB, 871.44 MB,
1.34 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.37 GB, 1.31 GB, 1.14 GB, 825.57
MB, 1.32 GB, 1.40 GB, 1.40 GB, 1.37 GB, 1.28 GB, 1.17 GB, 1.02 GB,
856.05 MB, 694.00 MB
        ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
         - AverageThreadTokens: 8.94
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 1.41 GB (1512855058)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 1.41 GB (1512855058)
         - RowsProduced: 427.11M (427110203)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 2m27s
         - TotalThreadsInvoluntaryContextSwitches: 375.92K (375918)
         - TotalThreadsTotalWallClockTime: 257h33m
           - TotalThreadsSysTime: 2m12s
           - TotalThreadsUserTime: 1h9m
         - TotalThreadsVoluntaryContextSwitches: 431.39K (431386)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 28h48m
             - ExecTreeExecTime: 4s366ms
           - OpenTime: 335.089ms
             - ExecTreeOpenTime: 40.091us
           - PrepareTime: 13.257ms
             - ExecTreePrepareTime: 12.776ms
        KuduTableSink:(Total: 28h48m, non-child: 28h48m, % non-child: 100.00%)
           - KuduApplyTimer: 27h51m
           - NumRowErrors: 347.46M (347457539)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 4.12 K/sec
           - TotalNumRows: 427.11M (427110203)
        HDFS_SCAN_NODE (id=0):(Total: 4s001ms, non-child: 4s001ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
          Hdfs Read Thread Concurrency Bucket: 0:99.69% 1:0.3106% 2:0%
3:0% 4:0% 5:0% 6:0%
          File Formats: PARQUET/SNAPPY:56
          BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94
GB, 2.08 GB, 3.22 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB,
3.94 GB, 4.08 GB, 5.02 GB, 5.30 GB, 5.48 GB, 5.66 GB, 5.80 GB, 5.91
GB, 6.00 GB, 7.04 GB, 7.37 GB, 7.61 GB, 7.77 GB, 7.90 GB, 8.01 GB,
8.95 GB, 9.39 GB, 9.61 GB, 9.77 GB, 9.88 GB, 10.17 GB, 11.12 GB, 11.26
GB, 11.40 GB, 11.56 GB, 11.66 GB, 11.73 GB, 11.82 GB, 11.97 GB, 13.08
GB, 13.48 GB, 13.60 GB, 13.68 GB, 13.74 GB, 13.80 GB, 13.85 GB, 13.87
GB, 13.88 GB
           - FooterProcessingTime: (Avg: 339.758ms ; Min: 55.620ms ;
Max: 2s575ms ; Number of samples: 56)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 7.94
           - BytesRead: 13.88 GB (14898474484)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 13.88 GB (14898474484)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 2.29K (2294)
           - CachedFileHandlesMissCount: 56 (56)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 23s940ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 16 (16)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 56 (56)
           - NumScannerThreadsStarted: 8 (8)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 1.39 GB (1491692569)
           - PerReadThreadRawHdfsThroughput: 44.62 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 1s156ms
           - RowBatchQueuePutWaitTime: 228h40m
           - RowsRead: 427.11M (427110203)
           - RowsReturned: 427.11M (427110203)
           - RowsReturnedRate: 106.74 M/sec
           - ScanRangesComplete: 56 (56)
           - ScannerThreadsInvoluntaryContextSwitches: 7.41K (7412)
           - ScannerThreadsTotalWallClockTime: 228h45m
             - MaterializeTupleTime(*): 2m
             - ScannerThreadsSysTime: 4s574ms
             - ScannerThreadsUserTime: 2m39s
           - ScannerThreadsVoluntaryContextSwitches: 418.25K (418255)
           - TotalRawHdfsOpenFileTime(*): 4s930ms
           - TotalRawHdfsReadTime(*): 5m18s
           - TotalReadThroughput: 140.35 KB/sec
        CodeGen:(Total: 316.986ms, non-child: 316.986ms, % non-child: 100.00%)
           - CodegenInvoluntaryContextSwitches: 7 (7)
           - CodegenTotalWallClockTime: 316.976ms
             - CodegenSysTime: 19.346ms
             - CodegenUserTime: 35.434ms
           - CodegenVoluntaryContextSwitches: 87 (87)
           - CompileTime: 104.224ms
           - IrGenerationTime: 54.920ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 2.26 MB (2373832)
           - NumFunctions: 19 (19)
           - NumInstructions: 706 (706)
           - OptimizationTime: 143.706ms
           - PeakMemoryUsage: 353.00 KB (361472)
           - PrepareTime: 13.284ms
      Instance 9649f7ab3bcc5fb8:f4d6a60700000005
(host=slave5:22000):(Total: 28h47m, non-child: 30s821ms, % non-child:
0.03%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
        Fragment Instance Lifecycle Event Timeline: 28h47m
           - Prepare Finished: 25.961ms (25.961ms)
           - Open Finished: 92.181ms (66.220ms)
           - First Batch Produced: 1s428ms (1s336ms)
           - First Batch Sent: 1s432ms (3.458ms)
           - ExecInternal Finished: 28h47m (28h47m)
        MemoryUsage(34m8s): 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB, 1.07
GB, 786.07 MB, 1.37 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.31 GB, 1.19 GB,
973.49 MB, 778.55 MB, 1.26 GB, 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB,
1.00 GB, 812.17 MB, 1.29 GB, 1.41 GB, 1.39 GB, 1.14 GB, 881.39 MB,
1.30 GB, 1.37 GB, 1.37 GB, 1.33 GB, 1.16 GB, 910.68 MB, 880.78 MB,
1.32 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.37 GB, 1.29 GB, 1.13 GB, 843.04
MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.35 GB, 1.27 GB, 1.16 GB, 1.00 GB,
848.32 MB, 709.30 MB
        ThreadUsage(34m8s): 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
         - AverageThreadTokens: 8.94
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 1.41 GB (1513059939)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 1.41 GB (1513059939)
         - RowsProduced: 426.83M (426832398)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 1m16s
         - TotalThreadsInvoluntaryContextSwitches: 365.02K (365022)
         - TotalThreadsTotalWallClockTime: 257h18m
           - TotalThreadsSysTime: 1m56s
           - TotalThreadsUserTime: 1h9m
         - TotalThreadsVoluntaryContextSwitches: 431.61K (431609)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 28h47m
             - ExecTreeExecTime: 4s226ms
           - OpenTime: 66.213ms
             - ExecTreeOpenTime: 30.358us
           - PrepareTime: 25.760ms
             - ExecTreePrepareTime: 25.484ms
        KuduTableSink:(Total: 28h46m, non-child: 28h46m, % non-child: 100.00%)
           - KuduApplyTimer: 27h51m
           - NumRowErrors: 345.67M (345674429)
           - PeakMemoryUsage: 20.00 MB (20971520)
           - RowsProcessedRate: 4.12 K/sec
           - TotalNumRows: 426.83M (426832398)
        HDFS_SCAN_NODE (id=0):(Total: 3s937ms, non-child: 3s937ms, %
non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
          ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
          Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.138% 2:0%
3:0% 4:0% 5:0% 6:0%
          File Formats: PARQUET/SNAPPY:56
          BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94
GB, 2.08 GB, 3.23 GB, 3.30 GB, 3.48 GB, 3.67 GB, 3.78 GB, 3.87 GB,
3.94 GB, 4.09 GB, 5.01 GB, 5.31 GB, 5.49 GB, 5.66 GB, 5.81 GB, 5.92
GB, 6.09 GB, 7.05 GB, 7.45 GB, 7.69 GB, 7.86 GB, 8.14 GB, 9.07 GB,
9.35 GB, 9.56 GB, 9.70 GB, 9.81 GB, 9.90 GB, 10.23 GB, 11.09 GB, 11.27
GB, 11.41 GB, 11.56 GB, 11.66 GB, 11.73 GB, 11.82 GB, 12.00 GB, 13.08
GB, 13.48 GB, 13.61 GB, 13.68 GB, 13.74 GB, 13.80 GB, 13.85 GB, 13.87
GB, 13.88 GB
           - FooterProcessingTime: (Avg: 256.473ms ; Min: 49.349ms ;
Max: 2s343ms ; Number of samples: 56)
           - AverageHdfsReadThreadConcurrency: 0.00
           - AverageScannerThreadConcurrency: 7.94
           - BytesRead: 13.88 GB (14898579078)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 13.88 GB (14898579078)
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandlesHitCount: 2.29K (2293)
           - CachedFileHandlesMissCount: 56 (56)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 22s127ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 16 (16)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 56 (56)
           - NumScannerThreadsStarted: 8 (8)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 1.39 GB (1491831945)
           - PerReadThreadRawHdfsThroughput: 100.21 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 1s334ms
           - RowBatchQueuePutWaitTime: 228h26m
           - RowsRead: 426.83M (426832398)
           - RowsReturned: 426.83M (426832398)
           - RowsReturnedRate: 108.39 M/sec
           - ScanRangesComplete: 56 (56)
           - ScannerThreadsInvoluntaryContextSwitches: 14.55K (14553)
           - ScannerThreadsTotalWallClockTime: 228h30m
             - MaterializeTupleTime(*): 2m3s
             - ScannerThreadsSysTime: 4s079ms
             - ScannerThreadsUserTime: 2m39s
           - ScannerThreadsVoluntaryContextSwitches: 417.98K (417975)
           - TotalRawHdfsOpenFileTime(*): 3s114ms
           - TotalRawHdfsReadTime(*): 2m21s
           - TotalReadThroughput: 140.45 KB/sec
        CodeGen:(Total: 56.104ms, non-child: 56.104ms, % non-child: 100.00%)
           - CodegenInvoluntaryContextSwitches: 3 (3)
           - CodegenTotalWallClockTime: 56.095ms
             - CodegenSysTime: 0.000ns
             - CodegenUserTime: 37.520ms
           - CodegenVoluntaryContextSwitches: 1 (1)
           - CompileTime: 23.495ms
           - IrGenerationTime: 1.462ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 2.26 MB (2373832)
           - NumFunctions: 19 (19)
           - NumInstructions: 706 (706)
           - OptimizationTime: 15.936ms
           - PeakMemoryUsage: 353.00 KB (361472)
           - PrepareTime: 14.604ms



On Thu, May 10, 2018 at 10:33 AM, Mostafa Mokhtar <mmokhtar@cloudera.com>
wrote:

> Can you share the query profile for the successful insert query?
>
> Thanks
> Mostafa
>
> On May 9, 2018, at 9:55 PM, Geetika Gupta <geetika.gupta@knoldus.in>
> wrote:
>
> Thanks, Jeszy.
>
> We build impala again with --release flag and data load was successful
> after that.
>
> But now we are facing another issue. The table in which we loaded the data
> has less number of rows. We executed the following command:
>
> insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
>
> This query was successful, but when we tried the count(*) on both the
> tables, row count was different:
>
> 0: jdbc:hive2://slave2:21050/default> select count(*) from lineitem
> . . . . . . . . . . . . . . . . . . > ;
> 536870912
>
> 0: jdbc:hive2://slave2:21050/default> select count(*) from
> parquetimpala500.lineitem;
> 3000028242
>
> Do you have any idea about this issue.
>
>
> On Mon, May 7, 2018 at 12:06 PM, Jeszy <jeszyb@gmail.com> wrote:
>
>> Impala doesn't store the data itself, so you can switch versions
>> without rewriting data. But you don't have to do that, you would just
>> have to build impala using the -release flag (of buildall.sh) and run
>> it using the release binaries (versus the debug ones). If you would be
>> looking at performance, using the release version is highly
>> recommended anyway.
>>
>> On 7 May 2018 at 08:30, Geetika Gupta <geetika.gupta@knoldus.in> wrote:
>> > Hi Jeszy,
>> >
>> > Currently, we are using the apache impala's Github master branch code.
>> We
>> > tried using the released version but we encountered some errors related
>> to
>> > downloading of dependencies and could not complete the installation.
>> >
>> > The current version of impala we are using: 2.12
>> >
>> > We can't try with the new release as we have already loaded 500GB of
>> TPCH
>> > data on our cluster.
>> >
>> > On Mon, May 7, 2018 at 11:43 AM, Jeszy <jeszyb@gmail.com> wrote:
>> >>
>> >> What version of Impala are you using?
>> >> DCHECKs won't be triggered if you run a release build. Looking at the
>> >> code, it should work with bad values if not for the DCHECK. Can you
>> >> try using a release build?
>> >>
>> >> On 7 May 2018 at 08:04, Geetika Gupta <geetika.gupta@knoldus.in>
>> wrote:
>> >> > Hi community,
>> >> >
>> >> > I was trying to load 500GB of TPCH data into kudu table using the
>> >> > following
>> >> > query:
>> >> >
>> >> > insert into lineitem select * from PARQUETIMPALA500.LINEITEM
>> >> >
>> >> > While executing the query for around 17 hrs it got cancelled as the
>> >> > impalad
>> >> > process of that machine got aborted. Here are the logs of the impalad
>> >> > process.
>> >> >
>> >> > impalad.ERROR
>> >> >
>> >> > Log file created at: 2018/05/06 13:40:34
>> >> > Running on machine: slave2
>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>> >> > E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to
>> >> > this
>> >> > file.
>> >> > SLF4J: Class path contains multiple SLF4J bindings.
>> >> > SLF4J: Found binding in
>> >> >
>> >> > [jar:file:/root/softwares/impala/fe/target/dependency/slf4j-
>> log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> >> > SLF4J: Found binding in
>> >> >
>> >> > [jar:file:/root/softwares/impala/testdata/target/dependency/
>> slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> >> > explanation.
>> >> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>> >> > 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load
>> >> > native-hadoop
>> >> > library for your platform... using builtin-java classes where
>> applicable
>> >> > 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
>> >> > short-circuit
>> >> > local reads feature cannot be used because libhadoop cannot be
>> loaded.
>> >> > tcmalloc: large alloc 1073741824 bytes == 0x484434000 @  0x4135176
>> >> > 0x7fd9e9fc3929
>> >> > tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @  0x4135176
>> >> > 0x7fd9e9fc3929
>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> >> > log_entry.count
>> >> >> 0 (-1831809966 vs. 0)
>> >> > *** Check failure stack trace: ***
>> >> >     @          0x3fc0c0d  google::LogMessage::Fail()
>> >> >     @          0x3fc24b2  google::LogMessage::SendToLog()
>> >> >     @          0x3fc05e7  google::LogMessage::Flush()
>> >> >     @          0x3fc3bae  google::LogMessageFatal::~LogM
>> essageFatal()
>> >> >     @          0x1bbcb31  impala::PrintErrorMap()
>> >> >     @          0x1bbcd07  impala::PrintErrorMapToString()
>> >> >     @          0x2decbd7  impala::Coordinator::GetErrorLog()
>> >> >     @          0x1a8d634  impala::ImpalaServer::UnregisterQuery()
>> >> >     @          0x1b29264  impala::ImpalaServer::CloseOperation()
>> >> >     @          0x2c5ce86
>> >> >
>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::pr
>> ocess_CloseOperation()
>> >> >     @          0x2c56b8c
>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::di
>> spatchCall()
>> >> >     @          0x2c2fcb1
>> >> > impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
>> >> >     @          0x16fdb20  apache::thrift::TDispatchProce
>> ssor::process()
>> >> >     @          0x18ea6b3
>> >> > apache::thrift::server::TAcceptQueueServer::Task::run()
>> >> >     @          0x18e2181  impala::ThriftThread::RunRunnable()
>> >> >     @          0x18e3885  boost::_mfi::mf2<>::operator()()
>> >> >     @          0x18e371b  boost::_bi::list3<>::operator()<>()
>> >> >     @          0x18e3467  boost::_bi::bind_t<>::operator()()
>> >> >     @          0x18e337a
>> >> > boost::detail::function::void_function_obj_invoker0<>::invoke()
>> >> >     @          0x192761c  boost::function0<>::operator()()
>> >> >     @          0x1c3ebf7  impala::Thread::SuperviseThread()
>> >> >     @          0x1c470cd  boost::_bi::list5<>::operator()<>()
>> >> >     @          0x1c46ff1  boost::_bi::bind_t<>::operator()()
>> >> >     @          0x1c46fb4  boost::detail::thread_data<>::run()
>> >> >     @          0x2eedb4a  thread_proxy
>> >> >     @     0x7fda1dbb16ba  start_thread
>> >> >     @     0x7fda1d8e741d  clone
>> >> > Wrote minidump to
>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>> >> >
>> >> > impalad.FATAL
>> >> >
>> >> > Log file created at: 2018/05/07 09:46:12
>> >> > Running on machine: slave2
>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> >> > log_entry.count
>> >> >> 0 (-1831809966 vs. 0)
>> >> >
>> >> > Impalad.INFO
>> >> > edentials={real_user=root}} blocked reactor thread for 34288.6us
>> >> > I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for
>> RPC
>> >> > call
>> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>> for
>> >> > 35859.8us
>> >> > I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for
>> RPC
>> >> > call
>> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>> for
>> >> > 40664.9us
>> >> > I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for
>> RPC
>> >> > call
>> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>> for
>> >> > 49514.6us
>> >> > I0507 09:46:12.664149  4507 coordinator.cc:783] Release admission
>> >> > control
>> >> > resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> >> > log_entry.count
>> >> >> 0 (-1831809966 vs. 0)
>> >> > Wrote minidump to
>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>> >> >
>> >> > Note:
>> >> > We are executing the queries on 8 node cluster with the following
>> >> > configuration
>> >> > Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk
>> >> > each,
>> >> > Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>> >> >
>> >> >
>> >> > --
>> >> > Regards,
>> >> > Geetika Gupta
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Geetika Gupta
>>
>
>
>
> --
> Regards,
> Geetika Gupta
>
>


-- 
Regards,
Geetika Gupta

Mime
View raw message