impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Jacobs <jacobs...@gmail.com>
Subject Re: impala submission control
Date Thu, 08 Feb 2018 17:00:47 GMT
As far as I can tell from this information, your configuration looks
fine. The only other potential issue I can think of is that the
secondary groups placement rule isn't picking up your LDAP groups
properly. Does the ACL behavior work if you remove the secondary
groups rule?

On Wed, Feb 7, 2018 at 10:09 PM, Fawze Abujaber <fawzeaj@gmail.com> wrote:
> Hi Matt,
>
> v2.11.0-cdh5.14.0
>
> Query Runtime Profile:
> Query (id=c843dae48ce9b289:6ad3ffc000000000):
>   Summary:
>     Session ID: ca420ac477772b56:51ae460ce9e44999
>     Session Type: BEESWAX
>     Start Time: 2018-02-08 01:02:01.023558000
>     End Time: 2018-02-08 01:02:01.501506000
>     Query Type: QUERY
>     Query State: FINISHED
>     Query Status: OK
>     Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE (build
> d68206561bce6b26762d62c01a78e6cd27aa7690)
>     User: fawzea
>     Connected User: fawzea
>     Delegated User:
>     Network Address: ::ffff:10.16.144.131:50078
>     Default Db: default
>     Sql Statement: select count(*) from dv_entersiteevent_prq_remote where
> year=2017 and month=10
>     Coordinator: aopr-dhc001.lpdomain.com:22000
>     Query Options (set by configuration): REQUEST_POOL=impreports
>     Query Options (set by configuration and planner):
> REQUEST_POOL=impreports,MT_DOP=0
>     Plan:
> ----------------
> Max Per-Host Resource Reservation: Memory=0B
> Per-Host Resource Estimates: Memory=21.00MB
> WARNING: The following tables are missing relevant table and/or column
> statistics.
> default.dv_entersiteevent_prq_remote
>
> F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> |  Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
> PLAN-ROOT SINK
> |  mem-estimate=0B mem-reservation=0B
> |
> 03:AGGREGATE [FINALIZE]
> |  output: count:merge(*)
> |  mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
> |  tuple-ids=1 row-size=8B cardinality=1
> |
> 02:EXCHANGE [UNPARTITIONED]
> |  mem-estimate=0B mem-reservation=0B
> |  tuple-ids=1 row-size=8B cardinality=1
> |
> F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
> Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
> 01:AGGREGATE
> |  output: sum_init_zero(default.dv_entersiteevent_prq_remote.parquet-stats:
> num_rows)
> |  mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
> |  tuple-ids=1 row-size=8B cardinality=1
> |
> 00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
>    partitions=31/441 files=2292 size=1.39GB
>    stats-rows=unavailable extrapolated-rows=disabled
>    table stats: rows=unavailable size=unavailable
>    column stats: all
>    mem-estimate=1.00MB mem-reservation=0B
>    tuple-ids=0 row-size=8B cardinality=unavailable
> ----------------
>     Estimated Per-Host Mem: 22020096
>     Tables Missing Stats: default.dv_entersiteevent_prq_remote
>     Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0)
> aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000(0)
> aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000(0)
>     Request Pool: root.impreports
>     Admission result: Admitted immediately
>     ExecSummary:
> Operator       #Hosts   Avg Time   Max Time  #Rows  Est. #Rows  Peak Mem
> Est. Peak Mem  Detail
> ------------------------------------------------------------------------------------------------------------------------------
> 03:AGGREGATE        1  222.958us  222.958us      1           1  36.00 KB
> 10.00 MB  FINALIZE
> 02:EXCHANGE         1  357.508ms  357.508ms      5           1         0
> 0  UNPARTITIONED
> 01:AGGREGATE        5    1.027ms    1.541ms      5           1  20.00 KB
> 10.00 MB
> 00:SCAN HDFS        5  282.797ms  333.715ms  2.29K          -1   2.62 MB
> 1.00 MB  default.dv_entersiteevent_prq_remote
>     Errors:
>     Planner Timeline: 13.508ms
>        - Analysis finished: 3.303ms (3.303ms)
>        - Value transfer graph computed: 3.465ms (162.667us)
>        - Single node plan created: 10.352ms (6.886ms)
>        - Runtime filters computed: 10.677ms (325.364us)
>        - Distributed plan created: 10.733ms (55.854us)
>        - Lineage info computed: 10.898ms (164.937us)
>        - Planning finished: 13.508ms (2.610ms)
>     Query Timeline: 481.950ms
>        - Query submitted: 79.971us (79.971us)
>        - Planning finished: 33.947ms (33.867ms)
>        - Submit for admission: 40.067ms (6.120ms)
>        - Completed admission: 40.197ms (130.146us)
>        - Ready to start on 5 backends: 40.905ms (707.852us)
>        - All 5 execution backends (6 fragment instances) started: 46.417ms
> (5.512ms)
>        - Rows available: 424.548ms (378.131ms)
>        - First row fetched: 473.488ms (48.939ms)
>        - Unregister query: 477.951ms (4.462ms)
>      - ComputeScanRangeAssignmentTimer: 3.980ms
>   ImpalaServer:
>      - ClientFetchWaitTimer: 51.659ms
>      - RowMaterializationTimer: 1.726ms
>   Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total: 385.029ms,
> non-child: 0.000ns, % non-child: 0.00%)
>     Number of filters: 0
>     Filter routing table:
>  ID  Src. Node  Tgt. Node(s)  Target type  Partition filter  Pending
> (Expected)  First arrived  Completed   Enabled
> -------------------------------------------------------------------------------------------------------------------
>
>     Backend startup latencies: Count: 5, min / max: 3ms / 4ms, 25th %-ile:
> 3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th %-ile: 4ms,
> 99.9th %-ile: 4ms
>     Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68 MB)
> aopr-dhc002.lpdomain.com:22000(2.54 MB) aopr-dhc003.lpdomain.com:22000(2.54
> MB) aopr-dhc004.lpdomain.com:22000(2.41 MB)
> aopr-dhc005.lpdomain.com:22000(2.56 MB)
>      - FiltersReceived: 0 (0)
>      - FinalizationTimer: 0.000ns
>      - NumBackends: 5 (5)
>      - NumFragmentInstances: 6 (6)
>      - NumFragments: 2 (2)
>     Averaged Fragment F01:(Total: 427.530ms, non-child: 49.428ms, %
> non-child: 11.56%)
>       split sizes:  min: 0, max: 0, avg: 0, stddev: 0
>       completion times: min:428.693ms  max:428.693ms  mean: 428.693ms
> stddev:0.000ns
>       execution rates: min:0.00 /sec  max:0.00 /sec  mean:0.00 /sec
> stddev:0.00 /sec
>       num instances: 1
>        - AverageThreadTokens: 0.00
>        - BloomFilterBytes: 0
>        - PeakMemoryUsage: 44.02 KB (45072)
>        - PeakReservation: 0
>        - PeakUsedReservation: 0
>        - PerHostPeakMemUsage: 2.68 MB (2806113)
>        - RowsProduced: 1 (1)
>        - TotalNetworkReceiveTime: 357.481ms
>        - TotalNetworkSendTime: 0.000ns
>        - TotalStorageWaitTime: 0.000ns
>        - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>        - TotalThreadsTotalWallClockTime: 406.708ms
>          - TotalThreadsSysTime: 0.000ns
>          - TotalThreadsUserTime: 0.000ns
>        - TotalThreadsVoluntaryContextSwitches: 7 (7)
>       Fragment Instance Lifecycle Timings:
>          - ExecTime: 49.055ms
>            - ExecTreeExecTime: 4.251us
>          - OpenTime: 357.663ms
>            - ExecTreeOpenTime: 357.649ms
>          - PrepareTime: 20.764ms
>            - ExecTreePrepareTime: 81.759us
>       PLAN_ROOT_SINK:
>          - PeakMemoryUsage: 0
>       CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child: 100.00%)
>          - CodegenTime: 0.000ns
>          - CompileTime: 0.000ns
>          - LoadTime: 0.000ns
>          - ModuleBitcodeSize: 1.86 MB (1950880)
>          - NumFunctions: 0 (0)
>          - NumInstructions: 0 (0)
>          - OptimizationTime: 0.000ns
>          - PeakMemoryUsage: 0
>          - PrepareTime: 19.685ms
>       AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us, %
> non-child: 0.06%)
>          - BuildTime: 7.368us
>          - GetResultsTime: 0.000ns
>          - HTResizeTime: 0.000ns
>          - HashBuckets: 0 (0)
>          - LargestPartitionPercent: 0 (0)
>          - MaxPartitionLevel: 0 (0)
>          - NumRepartitions: 0 (0)
>          - PartitionsCreated: 0 (0)
>          - PeakMemoryUsage: 36.00 KB (36864)
>          - RowsRepartitioned: 0 (0)
>          - RowsReturned: 1 (1)
>          - RowsReturnedRate: 2.00 /sec
>          - SpilledPartitions: 0 (0)
>       EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
> non-child: 100.00%)
>          - ConvertRowBatchTime: 8.881us
>          - PeakMemoryUsage: 0
>          - RowsReturned: 5 (5)
>          - RowsReturnedRate: 13.00 /sec
>         DataStreamReceiver:
>            - BytesReceived: 80.00 B (80)
>            - DeserializeRowBatchTimer: 24.165us
>            - FirstBatchArrivalWaitTime: 277.194ms
>            - PeakMemoryUsage: 4.02 KB (4112)
>            - SendersBlockedTimer: 0.000ns
>            - SendersBlockedTotalTimer(*): 0.000ns
>     Coordinator Fragment F01:
>       Instance c843dae48ce9b289:6ad3ffc000000000
> (host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms, non-child:
> 49.428ms, % non-child: 11.56%)
>          - AverageThreadTokens: 0.00
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 44.02 KB (45072)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 2.68 MB (2806113)
>          - RowsProduced: 1 (1)
>          - TotalNetworkReceiveTime: 357.481ms
>          - TotalNetworkSendTime: 0.000ns
>          - TotalStorageWaitTime: 0.000ns
>          - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>          - TotalThreadsTotalWallClockTime: 406.708ms
>            - TotalThreadsSysTime: 0.000ns
>            - TotalThreadsUserTime: 0.000ns
>          - TotalThreadsVoluntaryContextSwitches: 7 (7)
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 49.055ms
>              - ExecTreeExecTime: 4.251us
>            - OpenTime: 357.663ms
>              - ExecTreeOpenTime: 357.649ms
>            - PrepareTime: 20.764ms
>              - ExecTreePrepareTime: 81.759us
>         PLAN_ROOT_SINK:
>            - PeakMemoryUsage: 0
>         CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child: 100.00%)
>            - CodegenTime: 0.000ns
>            - CompileTime: 0.000ns
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 1.86 MB (1950880)
>            - NumFunctions: 0 (0)
>            - NumInstructions: 0 (0)
>            - OptimizationTime: 0.000ns
>            - PeakMemoryUsage: 0
>            - PrepareTime: 19.685ms
>         AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us, %
> non-child: 0.06%)
>            - BuildTime: 7.368us
>            - GetResultsTime: 0.000ns
>            - HTResizeTime: 0.000ns
>            - HashBuckets: 0 (0)
>            - LargestPartitionPercent: 0 (0)
>            - MaxPartitionLevel: 0 (0)
>            - NumRepartitions: 0 (0)
>            - PartitionsCreated: 0 (0)
>            - PeakMemoryUsage: 36.00 KB (36864)
>            - RowsRepartitioned: 0 (0)
>            - RowsReturned: 1 (1)
>            - RowsReturnedRate: 2.00 /sec
>            - SpilledPartitions: 0 (0)
>         EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
> non-child: 100.00%)
>            - ConvertRowBatchTime: 8.881us
>            - PeakMemoryUsage: 0
>            - RowsReturned: 5 (5)
>            - RowsReturnedRate: 13.00 /sec
>           DataStreamReceiver:
>              - BytesReceived: 80.00 B (80)
>              - DeserializeRowBatchTimer: 24.165us
>              - FirstBatchArrivalWaitTime: 277.194ms
>              - PeakMemoryUsage: 4.02 KB (4112)
>              - SendersBlockedTimer: 0.000ns
>              - SendersBlockedTotalTimer(*): 0.000ns
>     Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms, %
> non-child: 0.48%)
>       split sizes:  min: 283.31 MB, max: 284.33 MB, avg: 284.04 MB, stddev:
> 382.00 KB
>       completion times: min:299.457ms  max:378.950ms  mean: 328.881ms
> stddev:31.507ms
>       execution rates: min:750.18 MB/sec  max:949.01 MB/sec  mean:871.27
> MB/sec  stddev:79.74 MB/sec
>       num instances: 5
>        - AverageThreadTokens: 15.00
>        - BloomFilterBytes: 0
>        - PeakMemoryUsage: 2.54 MB (2666852)
>        - PeakReservation: 0
>        - PeakUsedReservation: 0
>        - PerHostPeakMemUsage: 2.55 MB (2669309)
>        - RowsProduced: 1 (1)
>        - TotalNetworkReceiveTime: 0.000ns
>        - TotalNetworkSendTime: 303.541us
>        - TotalStorageWaitTime: 4s821ms
>        - TotalThreadsInvoluntaryContextSwitches: 104 (104)
>        - TotalThreadsTotalWallClockTime: 6s835ms
>          - TotalThreadsSysTime: 31.784ms
>          - TotalThreadsUserTime: 682.084ms
>        - TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
>       Fragment Instance Lifecycle Timings:
>          - ExecTime: 500.816us
>            - ExecTreeExecTime: 8.200us
>          - OpenTime: 302.848ms
>            - ExecTreeOpenTime: 281.309ms
>          - PrepareTime: 23.946ms
>            - ExecTreePrepareTime: 2.513ms
>       DataStreamSender (dst_id=2):(Total: 269.902us, non-child: 269.902us, %
> non-child: 100.00%)
>          - BytesSent: 16.00 B (16)
>          - NetworkThroughput(*): 373.13 KB/sec
>          - OverallThroughput: 63.53 KB/sec
>          - PeakMemoryUsage: 16.00 KB (16384)
>          - RowsReturned: 1 (1)
>          - SerializeBatchTime: 14.525us
>          - TransmitDataRPCTime: 43.122us
>          - UncompressedRowBatchSize: 16.00 B (16)
>       CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child: 100.00%)
>          - CodegenTime: 1.194ms
>          - CompileTime: 7.360ms
>          - LoadTime: 0.000ns
>          - ModuleBitcodeSize: 1.86 MB (1950880)
>          - NumFunctions: 15 (15)
>          - NumInstructions: 214 (214)
>          - OptimizationTime: 13.529ms
>          - PeakMemoryUsage: 107.00 KB (109568)
>          - PrepareTime: 20.084ms
>       AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child: 1.027ms, %
> non-child: 0.36%)
>          - BuildTime: 1.259ms
>          - GetResultsTime: 0.000ns
>          - HTResizeTime: 0.000ns
>          - HashBuckets: 0 (0)
>          - LargestPartitionPercent: 0 (0)
>          - MaxPartitionLevel: 0 (0)
>          - NumRepartitions: 0 (0)
>          - PartitionsCreated: 0 (0)
>          - PeakMemoryUsage: 20.00 KB (20480)
>          - RowsRepartitioned: 0 (0)
>          - RowsReturned: 1 (1)
>          - RowsReturnedRate: 2.00 /sec
>          - SpilledPartitions: 0 (0)
>       HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child: 282.797ms, %
> non-child: 100.00%)
>          - AverageHdfsReadThreadConcurrency: 0.00
>          - AverageScannerThreadConcurrency: 14.40
>          - BytesRead: 44.52 MB (46686028)
>          - BytesReadDataNodeCache: 0
>          - BytesReadLocal: 44.52 MB (46686028)
>          - BytesReadRemoteUnexpected: 0
>          - BytesReadShortCircuit: 44.52 MB (46686028)
>          - CachedFileHandlesHitCount: 0 (0)
>          - CachedFileHandlesMissCount: 458 (458)
>          - CollectionItemsRead: 0 (0)
>          - DecompressionTime: 0.000ns
>          - MaxCompressedTextFileLength: 0
>          - NumColumns: 0 (0)
>          - NumDictFilteredRowGroups: 0 (0)
>          - NumDisksAccessed: 12 (12)
>          - NumRowGroups: 458 (458)
>          - NumScannerThreadsStarted: 24 (24)
>          - NumScannersWithNoReads: 0 (0)
>          - NumStatsFilteredRowGroups: 0 (0)
>          - PeakMemoryUsage: 2.50 MB (2625539)
>          - PerReadThreadRawHdfsThroughput: 1.29 GB/sec
>          - RemoteScanRanges: 0 (0)
>          - RowBatchQueueGetWaitTime: 269.787ms
>          - RowBatchQueuePutWaitTime: 0.000ns
>          - RowsRead: 0 (0)
>          - RowsReturned: 458 (458)
>          - RowsReturnedRate: 1.64 K/sec
>          - ScanRangesComplete: 458 (458)
>          - ScannerThreadsInvoluntaryContextSwitches: 99 (99)
>          - ScannerThreadsTotalWallClockTime: 6s532ms
>            - MaterializeTupleTime(*): 70.664us
>            - ScannerThreadsSysTime: 26.184ms
>            - ScannerThreadsUserTime: 650.089ms
>          - ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
>          - TotalRawHdfsReadTime(*): 34.416ms
>          - TotalReadThroughput: 36.86 MB/sec
>     Fragment F00:
>       Instance c843dae48ce9b289:6ad3ffc000000004
> (host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms, non-child: 1.619ms,
> % non-child: 0.43%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
> 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
> 4:42/26.73 MB 8:34/21.25 MB
>         MemoryUsage(500.000ms): 1.91 MB
>         ThreadUsage(500.000ms): 25
>          - AverageThreadTokens: 25.00
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 2.41 MB (2531689)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 2.41 MB (2531689)
>          - RowsProduced: 1 (1)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 382.629us
>          - TotalStorageWaitTime: 5s790ms
>          - TotalThreadsInvoluntaryContextSwitches: 115 (115)
>          - TotalThreadsTotalWallClockTime: 8s122ms
>            - TotalThreadsSysTime: 33.984ms
>            - TotalThreadsUserTime: 702.879ms
>          - TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 604.005us
>              - ExecTreeExecTime: 8.338us
>            - OpenTime: 352.927ms
>              - ExecTreeOpenTime: 331.921ms
>            - PrepareTime: 23.783ms
>              - ExecTreePrepareTime: 2.559ms
>         DataStreamSender (dst_id=2):(Total: 331.818us, non-child: 331.818us,
> % non-child: 100.00%)
>            - BytesSent: 16.00 B (16)
>            - NetworkThroughput(*): 289.00 KB/sec
>            - OverallThroughput: 47.09 KB/sec
>            - PeakMemoryUsage: 16.00 KB (16384)
>            - RowsReturned: 1 (1)
>            - SerializeBatchTime: 14.466us
>            - TransmitDataRPCTime: 54.066us
>            - UncompressedRowBatchSize: 16.00 B (16)
>         CodeGen:(Total: 40.950ms, non-child: 40.950ms, % non-child: 100.00%)
>            - CodegenTime: 1.122ms
>            - CompileTime: 7.318ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 1.86 MB (1950880)
>            - NumFunctions: 15 (15)
>            - NumInstructions: 214 (214)
>            - OptimizationTime: 13.106ms
>            - PeakMemoryUsage: 107.00 KB (109568)
>            - PrepareTime: 19.773ms
>         AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child: 766.527us, %
> non-child: 0.23%)
>           ExecOption: Codegen Enabled
>            - BuildTime: 1.192ms
>            - GetResultsTime: 0.000ns
>            - HTResizeTime: 0.000ns
>            - HashBuckets: 0 (0)
>            - LargestPartitionPercent: 0 (0)
>            - MaxPartitionLevel: 0 (0)
>            - NumRepartitions: 0 (0)
>            - PartitionsCreated: 0 (0)
>            - PeakMemoryUsage: 20.00 KB (20480)
>            - RowsRepartitioned: 0 (0)
>            - RowsReturned: 1 (1)
>            - RowsReturnedRate: 2.00 /sec
>            - SpilledPartitions: 0 (0)
>         HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child: 333.715ms, %
> non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
> 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
> 4:42/26.73 MB 8:34/21.25 MB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 464 out of
> 464
>           Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>           File Formats: PARQUET/Unknown(Skipped):464
>           BytesRead(500.000ms): 12.27 MB
>            - FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms ; Max:
> 62.957ms ; Number of samples: 464)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 24.00
>            - BytesRead: 45.01 MB (47193988)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 45.01 MB (47193988)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 45.01 MB (47193988)
>            - CachedFileHandlesHitCount: 0 (0)
>            - CachedFileHandlesMissCount: 464 (464)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 0.000ns
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 0 (0)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 12 (12)
>            - NumRowGroups: 464 (464)
>            - NumScannerThreadsStarted: 24 (24)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 2.38 MB (2490376)
>            - PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 320.860ms
>            - RowBatchQueuePutWaitTime: 0.000ns
>            - RowsRead: 0 (0)
>            - RowsReturned: 464 (464)
>            - RowsReturnedRate: 1.39 K/sec
>            - ScanRangesComplete: 464 (464)
>            - ScannerThreadsInvoluntaryContextSwitches: 108 (108)
>            - ScannerThreadsTotalWallClockTime: 7s769ms
>              - MaterializeTupleTime(*): 75.514us
>              - ScannerThreadsSysTime: 29.984ms
>              - ScannerThreadsUserTime: 669.884ms
>            - ScannerThreadsVoluntaryContextSwitches: 1.60K (1597)
>            - TotalRawHdfsReadTime(*): 44.279ms
>            - TotalReadThroughput: 24.53 MB/sec
>       Instance c843dae48ce9b289:6ad3ffc000000002
> (host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms, non-child: 1.641ms,
> % non-child: 0.47%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
> 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
> 4:60/37.59 MB 1:44/27.19 MB
>         MemoryUsage(500.000ms): 1.16 MB
>         ThreadUsage(500.000ms): 25
>          - AverageThreadTokens: 25.00
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 2.54 MB (2662753)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 2.54 MB (2662753)
>          - RowsProduced: 1 (1)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 342.746us
>          - TotalStorageWaitTime: 4s796ms
>          - TotalThreadsInvoluntaryContextSwitches: 93 (93)
>          - TotalThreadsTotalWallClockTime: 7s286ms
>            - TotalThreadsSysTime: 27.988ms
>            - TotalThreadsUserTime: 671.887ms
>          - TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 524.006us
>              - ExecTreeExecTime: 7.920us
>            - OpenTime: 327.802ms
>              - ExecTreeOpenTime: 307.154ms
>            - PrepareTime: 22.617ms
>              - ExecTreePrepareTime: 2.650ms
>         DataStreamSender (dst_id=2):(Total: 348.990us, non-child: 348.990us,
> % non-child: 100.00%)
>            - BytesSent: 16.00 B (16)
>            - NetworkThroughput(*): 354.47 KB/sec
>            - OverallThroughput: 44.77 KB/sec
>            - PeakMemoryUsage: 16.00 KB (16384)
>            - RowsReturned: 1 (1)
>            - SerializeBatchTime: 15.602us
>            - TransmitDataRPCTime: 44.080us
>            - UncompressedRowBatchSize: 16.00 B (16)
>         CodeGen:(Total: 39.215ms, non-child: 39.215ms, % non-child: 100.00%)
>            - CodegenTime: 1.183ms
>            - CompileTime: 7.168ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 1.86 MB (1950880)
>            - NumFunctions: 15 (15)
>            - NumInstructions: 214 (214)
>            - OptimizationTime: 12.929ms
>            - PeakMemoryUsage: 107.00 KB (109568)
>            - PrepareTime: 18.453ms
>         AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child: 948.677us, %
> non-child: 0.31%)
>           ExecOption: Codegen Enabled
>            - BuildTime: 1.552ms
>            - GetResultsTime: 0.000ns
>            - HTResizeTime: 0.000ns
>            - HashBuckets: 0 (0)
>            - LargestPartitionPercent: 0 (0)
>            - MaxPartitionLevel: 0 (0)
>            - NumRepartitions: 0 (0)
>            - PartitionsCreated: 0 (0)
>            - PeakMemoryUsage: 20.00 KB (20480)
>            - RowsRepartitioned: 0 (0)
>            - RowsReturned: 1 (1)
>            - RowsReturnedRate: 3.00 /sec
>            - SpilledPartitions: 0 (0)
>         HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child: 308.859ms, %
> non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
> 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
> 4:60/37.59 MB 1:44/27.19 MB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 458 out of
> 458
>           Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>           File Formats: PARQUET/Unknown(Skipped):458
>           BytesRead(500.000ms): 36.74 MB
>            - FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms ; Max:
> 23.564ms ; Number of samples: 458)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 24.00
>            - BytesRead: 44.51 MB (46668432)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 44.51 MB (46668432)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 44.51 MB (46668432)
>            - CachedFileHandlesHitCount: 0 (0)
>            - CachedFileHandlesMissCount: 458 (458)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 0.000ns
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 0 (0)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 12 (12)
>            - NumRowGroups: 458 (458)
>            - NumScannerThreadsStarted: 24 (24)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 2.50 MB (2621440)
>            - PerReadThreadRawHdfsThroughput: 1.26 GB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 295.595ms
>            - RowBatchQueuePutWaitTime: 0.000ns
>            - RowsRead: 0 (0)
>            - RowsReturned: 458 (458)
>            - RowsReturnedRate: 1.48 K/sec
>            - ScanRangesComplete: 458 (458)
>            - ScannerThreadsInvoluntaryContextSwitches: 91 (91)
>            - ScannerThreadsTotalWallClockTime: 6s958ms
>              - MaterializeTupleTime(*): 71.630us
>              - ScannerThreadsSysTime: 16.989ms
>              - ScannerThreadsUserTime: 645.891ms
>            - ScannerThreadsVoluntaryContextSwitches: 1.63K (1630)
>            - TotalRawHdfsReadTime(*): 34.377ms
>            - TotalReadThroughput: 73.49 MB/sec
>       Instance c843dae48ce9b289:6ad3ffc000000001
> (host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms, non-child: 1.784ms,
> % non-child: 0.58%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
> 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
> 4:40/24.29 MB 11:33/20.77 MB
>          - AverageThreadTokens: 0.00
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 2.56 MB (2683241)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 2.56 MB (2683241)
>          - RowsProduced: 1 (1)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 292.765us
>          - TotalStorageWaitTime: 4s482ms
>          - TotalThreadsInvoluntaryContextSwitches: 113 (113)
>          - TotalThreadsTotalWallClockTime: 6s310ms
>            - TotalThreadsSysTime: 35.983ms
>            - TotalThreadsUserTime: 682.884ms
>          - TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 532.572us
>              - ExecTreeExecTime: 8.377us
>            - OpenTime: 280.241ms
>              - ExecTreeOpenTime: 258.652ms
>            - PrepareTime: 27.342ms
>              - ExecTreePrepareTime: 2.863ms
>         DataStreamSender (dst_id=2):(Total: 309.794us, non-child: 309.794us,
> % non-child: 100.00%)
>            - BytesSent: 16.00 B (16)
>            - NetworkThroughput(*): 356.91 KB/sec
>            - OverallThroughput: 50.44 KB/sec
>            - PeakMemoryUsage: 16.00 KB (16384)
>            - RowsReturned: 1 (1)
>            - SerializeBatchTime: 14.262us
>            - TransmitDataRPCTime: 43.778us
>            - UncompressedRowBatchSize: 16.00 B (16)
>         CodeGen:(Total: 44.584ms, non-child: 44.584ms, % non-child: 100.00%)
>            - CodegenTime: 1.382ms
>            - CompileTime: 7.346ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 1.86 MB (1950880)
>            - NumFunctions: 15 (15)
>            - NumInstructions: 214 (214)
>            - OptimizationTime: 13.462ms
>            - PeakMemoryUsage: 107.00 KB (109568)
>            - PrepareTime: 23.006ms
>         AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child: 836.656us, %
> non-child: 0.32%)
>           ExecOption: Codegen Enabled
>            - BuildTime: 1.280ms
>            - GetResultsTime: 0.000ns
>            - HTResizeTime: 0.000ns
>            - HashBuckets: 0 (0)
>            - LargestPartitionPercent: 0 (0)
>            - MaxPartitionLevel: 0 (0)
>            - NumRepartitions: 0 (0)
>            - PartitionsCreated: 0 (0)
>            - PeakMemoryUsage: 20.00 KB (20480)
>            - RowsRepartitioned: 0 (0)
>            - RowsReturned: 1 (1)
>            - RowsReturnedRate: 3.00 /sec
>            - SpilledPartitions: 0 (0)
>         HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child: 260.680ms, %
> non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
> 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
> 4:40/24.29 MB 11:33/20.77 MB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 461 out of
> 461
>           Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0% 5:0%
> 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>           File Formats: PARQUET/Unknown(Skipped):461
>            - FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms ; Max:
> 20.253ms ; Number of samples: 461)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 0.00
>            - BytesRead: 44.73 MB (46906407)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 44.73 MB (46906407)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 44.73 MB (46906407)
>            - CachedFileHandlesHitCount: 0 (0)
>            - CachedFileHandlesMissCount: 461 (461)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 0.000ns
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 0 (0)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 12 (12)
>            - NumRowGroups: 461 (461)
>            - NumScannerThreadsStarted: 24 (24)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 2.52 MB (2641928)
>            - PerReadThreadRawHdfsThroughput: 1.31 GB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 246.691ms
>            - RowBatchQueuePutWaitTime: 0.000ns
>            - RowsRead: 0 (0)
>            - RowsReturned: 461 (461)
>            - RowsReturnedRate: 1.77 K/sec
>            - ScanRangesComplete: 461 (461)
>            - ScannerThreadsInvoluntaryContextSwitches: 109 (109)
>            - ScannerThreadsTotalWallClockTime: 6s030ms
>              - MaterializeTupleTime(*): 68.735us
>              - ScannerThreadsSysTime: 33.983ms
>              - ScannerThreadsUserTime: 647.889ms
>            - ScannerThreadsVoluntaryContextSwitches: 1.70K (1696)
>            - TotalRawHdfsReadTime(*): 33.405ms
>            - TotalReadThroughput: 0.00 /sec
>       Instance c843dae48ce9b289:6ad3ffc000000003
> (host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms, non-child: 1.604ms,
> % non-child: 0.53%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB
> 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
> 4:41/25.99 MB 8:38/23.79 MB
>         MemoryUsage(500.000ms): 700.35 KB
>         ThreadUsage(500.000ms): 25
>          - AverageThreadTokens: 25.00
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 2.54 MB (2662753)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 2.54 MB (2662753)
>          - RowsProduced: 1 (1)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 300.675us
>          - TotalStorageWaitTime: 4s661ms
>          - TotalThreadsInvoluntaryContextSwitches: 90 (90)
>          - TotalThreadsTotalWallClockTime: 6s256ms
>            - TotalThreadsSysTime: 30.983ms
>            - TotalThreadsUserTime: 709.880ms
>          - TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 492.627us
>              - ExecTreeExecTime: 8.191us
>            - OpenTime: 277.922ms
>              - ExecTreeOpenTime: 255.395ms
>            - PrepareTime: 23.647ms
>              - ExecTreePrepareTime: 2.243ms
>         DataStreamSender (dst_id=2):(Total: 194.517us, non-child: 194.517us,
> % non-child: 100.00%)
>            - BytesSent: 16.00 B (16)
>            - NetworkThroughput(*): 371.86 KB/sec
>            - OverallThroughput: 80.33 KB/sec
>            - PeakMemoryUsage: 16.00 KB (16384)
>            - RowsReturned: 1 (1)
>            - SerializeBatchTime: 14.035us
>            - TransmitDataRPCTime: 42.019us
>            - UncompressedRowBatchSize: 16.00 B (16)
>         CodeGen:(Total: 42.676ms, non-child: 42.676ms, % non-child: 100.00%)
>            - CodegenTime: 1.236ms
>            - CompileTime: 7.717ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 1.86 MB (1950880)
>            - NumFunctions: 15 (15)
>            - NumInstructions: 214 (214)
>            - OptimizationTime: 14.130ms
>            - PeakMemoryUsage: 107.00 KB (109568)
>            - PrepareTime: 20.109ms
>         AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child: 1.541ms, %
> non-child: 0.60%)
>           ExecOption: Codegen Enabled
>            - BuildTime: 1.252ms
>            - GetResultsTime: 0.000ns
>            - HTResizeTime: 0.000ns
>            - HashBuckets: 0 (0)
>            - LargestPartitionPercent: 0 (0)
>            - MaxPartitionLevel: 0 (0)
>            - NumRepartitions: 0 (0)
>            - PartitionsCreated: 0 (0)
>            - PeakMemoryUsage: 20.00 KB (20480)
>            - RowsRepartitioned: 0 (0)
>            - RowsReturned: 1 (1)
>            - RowsReturnedRate: 3.00 /sec
>            - SpilledPartitions: 0 (0)
>         HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child: 256.099ms, %
> non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB
> 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
> 4:41/25.99 MB 8:38/23.79 MB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 459 out of
> 459
>           Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>           File Formats: PARQUET/Unknown(Skipped):459
>           BytesRead(500.000ms): 43.15 MB
>            - FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms ; Max:
> 23.004ms ; Number of samples: 459)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 24.00
>            - BytesRead: 44.52 MB (46678031)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 44.52 MB (46678031)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 44.52 MB (46678031)
>            - CachedFileHandlesHitCount: 0 (0)
>            - CachedFileHandlesMissCount: 459 (459)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 0.000ns
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 0 (0)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 12 (12)
>            - NumRowGroups: 459 (459)
>            - NumScannerThreadsStarted: 24 (24)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 2.50 MB (2621440)
>            - PerReadThreadRawHdfsThroughput: 1.33 GB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 243.426ms
>            - RowBatchQueuePutWaitTime: 0.000ns
>            - RowsRead: 0 (0)
>            - RowsReturned: 459 (459)
>            - RowsReturnedRate: 1.79 K/sec
>            - ScanRangesComplete: 459 (459)
>            - ScannerThreadsInvoluntaryContextSwitches: 85 (85)
>            - ScannerThreadsTotalWallClockTime: 5s978ms
>              - MaterializeTupleTime(*): 67.997us
>              - ScannerThreadsSysTime: 22.984ms
>              - ScannerThreadsUserTime: 678.884ms
>            - ScannerThreadsVoluntaryContextSwitches: 1.72K (1724)
>            - TotalRawHdfsReadTime(*): 32.626ms
>            - TotalReadThroughput: 86.30 MB/sec
>       Instance c843dae48ce9b289:6ad3ffc000000005
> (host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms, non-child: 1.242ms,
> % non-child: 0.42%)
>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
> 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
> 6:42/27.03 MB 7:47/29.22 MB
>          - AverageThreadTokens: 0.00
>          - BloomFilterBytes: 0
>          - PeakMemoryUsage: 2.66 MB (2793825)
>          - PeakReservation: 0
>          - PeakUsedReservation: 0
>          - PerHostPeakMemUsage: 2.68 MB (2806113)
>          - RowsProduced: 1 (1)
>          - TotalNetworkReceiveTime: 0.000ns
>          - TotalNetworkSendTime: 198.894us
>          - TotalStorageWaitTime: 4s374ms
>          - TotalThreadsInvoluntaryContextSwitches: 110 (110)
>          - TotalThreadsTotalWallClockTime: 6s201ms
>            - TotalThreadsSysTime: 29.983ms
>            - TotalThreadsUserTime: 642.892ms
>          - TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
>         Fragment Instance Lifecycle Timings:
>            - ExecTime: 350.873us
>              - ExecTreeExecTime: 8.173us
>            - OpenTime: 275.347ms
>              - ExecTreeOpenTime: 253.425ms
>            - PrepareTime: 22.341ms
>              - ExecTreePrepareTime: 2.251ms
>         DataStreamSender (dst_id=2):(Total: 164.394us, non-child: 164.394us,
> % non-child: 100.00%)
>            - BytesSent: 16.00 B (16)
>            - NetworkThroughput(*): 493.42 KB/sec
>            - OverallThroughput: 95.05 KB/sec
>            - PeakMemoryUsage: 16.00 KB (16384)
>            - RowsReturned: 1 (1)
>            - SerializeBatchTime: 14.262us
>            - TransmitDataRPCTime: 31.667us
>            - UncompressedRowBatchSize: 16.00 B (16)
>         CodeGen:(Total: 41.008ms, non-child: 41.008ms, % non-child: 100.00%)
>            - CodegenTime: 1.049ms
>            - CompileTime: 7.248ms
>            - LoadTime: 0.000ns
>            - ModuleBitcodeSize: 1.86 MB (1950880)
>            - NumFunctions: 15 (15)
>            - NumInstructions: 214 (214)
>            - OptimizationTime: 14.018ms
>            - PeakMemoryUsage: 107.00 KB (109568)
>            - PrepareTime: 19.080ms
>         AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child: 1.046ms, %
> non-child: 0.41%)
>           ExecOption: Codegen Enabled
>            - BuildTime: 1.015ms
>            - GetResultsTime: 0.000ns
>            - HTResizeTime: 0.000ns
>            - HashBuckets: 0 (0)
>            - LargestPartitionPercent: 0 (0)
>            - MaxPartitionLevel: 0 (0)
>            - NumRepartitions: 0 (0)
>            - PartitionsCreated: 0 (0)
>            - PeakMemoryUsage: 20.00 KB (20480)
>            - RowsRepartitioned: 0 (0)
>            - RowsReturned: 1 (1)
>            - RowsReturnedRate: 3.00 /sec
>            - SpilledPartitions: 0 (0)
>         HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child: 254.631ms, %
> non-child: 100.00%)
>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
> 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
> 6:42/27.03 MB 7:47/29.22 MB
>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 450 out of
> 450
>           Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0% 5:0%
> 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>           File Formats: PARQUET/Unknown(Skipped):450
>            - FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms ; Max:
> 23.755ms ; Number of samples: 450)
>            - AverageHdfsReadThreadConcurrency: 0.00
>            - AverageScannerThreadConcurrency: 0.00
>            - BytesRead: 43.85 MB (45983284)
>            - BytesReadDataNodeCache: 0
>            - BytesReadLocal: 43.85 MB (45983284)
>            - BytesReadRemoteUnexpected: 0
>            - BytesReadShortCircuit: 43.85 MB (45983284)
>            - CachedFileHandlesHitCount: 0 (0)
>            - CachedFileHandlesMissCount: 450 (450)
>            - CollectionItemsRead: 0 (0)
>            - DecompressionTime: 0.000ns
>            - MaxCompressedTextFileLength: 0
>            - NumColumns: 0 (0)
>            - NumDictFilteredRowGroups: 0 (0)
>            - NumDisksAccessed: 12 (12)
>            - NumRowGroups: 450 (450)
>            - NumScannerThreadsStarted: 24 (24)
>            - NumScannersWithNoReads: 0 (0)
>            - NumStatsFilteredRowGroups: 0 (0)
>            - PeakMemoryUsage: 2.62 MB (2752512)
>            - PerReadThreadRawHdfsThroughput: 1.56 GB/sec
>            - RemoteScanRanges: 0 (0)
>            - RowBatchQueueGetWaitTime: 242.362ms
>            - RowBatchQueuePutWaitTime: 0.000ns
>            - RowsRead: 0 (0)
>            - RowsReturned: 450 (450)
>            - RowsReturnedRate: 1.77 K/sec
>            - ScanRangesComplete: 450 (450)
>            - ScannerThreadsInvoluntaryContextSwitches: 106 (106)
>            - ScannerThreadsTotalWallClockTime: 5s926ms
>              - MaterializeTupleTime(*): 69.447us
>              - ScannerThreadsSysTime: 26.984ms
>              - ScannerThreadsUserTime: 607.897ms
>            - ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
>            - TotalRawHdfsReadTime(*): 27.393ms
>            - TotalReadThroughput: 0.00 /sec
> ===================================================
>
> Coordinator log:
>
>  01: query (string) = "select count(*) from dv_entersiteevent_prq_remote
> where year=2017 and month=10",
>   03: configuration (list) = list<string>[1] {
>     [0] = "REQUEST_POOL=impreports",
>   },
>   04: hadoop_user (string) = "fawzea",
> }
> I0208 01:07:06.321281  7154 impala-beeswax-server.cc:459]
> TClientRequest.queryOptions: TQueryOptions {
>   01: abort_on_error (bool) = false,
>   02: max_errors (i32) = 100,
>   03: disable_codegen (bool) = false,
>   04: batch_size (i32) = 0,
>   05: num_nodes (i32) = 0,
>   06: max_scan_range_length (i64) = 0,
>   07: num_scanner_threads (i32) = 0,
>   08: max_io_buffers (i32) = 0,
>   09: allow_unsupported_formats (bool) = false,
>   10: default_order_by_limit (i64) = -1,
>   11: debug_action (string) = "",
>   12: mem_limit (i64) = 0,
>   13: abort_on_default_limit_exceeded (bool) = false,
>   15: hbase_caching (i32) = 0,
>   16: hbase_cache_blocks (bool) = false,
>   17: parquet_file_size (i64) = 0,
>   18: explain_level (i32) = 1,
>   19: sync_ddl (bool) = false,
>   20: request_pool (string) = "impreports",
>   23: disable_cached_reads (bool) = false,
>   24: disable_outermost_topn (bool) = false,
>   25: rm_initial_mem (i64) = 0,
>   26: query_timeout_s (i32) = 0,
>   28: appx_count_distinct (bool) = false,
>   29: disable_unsafe_spills (bool) = false,
>   31: exec_single_node_rows_threshold (i32) = 100,
>   32: optimize_partition_key_scans (bool) = false,
>   33: replica_preference (i32) = 0,
>   34: schedule_random_replica (bool) = false,
>   35: scan_node_codegen_threshold (i64) = 1800000,
>   36: disable_streaming_preaggregations (bool) = false,
>   37: runtime_filter_mode (i32) = 2,
>   38: runtime_bloom_filter_size (i32) = 1048576,
>   39: runtime_filter_wait_time_ms (i32) = 0,
>   40: disable_row_runtime_filtering (bool) = false,
>   41: max_num_runtime_filters (i32) = 10,
>   42: parquet_annotate_strings_utf8 (bool) = false,
>   43: parquet_fallback_schema_resolution (i32) = 0,
>   45: s3_skip_insert_staging (bool) = true,
>   46: runtime_filter_min_size (i32) = 1048576,
>   47: runtime_filter_max_size (i32) = 16777216,
>   48: prefetch_mode (i32) = 1,
>   49: strict_mode (bool) = false,
>   50: scratch_limit (i64) = -1,
>   51: enable_expr_rewrites (bool) = true,
>   52: decimal_v2 (bool) = false,
>   53: parquet_dictionary_filtering (bool) = true,
>   54: parquet_array_resolution (i32) = 2,
>   55: parquet_read_statistics (bool) = true,
>   56: default_join_distribution_mode (i32) = 0,
>   57: disable_codegen_rows_threshold (i32) = 50000,
>   58: default_spillable_buffer_size (i64) = 2097152,
>   59: min_spillable_buffer_size (i64) = 65536,
>   60: max_row_size (i64) = 524288,
> }
> I0208 01:07:06.324410  7154 Frontend.java:909] Compiling query: select
> count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10
> I0208 01:07:06.327203  7154 Frontend.java:948] Compiled query.
> I0208 01:07:06.359748  7154 admission-controller.cc:510] Schedule for
> id=5f49e927591a9042:faf8f28600000000 in pool_name=root.impreports
> cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1 max_queued=200
> max_mem=-1.00 B
> I0208 01:07:06.359776  7154 admission-controller.cc:515] Stats:
> agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
> local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0,
> backend_mem_reserved=0)
> I0208 01:07:06.359784  7154 admission-controller.cc:531] Admitted query
> id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.359807  7154 coordinator.cc:99] Exec()
> query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*) from
> dv_entersiteevent_prq_remote where year=2017 and month=10
> I0208 01:07:06.360694  7154 coordinator.cc:357] starting execution on 5
> backends for query 5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.363255 63583 impala-internal-service.cc:44]
> ExecQueryFInstances(): query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.363282 63583 query-exec-mgr.cc:46] StartQueryFInstances()
> query_id=5f49e927591a9042:faf8f28600000000
> coord=aopr-dhc001.lpdomain.com:22000
> I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool limit for
> 5f49e927591a9042:faf8f28600000000: 10802010521
> I0208 01:07:06.363364 63583 initial-reservations.cc:60] Successfully claimed
> initial reservations (0) for query 5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.363544  7155 query-state.cc:286] StartFInstances():
> query_id=5f49e927591a9042:faf8f28600000000 #instances=2
> I0208 01:07:06.363844  7155 query-state.cc:299] descriptor table for
> query=5f49e927591a9042:faf8f28600000000
> tuples:
> Tuple(id=2 size=0 slots=[] tuple_path=[])
> Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[] offset=0
> null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
> Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0
> null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
> I0208 01:07:06.364013  7156 query-state.cc:377] Executing instance.
> instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0
> per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
> I0208 01:07:06.364135  7157 query-state.cc:377] Executing instance.
> instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1
> per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
> I0208 01:07:06.364241  7157 hdfs-scan-node.cc:160] Max row batch queue size
> for scan node '0' in fragment instance '5f49e927591a9042:faf8f28600000005':
> 160
> I0208 01:07:06.366775  7154 coordinator.cc:370] started execution on 5
> backends for query 5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.385625  7155 query-exec-mgr.cc:149] ReleaseQueryState():
> query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
> I0208 01:07:06.671193  7559 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
> I0208 01:07:06.676138   544 data-stream-mgr.cc:228] Reduced stream ID cache
> from 2 items, to 0, eviction took: 0
> I0208 01:07:06.677202   544 coordinator.cc:955] Backend completed:
> host=aopr-dhc002.lpdomain.com:22000 remaining=4
> I0208 01:07:06.677214   544 coordinator-backend-state.cc:229]
> query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> aopr-dhc001.lpdomain.com:22000
> I0208 01:07:06.684221  7157 query-state.cc:384] Instance completed.
> instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1 status=OK
> I0208 01:07:06.684237  7157 query-exec-mgr.cc:149] ReleaseQueryState():
> query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
> I0208 01:07:06.700275   520 coordinator.cc:955] Backend completed:
> host=aopr-dhc005.lpdomain.com:22000 remaining=3
> I0208 01:07:06.700289   520 coordinator-backend-state.cc:229]
> query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> aopr-dhc001.lpdomain.com:22000
> I0208 01:07:06.704620 65258 coordinator.cc:955] Backend completed:
> host=aopr-dhc003.lpdomain.com:22000 remaining=2
> I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229]
> query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> aopr-dhc001.lpdomain.com:22000
> I0208 01:07:06.706041  7156 data-stream-mgr.cc:238] DeregisterRecvr():
> fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
> I0208 01:07:06.706070  7156 data-stream-recvr.cc:235] cancelled stream:
> fragment_instance_id_=5f49e927591a9042:faf8f28600000000 node_id=2
> I0208 01:07:06.706809 64916 coordinator.cc:955] Backend completed:
> host=aopr-dhc004.lpdomain.com:22000 remaining=1
> I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229]
> query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> aopr-dhc001.lpdomain.com:22000
> I0208 01:07:06.789568  7154 impala-beeswax-server.cc:193]
> get_results_metadata(): query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.790439  7154 coordinator.cc:789] Coordinator waiting for
> backends to finish, 1 remaining
> I0208 01:07:06.791052  7154 coordinator.cc:794] All backends finished
> successfully.
> I0208 01:07:06.791086  7154 coordinator.cc:1090] Release admssion control
> resources for query 5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.791363  7156 query-state.cc:384] Instance completed.
> instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0 status=OK
> I0208 01:07:06.791414  7156 query-exec-mgr.cc:149] ReleaseQueryState():
> query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
> I0208 01:07:06.793586  7154 impala-beeswax-server.cc:235] close():
> query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.793602  7154 impala-server.cc:992] UnregisterQuery():
> query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.793614  7154 impala-server.cc:1075] Cancel():
> query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.793624  7154 coordinator.cc:895] Cancel()
> query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.793637  7154 coordinator.cc:905] CancelBackends()
> query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0 backends
> I0208 01:07:06.802776  7154 query-exec-mgr.cc:149] ReleaseQueryState():
> query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
> I0208 01:07:06.814214  7154 impala-server.cc:1796] Connection from client
> ::ffff:10.16.144.131:51012 closed, closing 1 associated session(s)
> I0208 01:07:07.258891  7567 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
> I0208 01:07:11.600131  7819 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
> I0208 01:07:12.262739  7821 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
> I0208 01:07:16.665451  7822 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
> I0208 01:07:17.235321  7825 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.2:34752 closed, closing 1 associated session(s)
> I0208 01:07:21.410611  5858 impala-beeswax-server.cc:193]
> get_results_metadata(): query_id=1d4fbf0f2571f1e9:ea346f7900000000
> I0208 01:07:21.414175  5858 impala-beeswax-server.cc:235] close():
> query_id=1d4fbf0f2571f1e9:ea346f7900000000
> I0208 01:07:21.414185  5858 impala-server.cc:992] UnregisterQuery():
> query_id=1d4fbf0f2571f1e9:ea346f7900000000
> I0208 01:07:21.414191  5858 impala-server.cc:1075] Cancel():
> query_id=1d4fbf0f2571f1e9:ea346f7900000000
> I0208 01:07:21.421505  5858 impala-server.cc:1796] Connection from client
> ::ffff:10.26.180.75:60072 closed, closing 1 associated session(s)
> I0208 01:07:21.605690  7830 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.3:46660 closed, closing 1 associated session(s)
>
>
> On Thu, Feb 8, 2018 at 6:20 AM, Matthew Jacobs <mjacobs@apache.org> wrote:
>>
>> Thanks! I think that configuration looks OK.
>>
>> 1) What version of Impala is this?
>>
>> 2) Can you send the query profile and the Impala coordinator log for
>> the query where it's not behaving as expected?
>>
>> > When i'm running the following with fawzea user:
>> > set request_pool=impreports;
>> > select count(*) from dv_entersiteevent_prq_remote where year=2017 and
>> > month=10;
>> > I can see that the query is running with impreports pool.
>>
>> On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <fawzeaj@gmail.com> wrote:
>> > Hi Matt,
>> >
>> > [root@aopr-dhc001 ~]# cat
>> >
>> > /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/impala-conf/fair-scheduler.xml
>> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>> > <allocations>
>> >     <queue name="root">
>> >         <weight>1.0</weight>
>> >         <schedulingPolicy>fair</schedulingPolicy>
>> >         <queue name="impiaas">
>> >             <weight>1.0</weight>
>> >             <schedulingPolicy>fair</schedulingPolicy>
>> >             <aclSubmitApps>fawzea </aclSubmitApps>
>> >         </queue>
>> >         <queue name="impreports">
>> >             <weight>1.0</weight>
>> >             <schedulingPolicy>fair</schedulingPolicy>
>> >             <aclSubmitApps>nira </aclSubmitApps>
>> >         </queue>
>> >     </queue>
>> >     <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
>> >     <queuePlacementPolicy>
>> >         <rule name="specified" create="false"/>
>> >         <rule name="secondaryGroupExistingQueue"/>
>> >         <rule name="default"/>
>> >     </queuePlacementPolicy>
>> > </allocations>
>> >
>> >
>> > On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <jacobs.mh@gmail.com>
>> > wrote:
>> >>
>> >> Yes, I saw that, but I'd like to see the actual configuration that is
>> >> passed to Impala because it may not be specified in the right way to
>> >> accomplish what you're trying to do in "b".
>> >>
>> >> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <fawzeaj@gmail.com>
>> >> wrote:
>> >> > Hi Matt,
>> >> >
>> >> > I created the pools manually and it’s completely nfigurations as
>> >> > shown
>> >> > in b.
>> >> >
>> >> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <jacobs.mh@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> How, more specifically, are you configuring these pools? Can you
>> >> >> share
>> >> >> the configuration files?
>> >> >>
>> >> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber <fawzeaj@gmail.com>
>> >> >> wrote:
>> >> >> > Hello All,
>> >> >> >
>> >> >> > Will appreciate any insight of what i'm missing in my
>> >> >> > configuration
>> >> >> > to
>> >> >> > get
>> >> >> > the impala submission control working as expected ....
>> >> >> >
>> >> >> > a- Here is my placement Placement Rule:
>> >> >> > 1- Use the pool Specified at run time., only if the pool exists.
>> >> >> > 2- Use the pool root.[secondary group].
>> >> >> > 3- Use the pool root.default.
>> >> >> >
>> >> >> > b- Here is the impala resource pools
>> >> >> >
>> >> >> > Name                    Max Memory Max Running Queries Max Queued
>> >> >> > Queries
>> >> >> > Queue Timeout Default Query Memory Limit Access Control
>> >> >> > root.impiaas            Unlimited                   Unlimited
>> >> >> > 200                          1 minute(s)            No Default
>> >> >> > Custom
>> >> >> > root.impreports        Unlimited                   Unlimited
>> >> >> > 200                          1 minute(s)            No Default
>> >> >> > Custom
>> >> >> >
>> >> >> >
>> >> >> > C- Here is my Access control for root.impiaas pool:
>> >> >> >
>> >> >> > Allow these users and groups to submit to this pool
>> >> >> >
>> >> >> > Users: fawzea
>> >> >> >
>> >> >> > The same for impreports the allowed user and groups to submit to
>> >> >> > pool
>> >> >> > is
>> >> >> > only nira.
>> >> >> >
>> >> >> >
>> >> >> > LDAP groups for fawzea user:
>> >> >> >
>> >> >> >  Member in groups:
>> >> >> >
>> >> >> >  - dba
>> >> >> >  - impiaas
>> >> >> >  - mysql
>> >> >> >
>> >> >> >  LDAP group for nira user
>> >> >> >   Member in groups:
>> >> >> >  - dba
>> >> >> >  - impreports
>> >> >> >  - mysql
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > Impala Queries UI shows:
>> >> >> >
>> >> >> > 1- select count(*) from dv_entersiteevent_prq_remote where
>> >> >> > year=2017
>> >> >> >
>> >> >> > User: fawzea
>> >> >> > Pool: root.impiaas
>> >> >> >
>> >> >> > 2- select count(*) from dv_entersiteevent_prq_remote where
>> >> >> > year=2017
>> >> >> > and
>> >> >> > month=10
>> >> >> > User: nira
>> >> >> > Pool: root.impreports
>> >> >> >
>> >> >> > As shown above that the queries submitted based on the secondary
>> >> >> > group
>> >> >> > rule.
>> >> >> >
>> >> >> > When i'm running the following with fawzea user:
>> >> >> >
>> >> >> > set request_pool=impreports;
>> >> >> > select count(*) from dv_entersiteevent_prq_remote where year=2017
>> >> >> > and
>> >> >> > month=10;
>> >> >> >
>> >> >> > I can see that the query is running with impreports pool.
>> >> >> >
>> >> >> >
>> >> >> > What i'm missing here in my configuration to get this works as
>> >> >> > expected.
>> >> >> >
>> >
>> >
>
>

Mime
View raw message