impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeszy <jes...@gmail.com>
Subject Re: Estimate peak memory VS used peak memory
Date Fri, 23 Feb 2018 09:26:37 GMT
Queries will be killed based on actual usage (peak memory usage across
hosts), so the 200mb is the interesting value in your example.

Compare the pool's available memory to the query's mem requirement
(based on estimate or mem_limit, as discussed) to predict admission.

On 23 February 2018 at 10:06, Fawze Abujaber <fawzeaj@gmail.com> wrote:
> Thanks jezy for your detailed response.
>
> Yes I read the documentation.
>
> Let simplify my question:
>
> I have pools set up with memory limit per node and concurrency.
>
> If I’m looking on the historical impala queries that I have and the metrics
> I have per query, on which metrics I can understand that impala will kill
> the query, for example if I have a query with estimate of 2GB and the used
> per node is 200mb, what is the default memory values that i need to setup so
> the query will not fail.
>
> The second one is the distribution between pools, if one query is running
> which metrics o have to look into to know if I submit a query it fail or
> not.
>
> On Fri, 23 Feb 2018 at 10:48 Jeszy <jeszyb@gmail.com> wrote:
>>
>> Hey Fawze,
>>
>> Answers inline.
>>
>> On 23 February 2018 at 01:23, Fawze Abujaber <fawzeaj@gmail.com> wrote:
>> > There is no option in the admission control to setup memory limit per
>> > query,
>> > the memory limit is per pool and there is a default memory per node for
>> > query.
>>
>> per node for query memory limit multiplied by number of nodes gives
>> you a per query memory limit. I agree its confusing that the
>> configurations mix and match between per-node and aggregated values.
>> In this case there's a good reason though, as a single node running
>> out of memory will lead to query failure, meaning that in addition to
>> total memory used, distribution of memory usage between hosts also
>> matters.
>>
>> > I have hundreds of impala queries and more add hoc queries, making a
>> > pool
>> > for each query is not a visible solution.
>> >
>> > still waiting to understand how the estimate per node related to the
>> > default
>> > memory per node I set up per pool, is it used in the decision of queuing
>> > and
>> > killing the query? and if this is true how it was not kill a query that
>> > was
>> > estimated it needs 8.2TB memory per node.
>> >
>> > Understanding on which parameters impala decides to kill a query can
>> > help
>> > understand to define and divide the memory between the pools.
>>
>> If you set mem_limit at any level (service level, pool level, or query
>> level), it will be used for admission control purposes instead of
>> estimates. So a 8.2TB estimate would not be a problem, if impala can
>> reserve mem_limit amount on each host, it will start running the
>> query.
>>
>> > Passing memory limit per query manually is also not visible and such
>> > settings not needs admission control.
>> >
>> > I have support pool that runs ad hoc query and I can not ask them to use
>> > memory limit per query, and I have analytics pool which is fully
>> > business
>> > and I can rely on admission control if it extremely in accurate.
>>
>> It's a bit tricky to use memory-based admission control with
>> non-trivial ad hoc queries. For simple ad-hoc queries, you can try to
>> come up with a 'good enough' mem_limit, or omit mem_limit and trust
>> impala's estimations. You can check the estimated vs. actual values
>> for a representative set of ad hoc queries to see what would work in
>> your case. I've found that people tend to go with a large enough
>> mem_limit for the ad hoc pool.
>>
>> > Can someone explain me exactly which recommended setting to use per pool
>> > and
>> > which of them rely on impala memory estimates?
>>
>> The documentation of admission control
>> (https://impala.apache.org/docs/build/html/topics/impala_admission.html)
>> gives you a good view on how stuff works, but you will have to figure
>> out how to use these features for your specific use case. That said,
>> when using memory based admission control, it is best practice to
>> always use a mem_limit due to potential inaccuracy of estimates as
>> well as potential variance of estimates between Impala releases. Keep
>> in mind that you can opt to set a default mem_limit for one pool and
>> leave it unset for another.
>>
>> > So my conclusion right now to avoid using any settings rely on the
>> > estimates
>> > and to ignore the estimates when I want to evaluate query.
>>
>> Sounds good.
>>
>> > @mostafa, since my issue with all the query, I think the profile will
>> > not
>> > help me to solve such huge issue.
>> >
>> > I’m planning to move a way from Vertica and rely on impala as a sql
>> > engine
>> > and now fully confused how I can do this if I can’t use the admission
>> > control.
>> >
>> > Last think, is it recommend to use the impala admission control?
>>
>> Yes. Admission control can take a while to understand, but if done
>> right, it works.
>>
>> HTH
>>
>> > On Fri, 23 Feb 2018 at 1:56 Alexander Behm <alex.behm@cloudera.com>
>> > wrote:
>> >>
>> >> The planner memory estimates are conservative and sometimes extremely
>> >> inaccurate. In their current form, they are rarely appropriate for
>> >> admission
>> >> decisions.
>> >>
>> >> The recommended practice for memory-based admission control it to set a
>> >> mem_limit for every query. You can make this easier by setting up
>> >> different
>> >> pools with different mem_limits, e.g. a small/medium/big queries pool
>> >> or
>> >> similar.
>> >>
>> >> On Thu, Feb 22, 2018 at 3:00 PM, Mostafa Mokhtar
>> >> <mmokhtar@cloudera.com>
>> >> wrote:
>> >>>
>> >>> It is recommended to set a per query memory limit as part of admission
>> >>> and not rely on estimates as they are sometimes inaccurate.
>> >>> Can you please include the full query profile?
>> >>>
>> >>>
>> >>> On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <fawzeaj@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi Mostafa,
>> >>>>
>> >>>> It's not a specific query, almost all the query has such differene
>> >>>> between the 2 values.
>> >>>>
>> >>>> I can see even queries showing the estimate per node is 8.2 Tib
>> >>>>
>> >>>> User: psanalytics
>> >>>>
>> >>>> Database: default
>> >>>>
>> >>>> Query Type: QUERY
>> >>>> Coordinator: slpr-dhc014.lpdomain.com
>> >>>>
>> >>>> Duration: 6.48s
>> >>>>
>> >>>> Rows Produced: 708
>> >>>> Estimated per Node Peak Memory: 8.2 TiB
>> >>>>
>> >>>> Per Node Peak Memory Usage: 1.1 GiB
>> >>>>
>> >>>> Pool: root.impanalytics
>> >>>> Threads: CPU Time: 20.1m
>> >>>>
>> >>>>
>> >>>>
>> >>>> How you can explain this behavior, and for sure i don't have 8.2
Tib
>> >>>> memory per node to give neither you.
>> >>>>
>> >>>> Can you please explain me how i should treat Estimated per Node
Peak
>> >>>> Memory and if it used by impala for the resource pool and admission
>> >>>> control
>> >>>> and what is the relation of this value to the default memory per
node
>> >>>> that i
>> >>>> setup for each resource pool?
>> >>>>
>> >>>> Below is part of one of the queries profile which the estimate per
>> >>>> node
>> >>>> was ~ @GB and the used was 200MB per node.
>> >>>>
>> >>>>
>> >>>>  Instance 744de1b6228736fa:b54bfaa70000000f
>> >>>> (host=slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child:
>> >>>> 1s292ms, %
>> >>>> non-child: 88.82%)
>> >>>>         Hdfs split stats (<volume id>:<# splits>/<split
lengths>):
>> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB
>> >>>> 9:1/191.64
>> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB
>> >>>>         MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB
>> >>>>         ThreadUsage(500.000ms): 1, 1, 1
>> >>>>          - AverageThreadTokens: 1.00
>> >>>>          - BloomFilterBytes: 0
>> >>>>          - PeakMemoryUsage: 7.17 MB (7521751)
>> >>>>          - PeakReservation: 0
>> >>>>          - PeakUsedReservation: 0
>> >>>>          - PerHostPeakMemUsage: 106.53 MB (111709581)
>> >>>>          - RowsProduced: 32.83K (32826)
>> >>>>          - TotalNetworkReceiveTime: 0.000ns
>> >>>>          - TotalNetworkSendTime: 1s297ms
>> >>>>          - TotalStorageWaitTime: 234.356ms
>> >>>>          - TotalThreadsInvoluntaryContextSwitches: 66 (66)
>> >>>>          - TotalThreadsTotalWallClockTime: 1s715ms
>> >>>>            - TotalThreadsSysTime: 5.998ms
>> >>>>            - TotalThreadsUserTime: 124.975ms
>> >>>>          - TotalThreadsVoluntaryContextSwitches: 303 (303)
>> >>>>         Fragment Instance Lifecycle Timings:
>> >>>>            - ExecTime: 1s394ms
>> >>>>              - ExecTreeExecTime: 67.115ms
>> >>>>            - OpenTime: 32.795ms
>> >>>>              - ExecTreeOpenTime: 73.243us
>> >>>>            - PrepareTime: 27.602ms
>> >>>>              - ExecTreePrepareTime: 243.141us
>> >>>>         DataStreamSender (dst_id=11):(Total: 38.747ms, non-child:
>> >>>> 38.747ms, % non-child: 100.00%)
>> >>>>            - BytesSent: 39.71 MB (41643000)
>> >>>>            - NetworkThroughput(*): 1.97 GB/sec
>> >>>>            - OverallThroughput: 1.00 GB/sec
>> >>>>            - PeakMemoryUsage: 59.38 KB (60800)
>> >>>>            - RowsReturned: 32.83K (32826)
>> >>>>            - SerializeBatchTime: 16.860ms
>> >>>>            - TransmitDataRPCTime: 19.698ms
>> >>>>            - UncompressedRowBatchSize: 77.58 MB (81350840)
>> >>>>         CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child:
>> >>>> 100.00%)
>> >>>>            - CodegenTime: 1.299ms
>> >>>>            - CompileTime: 10.672ms
>> >>>>            - LoadTime: 0.000ns
>> >>>>            - ModuleBitcodeSize: 1.96 MB (2050180)
>> >>>>            - NumFunctions: 16 (16)
>> >>>>            - NumInstructions: 250 (250)
>> >>>>            - OptimizationTime: 21.023ms
>> >>>>            - PeakMemoryUsage: 125.00 KB (128000)
>> >>>>            - PrepareTime: 24.116ms
>> >>>>         SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms,
%
>> >>>> non-child: 17.85%)
>> >>>>            - PeakMemoryUsage: 627.94 KB (643015)
>> >>>>            - RowsReturned: 32.77K (32768)
>> >>>>            - RowsReturnedRate: 486.81 K/sec
>> >>>>           NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child:
>> >>>> 25.197ms, % non-child: 74.11%)
>> >>>>              - BuildRows: 0 (0)
>> >>>>              - BuildTime: 0.000ns
>> >>>>              - PeakMemoryUsage: 24.00 KB (24576)
>> >>>>              - ProbeRows: 32.83K (32826)
>> >>>>              - ProbeTime: 0.000ns
>> >>>>              - RowsReturned: 16.80M (16795311)
>> >>>>              - RowsReturnedRate: 493.99 M/sec
>> >>>>             Nested Loop Join Builder:
>> >>>>                - PeakMemoryUsage: 8.00 KB (8192)
>> >>>>             SINGULAR_ROW_SRC_NODE (id=7):
>> >>>>                - PeakMemoryUsage: 0
>> >>>>                - RowsReturned: 0 (0)
>> >>>>                - RowsReturnedRate: 0
>> >>>>           UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms,
%
>> >>>> non-child: 100.00%)
>> >>>>              - AvgCollectionSize: 1.00
>> >>>>              - MaxCollectionSize: 1 (1)
>> >>>>              - MinCollectionSize: 1 (1)
>> >>>>              - NumCollections: 32.83K (32826)
>> >>>>              - PeakMemoryUsage: 0
>> >>>>              - RowsReturned: 1 (1)
>> >>>>              - RowsReturnedRate: 113.00 /sec
>> >>>>         HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms,
>> >>>> %
>> >>>> non-child: 100.00%)
>> >>>>           Hdfs split stats (<volume id>:<# splits>/<split
lengths>):
>> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB
>> >>>> 9:1/191.64
>> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB
>> >>>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled:
11
>> >>>> out
>> >>>> of 11
>> >>>>           Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
3:0%
>> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
>> >>>> 16:0%
>> >>>>           File Formats: PARQUET/SNAPPY:66
>> >>>>           BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB
>> >>>>            - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms
;
>> >>>> Max:
>> >>>> 29.275ms ; Number of samples: 11)
>> >>>>            - AverageHdfsReadThreadConcurrency: 0.00
>> >>>>            - AverageScannerThreadConcurrency: 0.00
>> >>>>            - BytesRead: 2.26 MB (2373738)
>> >>>>            - BytesReadDataNodeCache: 0
>> >>>>            - BytesReadLocal: 2.26 MB (2373738)
>> >>>>            - BytesReadRemoteUnexpected: 0
>> >>>>            - BytesReadShortCircuit: 2.26 MB (2373738)
>> >>>>            - CachedFileHandlesHitCount: 0 (0)
>> >>>>            - CachedFileHandlesMissCount: 77 (77)
>> >>>>            - DecompressionTime: 1.543ms
>> >>>>            - MaxCompressedTextFileLength: 0
>> >>>>            - NumColumns: 6 (6)
>> >>>>            - NumDictFilteredRowGroups: 0 (0)
>> >>>>            - NumDisksAccessed: 8 (8)
>> >>>>            - NumRowGroups: 11 (11)
>> >>>>            - NumScannerThreadsStarted: 11 (11)
>> >>>>            - NumScannersWithNoReads: 0 (0)
>> >>>>            - NumStatsFilteredRowGroups: 0 (0)
>> >>>>            - PeakMemoryUsage: 6.51 MB (6821203)
>> >>>>            - PerReadThreadRawHdfsThroughput: 764.51 MB/sec
>> >>>>            - RemoteScanRanges: 0 (0)
>> >>>>            - RowBatchQueueGetWaitTime: 18.825ms
>> >>>>            - RowBatchQueuePutWaitTime: 0.000ns
>> >>>>            - RowsRead: 73.46K (73464)
>> >>>>            - RowsReturned: 32.83K (32826)
>> >>>>            - RowsReturnedRate: 1.54 M/sec
>> >>>>            - ScanRangesComplete: 11 (11)
>> >>>>            - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>> >>>>            - ScannerThreadsTotalWallClockTime: 287.702ms
>> >>>>              - MaterializeTupleTime(*): 11.921ms
>> >>>>              - ScannerThreadsSysTime: 999.000us
>> >>>>              - ScannerThreadsUserTime: 26.990ms
>> >>>>            - ScannerThreadsVoluntaryContextSwitches: 232 (232)
>> >>>>            - TotalRawHdfsReadTime(*): 2.961ms
>> >>>>            - TotalReadThroughput: 1.51 MB/sec
>> >>>>       Instance 744de1b6228736fa:b54bfaa70000000d
>> >>>> (host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child:
>> >>>> 1s254ms, %
>> >>>> non-child: 86.28%)
>> >>>>         Hdfs split stats (<volume id>:<# splits>/<split
lengths>):
>> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
>> >>>>         MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB
>> >>>>         ThreadUsage(500.000ms): 1, 1, 1
>> >>>>          - AverageThreadTokens: 1.00
>> >>>>          - BloomFilterBytes: 0
>> >>>>          - PeakMemoryUsage: 5.86 MB (6145964)
>> >>>>          - PeakReservation: 0
>> >>>>          - PeakUsedReservation: 0
>> >>>>          - PerHostPeakMemUsage: 106.66 MB (111842766)
>> >>>>          - RowsProduced: 30.34K (30340)
>> >>>>          - TotalNetworkReceiveTime: 0.000ns
>> >>>>          - TotalNetworkSendTime: 1s256ms
>> >>>>          - TotalStorageWaitTime: 106.530ms
>> >>>>          - TotalThreadsInvoluntaryContextSwitches: 177 (177)
>> >>>>          - TotalThreadsTotalWallClockTime: 1s557ms
>> >>>>            - TotalThreadsSysTime: 6.998ms
>> >>>>            - TotalThreadsUserTime: 143.976ms
>> >>>>          - TotalThreadsVoluntaryContextSwitches: 136 (136)
>> >>>>         Fragment Instance Lifecycle Timings:
>> >>>>            - ExecTime: 1s390ms
>> >>>>              - ExecTreeExecTime: 91.500ms
>> >>>>            - OpenTime: 34.085ms
>> >>>>              - ExecTreeOpenTime: 81.436us
>> >>>>            - PrepareTime: 28.524ms
>> >>>>              - ExecTreePrepareTime: 217.949us
>> >>>>         DataStreamSender (dst_id=11):(Total: 48.674ms, non-child:
>> >>>> 48.674ms, % non-child: 100.00%)
>> >>>>            - BytesSent: 36.61 MB (38388760)
>> >>>>            - NetworkThroughput(*): 1.51 GB/sec
>> >>>>            - OverallThroughput: 752.15 MB/sec
>> >>>>            - PeakMemoryUsage: 59.38 KB (60800)
>> >>>>            - RowsReturned: 30.34K (30340)
>> >>>>            - SerializeBatchTime: 23.132ms
>> >>>>            - TransmitDataRPCTime: 23.609ms
>> >>>>            - UncompressedRowBatchSize: 71.70 MB (75187520)
>> >>>>         CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child:
>> >>>> 100.00%)
>> >>>>            - CodegenTime: 1.382ms
>> >>>>            - CompileTime: 10.435ms
>> >>>>            - LoadTime: 0.000ns
>> >>>>            - ModuleBitcodeSize: 1.96 MB (2050180)
>> >>>>            - NumFunctions: 16 (16)
>> >>>>            - NumInstructions: 250 (250)
>> >>>>            - OptimizationTime: 22.325ms
>> >>>>            - PeakMemoryUsage: 125.00 KB (128000)
>> >>>>            - PrepareTime: 25.459ms
>> >>>>         SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms,
%
>> >>>> non-child: 22.90%)
>> >>>>            - PeakMemoryUsage: 699.55 KB (716335)
>> >>>>            - RowsReturned: 29.70K (29696)
>> >>>>            - RowsReturnedRate: 323.80 K/sec
>> >>>>           NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child:
>> >>>> 40.263ms, % non-child: 74.95%)
>> >>>>              - BuildRows: 0 (0)
>> >>>>              - BuildTime: 0.000ns
>> >>>>              - PeakMemoryUsage: 24.00 KB (24576)
>> >>>>              - ProbeRows: 30.34K (30340)
>> >>>>              - ProbeTime: 0.000ns
>> >>>>              - RowsReturned: 15.43M (15426890)
>> >>>>              - RowsReturnedRate: 287.16 M/sec
>> >>>>             Nested Loop Join Builder:
>> >>>>                - PeakMemoryUsage: 8.00 KB (8192)
>> >>>>             SINGULAR_ROW_SRC_NODE (id=7):
>> >>>>                - PeakMemoryUsage: 0
>> >>>>                - RowsReturned: 0 (0)
>> >>>>                - RowsReturnedRate: 0
>> >>>>           UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms,
%
>> >>>> non-child: 100.00%)
>> >>>>              - AvgCollectionSize: 1.00
>> >>>>              - MaxCollectionSize: 1 (1)
>> >>>>              - MinCollectionSize: 1 (1)
>> >>>>              - NumCollections: 30.34K (30340)
>> >>>>              - PeakMemoryUsage: 0
>> >>>>              - RowsReturned: 1 (1)
>> >>>>              - RowsReturnedRate: 74.00 /sec
>> >>>>         HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms,
>> >>>> %
>> >>>> non-child: 100.00%)
>> >>>>           Hdfs split stats (<volume id>:<# splits>/<split
lengths>):
>> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
>> >>>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled:
5 out
>> >>>> of
>> >>>> 5
>> >>>>           Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
3:0%
>> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
>> >>>> 16:0%
>> >>>>           File Formats: PARQUET/SNAPPY:30
>> >>>>           BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB
>> >>>>            - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms
;
>> >>>> Max:
>> >>>> 28.687ms ; Number of samples: 5)
>> >>>>            - AverageHdfsReadThreadConcurrency: 0.00
>> >>>>            - AverageScannerThreadConcurrency: 0.00
>> >>>>            - BytesRead: 1.56 MB (1630546)
>> >>>>            - BytesReadDataNodeCache: 0
>> >>>>            - BytesReadLocal: 1.56 MB (1630546)
>> >>>>            - BytesReadRemoteUnexpected: 0
>> >>>>            - BytesReadShortCircuit: 1.56 MB (1630546)
>> >>>>            - CachedFileHandlesHitCount: 0 (0)
>> >>>>            - CachedFileHandlesMissCount: 35 (35)
>> >>>>            - DecompressionTime: 1.267ms
>> >>>>            - MaxCompressedTextFileLength: 0
>> >>>>            - NumColumns: 6 (6)
>> >>>>            - NumDictFilteredRowGroups: 0 (0)
>> >>>>            - NumDisksAccessed: 4 (4)
>> >>>>            - NumRowGroups: 5 (5)
>> >>>>            - NumScannerThreadsStarted: 5 (5)
>> >>>>            - NumScannersWithNoReads: 0 (0)
>> >>>>            - NumStatsFilteredRowGroups: 0 (0)
>> >>>>            - PeakMemoryUsage: 5.59 MB (5859290)
>> >>>>            - PerReadThreadRawHdfsThroughput: 920.32 MB/sec
>> >>>>            - RemoteScanRanges: 0 (0)
>> >>>>            - RowBatchQueueGetWaitTime: 15.335ms
>> >>>>            - RowBatchQueuePutWaitTime: 0.000ns
>> >>>>            - RowsRead: 64.69K (64692)
>> >>>>            - RowsReturned: 30.34K (30340)
>> >>>>            - RowsReturnedRate: 1.79 M/sec
>> >>>>            - ScanRangesComplete: 5 (5)
>> >>>>            - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>> >>>>            - ScannerThreadsTotalWallClockTime: 132.548ms
>> >>>>              - MaterializeTupleTime(*): 10.585ms
>> >>>>              - ScannerThreadsSysTime: 999.000us
>> >>>>              - ScannerThreadsUserTime: 18.995ms
>> >>>>            - ScannerThreadsVoluntaryContextSwitches: 79 (79)
>> >>>>            - TotalRawHdfsReadTime(*): 1.689ms
>> >>>>            - TotalReadThroughput: 1.04 MB/sec
>> >>>>       Instance 744de1b6228736fa:b54bfaa700000002
>> >>>> (host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child:
>> >>>> 1s255ms, %
>> >>>> non-child: 86.41%)
>> >>>>         Hdfs split stats (<volume id>:<# splits>/<split
lengths>):
>> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84
MB
>> >>>>         MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB
>> >>>>         ThreadUsage(500.000ms): 1, 1, 1
>> >>>>          - AverageThreadTokens: 1.00
>> >>>>          - BloomFilterBytes: 0
>> >>>>          - PeakMemoryUsage: 6.17 MB (6473729)
>> >>>>          - PeakReservation: 0
>> >>>>          - PeakUsedReservation: 0
>> >>>>          - PerHostPeakMemUsage: 107.32 MB (112537640)
>> >>>>          - RowsProduced: 29.97K (29969)
>> >>>>          - TotalNetworkReceiveTime: 0.000ns
>> >>>>          - TotalNetworkSendTime: 1s260ms
>> >>>>          - TotalStorageWaitTime: 130.815ms
>> >>>>          - TotalThreadsInvoluntaryContextSwitches: 15 (15)
>> >>>>          - TotalThreadsTotalWallClockTime: 1s586ms
>> >>>>            - TotalThreadsSysTime: 11.995ms
>> >>>>            - TotalThreadsUserTime: 146.975ms
>> >>>>          - TotalThreadsVoluntaryContextSwitches: 197 (197)
>> >>>>         Fragment Instance Lifecycle Timings:
>> >>>>            - ExecTime: 1s354ms
>> >>>>              - ExecTreeExecTime: 67.791ms
>> >>>>            - OpenTime: 52.012ms
>> >>>>              - ExecTreeOpenTime: 101.232us
>> >>>>            - PrepareTime: 45.946ms
>> >>>>              - ExecTreePrepareTime: 309.644us
>> >>>>         DataStreamSender (dst_id=11):(Total: 38.995ms, non-child:
>> >>>> 38.995ms, % non-child: 100.00%)
>> >>>>            - BytesSent: 35.95 MB (37697260)
>> >>>>            - NetworkThroughput(*): 2.14 GB/sec
>> >>>>            - OverallThroughput: 921.92 MB/sec
>> >>>>            - PeakMemoryUsage: 59.38 KB (60800)
>> >>>>            - RowsReturned: 29.97K (29969)
>> >>>>            - SerializeBatchTime: 18.055ms
>> >>>>            - TransmitDataRPCTime: 16.428ms
>> >>>>            - UncompressedRowBatchSize: 70.83 MB (74269800)
>> >>>>         CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child:
>> >>>> 100.00%)
>> >>>>            - CodegenTime: 2.224ms
>> >>>>            - CompileTime: 15.517ms
>> >>>>            - LoadTime: 0.000ns
>> >>>>            - ModuleBitcodeSize: 1.96 MB (2050180)
>> >>>>            - NumFunctions: 16 (16)
>> >>>>            - NumInstructions: 250 (250)
>> >>>>            - OptimizationTime: 35.099ms
>> >>>>            - PeakMemoryUsage: 125.00 KB (128000)
>> >>>>            - PrepareTime: 38.847ms
>> >>>>         SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms,
%
>> >>>> non-child: 18.53%)
>> >>>>            - PeakMemoryUsage: 636.17 KB (651439)
>> >>>>            - RowsReturned: 29.70K (29696)
>> >>>>            - RowsReturnedRate: 435.89 K/sec
>> >>>>           NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child:
>> >>>> 27.325ms, % non-child: 77.31%)
>> >>>>              - BuildRows: 0 (0)
>> >>>>              - BuildTime: 0.000ns
>> >>>>              - PeakMemoryUsage: 24.00 KB (24576)
>> >>>>              - ProbeRows: 29.97K (29969)
>> >>>>              - ProbeTime: 0.000ns
>> >>>>              - RowsReturned: 15.26M (15256601)
>> >>>>              - RowsReturnedRate: 431.62 M/sec
>> >>>>             Nested Loop Join Builder:
>> >>>>                - PeakMemoryUsage: 8.00 KB (8192)
>> >>>>             SINGULAR_ROW_SRC_NODE (id=7):
>> >>>>                - PeakMemoryUsage: 0
>> >>>>                - RowsReturned: 0 (0)
>> >>>>                - RowsReturnedRate: 0
>> >>>>           UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms,
%
>> >>>> non-child: 100.00%)
>> >>>>              - AvgCollectionSize: 1.00
>> >>>>              - MaxCollectionSize: 1 (1)
>> >>>>              - MinCollectionSize: 1 (1)
>> >>>>              - NumCollections: 29.97K (29969)
>> >>>>              - PeakMemoryUsage: 0
>> >>>>              - RowsReturned: 1 (1)
>> >>>>              - RowsReturnedRate: 124.00 /sec
>> >>>>         HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms,
>> >>>> %
>> >>>> non-child: 100.00%)
>> >>>>           Hdfs split stats (<volume id>:<# splits>/<split
lengths>):
>> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84
MB
>> >>>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled:
7 out
>> >>>> of
>> >>>> 7
>> >>>>           Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
3:0%
>> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
>> >>>> 16:0%
>> >>>>           File Formats: PARQUET/SNAPPY:42
>> >>>>           BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB
>> >>>>            - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms
;
>> >>>> Max:
>> >>>> 10.464ms ; Number of samples: 7)
>> >>>>            - AverageHdfsReadThreadConcurrency: 0.00
>> >>>>            - AverageScannerThreadConcurrency: 0.00
>> >>>>            - BytesRead: 1.75 MB (1833560)
>> >>>>            - BytesReadDataNodeCache: 0
>> >>>>            - BytesReadLocal: 1.75 MB (1833560)
>> >>>>            - BytesReadRemoteUnexpected: 0
>> >>>>            - BytesReadShortCircuit: 1.75 MB (1833560)
>> >>>>            - CachedFileHandlesHitCount: 0 (0)
>> >>>>            - CachedFileHandlesMissCount: 49 (49)
>> >>>>            - DecompressionTime: 1.758ms
>> >>>>            - MaxCompressedTextFileLength: 0
>> >>>>            - NumColumns: 6 (6)
>> >>>>            - NumDictFilteredRowGroups: 0 (0)
>> >>>>            - NumDisksAccessed: 5 (5)
>> >>>>            - NumRowGroups: 7 (7)
>> >>>>            - NumScannerThreadsStarted: 7 (7)
>> >>>>            - NumScannersWithNoReads: 0 (0)
>> >>>>            - NumStatsFilteredRowGroups: 0 (0)
>> >>>>            - PeakMemoryUsage: 5.82 MB (6098991)
>> >>>>            - PerReadThreadRawHdfsThroughput: 621.09 MB/sec
>> >>>>            - RemoteScanRanges: 0 (0)
>> >>>>            - RowBatchQueueGetWaitTime: 17.938ms
>> >>>>            - RowBatchQueuePutWaitTime: 0.000ns
>> >>>>            - RowsRead: 65.17K (65166)
>> >>>>            - RowsReturned: 29.97K (29969)
>> >>>>            - RowsReturnedRate: 1.49 M/sec
>> >>>>            - ScanRangesComplete: 7 (7)
>> >>>>            - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>> >>>>            - ScannerThreadsTotalWallClockTime: 179.698ms
>> >>>>              - MaterializeTupleTime(*): 13.922ms
>> >>>>              - ScannerThreadsSysTime: 2.997ms
>> >>>>              - ScannerThreadsUserTime: 29.993ms
>> >>>>            - ScannerThreadsVoluntaryContextSwitches: 130 (130)
>> >>>>            - TotalRawHdfsReadTime(*): 2.815ms
>> >>>>            - TotalReadThroughput: 1.17 MB/sec
>> >>>>       Instance 744de1b6228736fa:b54bfaa700000008
>> >>>> (host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child:
>> >>>> 1s305ms, %
>> >>>> non-child: 90.01%)
>> >>>>         Hdfs split stats (<volume id>:<# splits>/<split
lengths>):
>> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
>> >>>>         MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB
>> >>>>         ThreadUsage(500.000ms): 1, 1, 1
>> >>>>          - AverageThreadTokens: 1.00
>> >>>>          - BloomFilterBytes: 0
>> >>>>          - PeakMemoryUsage: 6.69 MB (7017157)
>> >>>>          - PeakReservation: 0
>> >>>>          - PeakUsedReservation: 0
>> >>>>          - PerHostPeakMemUsage: 106.39 MB (111556990)
>> >>>>          - RowsProduced: 42.68K (42681)
>> >>>>          - TotalNetworkReceiveTime: 0.000ns
>> >>>>          - TotalNetworkSendTime: 1s315ms
>> >>>>          - TotalStorageWaitTime: 201.411ms
>> >>>>          - TotalThreadsInvoluntaryContextSwitches: 43 (43)
>> >>>>          - TotalThreadsTotalWallClockTime: 1s658ms
>> >>>>            - TotalThreadsSysTime: 6.997ms
>> >>>>            - TotalThreadsUserTime: 108.980ms
>> >>>>          - TotalThreadsVoluntaryContextSwitches: 179 (179)
>> >>>>         Fragment Instance Lifecycle Timings:
>> >>>>            - ExecTime: 1s411ms
>> >>>>              - ExecTreeExecTime: 68.017ms
>> >>>>            - OpenTime: 19.935ms
>> >>>>              - ExecTreeOpenTime: 48.883us
>> >>>>            - PrepareTime: 19.160ms
>> >>>>              - ExecTreePrepareTime: 134.576us
>> >>>>         DataStreamSender (dst_id=11):(Total: 40.671ms, non-child:
>> >>>> 40.671ms, % non-child: 100.00%)
>> >>>>            - BytesSent: 51.36 MB (53856680)
>> >>>>            - NetworkThroughput(*): 2.44 GB/sec
>> >>>>            - OverallThroughput: 1.23 GB/sec
>> >>>>            - PeakMemoryUsage: 59.38 KB (60800)
>> >>>>            - RowsReturned: 42.68K (42681)
>> >>>>            - SerializeBatchTime: 18.299ms
>> >>>>            - TransmitDataRPCTime: 20.591ms
>> >>>>            - UncompressedRowBatchSize: 100.88 MB (105782860)
>> >>>>         CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child:
>> >>>> 100.00%)
>> >>>>            - CodegenTime: 828.865us
>> >>>>            - CompileTime: 6.170ms
>> >>>>            - LoadTime: 0.000ns
>> >>>>            - ModuleBitcodeSize: 1.96 MB (2050180)
>> >>>>            - NumFunctions: 16 (16)
>> >>>>            - NumInstructions: 250 (250)
>> >>>>            - OptimizationTime: 12.988ms
>> >>>>            - PeakMemoryUsage: 125.00 KB (128000)
>> >>>>            - PrepareTime: 16.608ms
>> >>>>         SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms,
%
>> >>>> non-child: 18.27%)
>> >>>>            - PeakMemoryUsage: 689.95 KB (706507)
>> >>>>            - RowsReturned: 41.98K (41984)
>> >>>>            - RowsReturnedRate: 616.99 K/sec
>> >>>>           NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child:
>> >>>> 29.422ms, % non-child: 73.86%)
>> >>>>              - BuildRows: 0 (0)
>> >>>>              - BuildTime: 0.000ns
>> >>>>              - PeakMemoryUsage: 24.00 KB (24576)
>> >>>>              - ProbeRows: 42.68K (42681)
>> >>>>              - ProbeTime: 0.000ns
>> >>>>              - RowsReturned: 21.76M (21760053)
>> >>>>              - RowsReturnedRate: 546.22 M/sec
>> >>>>             Nested Loop Join Builder:
>> >>>>                - PeakMemoryUsage: 8.00 KB (8192)
>> >>>>             SINGULAR_ROW_SRC_NODE (id=7):
>> >>>>                - PeakMemoryUsage: 0
>> >>>>                - RowsReturned: 0 (0)
>> >>>>                - RowsReturnedRate: 0
>> >>>>           UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms,
%
>> >>>> non-child: 100.00%)
>> >>>>              - AvgCollectionSize: 1.00
>> >>>>              - MaxCollectionSize: 1 (1)
>> >>>>              - MinCollectionSize: 1 (1)
>> >>>>              - NumCollections: 42.68K (42681)
>> >>>>              - PeakMemoryUsage: 0
>> >>>>              - RowsReturned: 1 (1)
>> >>>>              - RowsReturnedRate: 96.00 /sec
>> >>>>         HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms,
>> >>>> %
>> >>>> non-child: 100.00%)
>> >>>>           Hdfs split stats (<volume id>:<# splits>/<split
lengths>):
>> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
>> >>>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled:
6 out
>> >>>> of
>> >>>> 6
>> >>>>           Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
3:0%
>> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
>> >>>> 16:0%
>> >>>> 17:0%
>> >>>>           File Formats: PARQUET/SNAPPY:36
>> >>>>           BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB
>> >>>>            - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms
;
>> >>>> Max:
>> >>>> 42.732ms ; Number of samples: 6)
>> >>>>            - AverageHdfsReadThreadConcurrency: 0.00
>> >>>>            - AverageScannerThreadConcurrency: 0.00
>> >>>>            - BytesRead: 2.08 MB (2186159)
>> >>>>            - BytesReadDataNodeCache: 0
>> >>>>            - BytesReadLocal: 2.08 MB (2186159)
>> >>>>            - BytesReadRemoteUnexpected: 0
>> >>>>            - BytesReadShortCircuit: 2.08 MB (2186159)
>> >>>>            - CachedFileHandlesHitCount: 0 (0)
>> >>>>            - CachedFileHandlesMissCount: 42 (42)
>> >>>>            - DecompressionTime: 1.480ms
>> >>>>            - MaxCompressedTextFileLength: 0
>> >>>>            - NumColumns: 6 (6)
>> >>>>            - NumDictFilteredRowGroups: 0 (0)
>> >>>>            - NumDisksAccessed: 4 (4)
>> >>>>            - NumRowGroups: 6 (6)
>> >>>>            - NumScannerThreadsStarted: 6 (6)
>> >>>>            - NumScannersWithNoReads: 0 (0)
>> >>>>            - NumStatsFilteredRowGroups: 0 (0)
>> >>>>            - PeakMemoryUsage: 6.42 MB (6730467)
>> >>>>            - PerReadThreadRawHdfsThroughput: 77.02 MB/sec
>> >>>>            - RemoteScanRanges: 0 (0)
>> >>>>            - RowBatchQueueGetWaitTime: 13.979ms
>> >>>>            - RowBatchQueuePutWaitTime: 0.000ns
>> >>>>            - RowsRead: 89.91K (89910)
>> >>>>            - RowsReturned: 42.68K (42681)
>> >>>>            - RowsReturnedRate: 2.71 M/sec
>> >>>>            - ScanRangesComplete: 6 (6)
>> >>>>            - ScannerThreadsInvoluntaryContextSwitches: 3 (3)
>> >>>>            - ScannerThreadsTotalWallClockTime: 227.825ms
>> >>>>

Mime
View raw message