impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (IMPALA-3110) Incorrect timing of scan node threads
Date Mon, 12 Jun 2017 21:20:00 GMT

     [ https://issues.apache.org/jira/browse/IMPALA-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tim Armstrong resolved IMPALA-3110.
-----------------------------------
    Resolution: Invalid

> Incorrect timing of scan node threads
> -------------------------------------
>
>                 Key: IMPALA-3110
>                 URL: https://issues.apache.org/jira/browse/IMPALA-3110
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.5.0
>            Reporter: Mostafa Mokhtar
>            Priority: Minor
>              Labels: supportability
>         Attachments: selectivity_20.sql.1.out
>
>
> Repro 
> Create household_demographics2 table 
> {code}
> create table household_demographics2 stored as parquet as select hd_demo_sk, hd_demo_sk
hd_demo_sk2 from household_demographics;
> {code}
> {code}
> set NUM_SCANNER_THREADS=1;
> select straight_join
>     count(*) as c
> from
>     store_sales,
>     time_dim,
>     store,
>     household_demographics2
> where
>     store_sales.ss_sold_time_sk = time_dim.t_time_sk
>         and store_sales.ss_store_sk = store.s_store_sk
>         and store_sales.ss_hdemo_sk = hd_demo_sk
>         and ss_sold_date_sk between 2451450 and 2451879
>         and hd_demo_sk2 < 7200
> {code}
> Profile snapshot, the scan ran with a single scanner thread and ScannerThreadsUserTime
is 52s172ms while the scan node completed in 1.5s
> {code}
>       HDFS_SCAN_NODE (id=0):(Total: 1s516ms, non-child: 1s516ms, % non-child: 100.00%)
>          - AverageHdfsReadThreadConcurrency: 0.01 
>          - AverageScannerThreadConcurrency: 1.00 
>          - BytesRead: 812.96 MB (852446917)
>          - BytesReadDataNodeCache: 0
>          - BytesReadLocal: 812.96 MB (852446917)
>          - BytesReadRemoteUnexpected: 0
>          - BytesReadShortCircuit: 812.96 MB (852446917)
>          - DecompressionTime: 2s711ms
>          - MaxCompressedTextFileLength: 0
>          - NumColumns: 3 (3)
>          - NumDisksAccessed: 10 (10)
>          - NumRowGroups: 85 (85)
>          - NumScannerThreadsStarted: 1 (1)
>          - PeakMemoryUsage: 28.55 MB (29933706)
>          - PerReadThreadRawHdfsThroughput: 2.18 GB/sec
>          - RemoteScanRanges: 0 (0)
>          - RowsRead: 531.05M (531054205)
>          - RowsReturned: 531.05M (531054205)
>          - RowsReturnedRate: 351.15 M/sec
>          - ScanRangesComplete: 85 (85)
>          - ScannerThreadsInvoluntaryContextSwitches: 5.87K (5872)
>          - ScannerThreadsTotalWallClockTime: 58s530ms
>            - MaterializeTupleTime(*): 54s743ms
>            - ScannerThreadsSysTime: 147.477ms
>            - ScannerThreadsUserTime: 52s172ms
>          - ScannerThreadsVoluntaryContextSwitches: 274.76K (274759)
>          - TotalRawHdfsReadTime(*): 365.443ms
>          - TotalReadThroughput: 13.79 MB/sec
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message