impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bharath Vissapragada (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-1427: Improvements to "Unknown disk-ID" warning
Date Mon, 06 Feb 2017 21:46:14 GMT
Bharath Vissapragada has posted comments on this change.

Change subject: IMPALA-1427: Improvements to "Unknown disk-ID" warning
......................................................................


Patch Set 7:

Thanks for the review Dan. I'll wait for Alex's +2.

I ran a quick test against an S3 table and I don't see missing disk ID warnings.

[localhost:21000] > show create table tpch.customer;
Query: show create table tpch.customer
+---------------------------------------------------------------------------------------------------------------------+
| result                                                                                 
                            |
+---------------------------------------------------------------------------------------------------------------------+
| CREATE EXTERNAL TABLE tpch.customer (                                                  
                            |
|   c_custkey BIGINT,                                                                    
                            |
|   c_name STRING,                                                                       
                            |
|   c_address STRING,                                                                    
                            |
|   c_nationkey SMALLINT,                                                                
                            |
|   c_phone STRING,                                                                      
                            |
|   c_acctbal DECIMAL(12,2),                                                             
                            |
|   c_mktsegment STRING,                                                                 
                            |
|   c_comment STRING                                                                     
                            |
| )                                                                                      
                            |
| ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'                                          
                            |
| WITH SERDEPROPERTIES ('serialization.format'='|', 'field.delim'='|')                   
                            |
| STORED AS TEXTFILE                                                                     
                            |
| LOCATION 's3a://impala-cdh5-trunk/test-warehouse/tpch.customer'                        
                            |
| TBLPROPERTIES ('numFiles'='0', 'COLUMN_STATS_ACCURATE'='true', 'numRows'='-1', 'totalSize'='0',
'rawDataSize'='-1') |
+---------------------------------------------------------------------------------------------------------------------+
Fetched 1 row(s) in 0.01s
[localhost:21000] > select count(*) from tpch.customer;
Query: select count(*) from tpch.customer
Query submitted at: 2017-02-06 13:43:02 (Coordinator: http://optimus:25000)
Query progress can be monitored at: http://localhost:25000/query_plan?query_id=4e4afd8ed0ed8a76:16fd658200000000
+----------+
| count(*) |
+----------+
| 150000   |
+----------+
Fetched 1 row(s) in 0.88s
[localhost:21000] > explain select count(*) from tpch.customer;
Query: explain select count(*) from tpch.customer
+------------------------------------------------------------------------------------+
| Explain String                                                                     |
+------------------------------------------------------------------------------------+
| Estimated Per-Host Requirements: Memory=74.00MB VCores=1                           |
| WARNING: The following tables are missing relevant table and/or column statistics. |
| tpch.customer                                                                      |
|                                                                                    |
| PLAN-ROOT SINK                                                                     |
| |                                                                                  |
| 03:AGGREGATE [FINALIZE]                                                            |
| |  output: count:merge(*)                                                          |
| |                                                                                  |
| 02:EXCHANGE [UNPARTITIONED]                                                        |
| |                                                                                  |
| 01:AGGREGATE                                                                       |
| |  output: count(*)                                                                |
| |                                                                                  |
| 00:SCAN HDFS [tpch.customer]                                                       |
|    partitions=1/1 files=1 size=23.08MB                                             |
+------------------------------------------------------------------------------------+
Fetched 16 row(s) in 0.01s
[localhost:21000] > set explain_level=3;
EXPLAIN_LEVEL set to 3
[localhost:21000] > explain select count(*) from tpch.customer;
Query: explain select count(*) from tpch.customer
+------------------------------------------------------------------------------------+
| Explain String                                                                     |
+------------------------------------------------------------------------------------+
| Estimated Per-Host Requirements: Memory=74.00MB VCores=1                           |
| WARNING: The following tables are missing relevant table and/or column statistics. |
| tpch.customer                                                                      |
|                                                                                    |
| F01:PLAN FRAGMENT [UNPARTITIONED]                                                  |
|   PLAN-ROOT SINK                                                                   |
|   |                                                                                |
|   03:AGGREGATE [FINALIZE]                                                          |
|   |  output: count:merge(*)                                                        |
|   |  hosts=1 per-host-mem=unavailable                                              |
|   |  tuple-ids=1 row-size=8B cardinality=1                                         |
|   |                                                                                |
|   02:EXCHANGE [UNPARTITIONED]                                                      |
|      hosts=1 per-host-mem=unavailable                                              |
|      tuple-ids=1 row-size=8B cardinality=1                                         |
|                                                                                    |
| F00:PLAN FRAGMENT [RANDOM]                                                         |
|   DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=02, UNPARTITIONED]                       |
|   01:AGGREGATE                                                                     |
|   |  output: count(*)                                                              |
|   |  hosts=1 per-host-mem=10.00MB                                                  |
|   |  tuple-ids=1 row-size=8B cardinality=1                                         |
|   |                                                                                |
|   00:SCAN HDFS [tpch.customer, RANDOM]                                             |
|      partitions=1/1 files=1 size=23.08MB                                           |
|      table stats: unavailable                                                      |
|      column stats: all                                                             |
|      hosts=1 per-host-mem=64.00MB                                                  |
|      tuple-ids=0 row-size=0B cardinality=unavailable                               |
+------------------------------------------------------------------------------------+
Fetched 29 row(s) in 0.01s

Profile snippet-

    Planner Timeline: 2.947ms
       - Analysis finished: 879.310us (879.310us)
       - Equivalence classes computed: 943.765us (64.455us)
       - Single node plan created: 1.337ms (393.340us)
       - Runtime filters computed: 1.357ms (20.630us)
       - Distributed plan created: 1.546ms (188.497us)
       - Planning finished: 2.947ms (1.401ms)
    Query Timeline: 623.505ms
       - Query submitted: 38.990us (38.990us)
       - Planning finished: 3.766ms (3.727ms)
       - Submit for admission: 3.946ms (180.034us)
       - Completed admission: 3.987ms (41.480us)
       - Ready to start 2 fragment instances: 4.125ms (137.963us)
       - All 2 fragment instances started: 4.729ms (603.115us)
       - Rows available: 543.038ms (538.309ms)
       - First row fetched: 620.644ms (77.606ms)
       - Unregister query: 622.297ms (1.652ms)
     - ComputeScanRangeAssignmentTimer: 40.299us
  ImpalaServer:
     - ClientFetchWaitTimer: 78.305ms
     - RowMaterializationTimer: 934.632us
  Execution Profile 3549f42079a127df:70e6a82900000000:(Total: 539.718ms, non-child: 0.000ns,
% non-child: 0.00%)
    Number of filters: 0
    Filter routing table: 
 ID  Src. Node  Tgt. Node(s)  Targets  Target type  Partition filter  Pending (Expected) 
First arrived  Completed   Enabled
----------------------------------------------------------------------------------------------------------------------------

    Fragment instance start latencies: Count: 2, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile:
1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms
    Per Node Peak Memory Usage: optimus:22000(24.06 MB) 
     - FiltersReceived: 0 (0)
     - FinalizationTimer: 0.000ns

-- 
To view, visit http://gerrit.cloudera.org:8080/5828
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Iddb132ff7ad66f3291b93bf9d8061bd0525ef1b2
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-HasComments: No

Mime
View raw message