drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chun Chang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-4190) TPCDS queries are running out of memory when hash join is disabled
Date Sat, 12 Dec 2015 00:20:46 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053838#comment-15053838
] 

Chun Chang commented on DRILL-4190:
-----------------------------------

My similar test on sf1 data (much smaller dataset) shown more verification failures.

./run.sh -s Advanced/tpcds/tpcds_sf1/original/parquet -g smoke,regression,functional -t 300
-n 1 -d

{noformat}
Verification Failures
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query73.sql
Query:
-- start query 73 in stream 0 using template query73.tpl
SELECT c_last_name,
       c_first_name,
       c_salutation,
       c_preferred_cust_flag,
       ss_ticket_number,
       cnt
FROM   (SELECT ss_ticket_number,
               ss_customer_sk,
               Count(*) cnt
        FROM   store_sales,
               date_dim,
               store,
               household_demographics
        WHERE  store_sales.ss_sold_date_sk = date_dim.d_date_sk
               AND store_sales.ss_store_sk = store.s_store_sk
               AND store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk
               AND date_dim.d_dom BETWEEN 1 AND 2
               AND ( household_demographics.hd_buy_potential = '>10000'
                      OR household_demographics.hd_buy_potential = '0-500' )
               AND household_demographics.hd_vehicle_count > 0
               AND CASE
                     WHEN household_demographics.hd_vehicle_count > 0 THEN
                     household_demographics.hd_dep_count /
                     household_demographics.hd_vehicle_count
                     ELSE NULL
                   END > 1
               AND date_dim.d_year IN ( 2000, 2000 + 1, 2000 + 2 )
               AND store.s_county IN ( 'Williamson County', 'Williamson County',
                                       'Williamson County',
                                                             'Williamson County'
                                     )
        GROUP  BY ss_ticket_number,
                  ss_customer_sk) dj,
       customer
WHERE  ss_customer_sk = c_customer_sk
       AND cnt BETWEEN 1 AND 5
ORDER  BY cnt DESC,
          c_last_name ASC
         Expected number of rows: 5
Actual number of rows from Drill: 6
         Number of matching rows: 5
          Number of rows missing: 0
       Number of rows unexpected: 1

These rows are not expected (first 10):
Sanderson	Joel	Sir	Y	228345	3
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query62.sql
Query:
-- start query 62 in stream 0 using template query62.tpl
SELECT Substr(w_warehouse_name, 1, 20),
               sm_type,
               web_name,
               Sum(CASE
                     WHEN ( ws_ship_date_sk - ws_sold_date_sk <= 30 ) THEN 1
                     ELSE 0
                   END) AS `30 days`,
               Sum(CASE
                     WHEN ( ws_ship_date_sk - ws_sold_date_sk > 30 )
                          AND ( ws_ship_date_sk - ws_sold_date_sk <= 60 ) THEN 1
                     ELSE 0
                   END) AS `31-60 days`,
               Sum(CASE
                     WHEN ( ws_ship_date_sk - ws_sold_date_sk > 60 )
                          AND ( ws_ship_date_sk - ws_sold_date_sk <= 90 ) THEN 1
                     ELSE 0
                   END) AS `61-90 days`,
               Sum(CASE
                     WHEN ( ws_ship_date_sk - ws_sold_date_sk > 90 )
                          AND ( ws_ship_date_sk - ws_sold_date_sk <= 120 ) THEN
                     1
                     ELSE 0
                   END) AS `91-120 days`,
               Sum(CASE
                     WHEN ( ws_ship_date_sk - ws_sold_date_sk > 120 ) THEN 1
                     ELSE 0
                   END) AS `>120 days`
FROM   web_sales,
       warehouse,
       ship_mode,
       web_site,
       date_dim
WHERE  d_month_seq BETWEEN 1222 AND 1222 + 11
       AND ws_ship_date_sk = d_date_sk
       AND ws_warehouse_sk = w_warehouse_sk
       AND ws_ship_mode_sk = sm_ship_mode_sk
       AND ws_web_site_sk = web_site_sk
GROUP  BY Substr(w_warehouse_name, 1, 20),
          sm_type,
          web_name
ORDER  BY Substr(w_warehouse_name, 1, 20),
          sm_type,
          web_name
LIMIT 100
         Expected number of rows: 100
Actual number of rows from Drill: 100
         Number of matching rows: 87
          Number of rows missing: 13
       Number of rows unexpected: 13

These rows are not expected (first 10):
Doors canno	NEXT DAY	site_4	224	245	236	237	0
Doors canno	NEXT DAY	site_2	176	172	181	150	0
Conventional childr	NEXT DAY	site_4	236	224	247	244	0
Conventional childr	NEXT DAY	site_3	201	245	247	237	0
Bad cards must make.	NEXT DAY	site_1	217	218	211	190	0
Bad cards must make.	NEXT DAY	site_4	194	215	201	218	0
Conventional childr	NEXT DAY	site_1	189	170	183	174	0
Bad cards must make.	NEXT DAY	site_0	251	302	235	242	0
Doors canno	NEXT DAY	site_1	203	195	193	213	0
Doors canno	NEXT DAY	site_3	203	218	190	207	0

These rows are missing (first 10):
Conventional childr	NEXT DAY	site_2	280	287	307	310	0 (1 time(s))
Bad cards must make.	NEXT DAY	site_0	276	331	266	271	0 (1 time(s))
Conventional childr	NEXT DAY	site_4	303	284	314	305	0 (1 time(s))
Conventional childr	NEXT DAY	site_3	250	302	305	302	0 (1 time(s))
Bad cards must make.	NEXT DAY	site_4	259	272	273	281	0 (1 time(s))
Doors canno	NEXT DAY	site_4	275	307	297	299	0 (1 time(s))
Doors canno	NEXT DAY	site_3	277	300	276	290	0 (1 time(s))
Bad cards must make.	NEXT DAY	site_3	287	337	340	306	0 (1 time(s))
Conventional childr	NEXT DAY	site_1	278	252	293	253	0 (1 time(s))
Doors canno	NEXT DAY	site_1	243	278	253	277	0 (1 time(s))
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query66.sql
Query:
-- start query 66 in stream 0 using template query66.tpl
SELECT w_warehouse_name,
               w_warehouse_sq_ft,
               w_city,
               w_county,
               w_state,
               w_country,
               ship_carriers,
               year1,
               Sum(jan_sales)                     AS jan_sales,
               Sum(feb_sales)                     AS feb_sales,
               Sum(mar_sales)                     AS mar_sales,
               Sum(apr_sales)                     AS apr_sales,
               Sum(may_sales)                     AS may_sales,
               Sum(jun_sales)                     AS jun_sales,
               Sum(jul_sales)                     AS jul_sales,
               Sum(aug_sales)                     AS aug_sales,
               Sum(sep_sales)                     AS sep_sales,
               Sum(oct_sales)                     AS oct_sales,
               Sum(nov_sales)                     AS nov_sales,
               Sum(dec_sales)                     AS dec_sales,
               Sum(jan_sales / w_warehouse_sq_ft) AS jan_sales_per_sq_foot,
               Sum(feb_sales / w_warehouse_sq_ft) AS feb_sales_per_sq_foot,
               Sum(mar_sales / w_warehouse_sq_ft) AS mar_sales_per_sq_foot,
               Sum(apr_sales / w_warehouse_sq_ft) AS apr_sales_per_sq_foot,
               Sum(may_sales / w_warehouse_sq_ft) AS may_sales_per_sq_foot,
               Sum(jun_sales / w_warehouse_sq_ft) AS jun_sales_per_sq_foot,
               Sum(jul_sales / w_warehouse_sq_ft) AS jul_sales_per_sq_foot,
               Sum(aug_sales / w_warehouse_sq_ft) AS aug_sales_per_sq_foot,
               Sum(sep_sales / w_warehouse_sq_ft) AS sep_sales_per_sq_foot,
               Sum(oct_sales / w_warehouse_sq_ft) AS oct_sales_per_sq_foot,
               Sum(nov_sales / w_warehouse_sq_ft) AS nov_sales_per_sq_foot,
               Sum(dec_sales / w_warehouse_sq_ft) AS dec_sales_per_sq_foot,
               Sum(jan_net)                       AS jan_net,
               Sum(feb_net)                       AS feb_net,
               Sum(mar_net)                       AS mar_net,
               Sum(apr_net)                       AS apr_net,
               Sum(may_net)                       AS may_net,
               Sum(jun_net)                       AS jun_net,
               Sum(jul_net)                       AS jul_net,
               Sum(aug_net)                       AS aug_net,
               Sum(sep_net)                       AS sep_net,
               Sum(oct_net)                       AS oct_net,
               Sum(nov_net)                       AS nov_net,
               Sum(dec_net)                       AS dec_net
FROM   (SELECT w_warehouse_name,
               w_warehouse_sq_ft,
               w_city,
               w_county,
               w_state,
               w_country,
               'ZOUROS'
               || ','
               || 'ZHOU' AS ship_carriers,
               d_year    AS year1,
               Sum(CASE
                     WHEN d_moy = 1 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS jan_sales,
               Sum(CASE
                     WHEN d_moy = 2 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS feb_sales,
               Sum(CASE
                     WHEN d_moy = 3 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS mar_sales,
               Sum(CASE
                     WHEN d_moy = 4 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS apr_sales,
               Sum(CASE
                     WHEN d_moy = 5 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS may_sales,
               Sum(CASE
                     WHEN d_moy = 6 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS jun_sales,
               Sum(CASE
                     WHEN d_moy = 7 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS jul_sales,
               Sum(CASE
                     WHEN d_moy = 8 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS aug_sales,
               Sum(CASE
                     WHEN d_moy = 9 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS sep_sales,
               Sum(CASE
                     WHEN d_moy = 10 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS oct_sales,
               Sum(CASE
                     WHEN d_moy = 11 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS nov_sales,
               Sum(CASE
                     WHEN d_moy = 12 THEN ws_ext_sales_price * ws_quantity
                     ELSE 0
                   END)  AS dec_sales,
               Sum(CASE
                     WHEN d_moy = 1 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS jan_net,
               Sum(CASE
                     WHEN d_moy = 2 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS feb_net,
               Sum(CASE
                     WHEN d_moy = 3 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS mar_net,
               Sum(CASE
                     WHEN d_moy = 4 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS apr_net,
               Sum(CASE
                     WHEN d_moy = 5 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS may_net,
               Sum(CASE
                     WHEN d_moy = 6 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS jun_net,
               Sum(CASE
                     WHEN d_moy = 7 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS jul_net,
               Sum(CASE
                     WHEN d_moy = 8 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS aug_net,
               Sum(CASE
                     WHEN d_moy = 9 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS sep_net,
               Sum(CASE
                     WHEN d_moy = 10 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS oct_net,
               Sum(CASE
                     WHEN d_moy = 11 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS nov_net,
               Sum(CASE
                     WHEN d_moy = 12 THEN ws_net_paid_inc_ship * ws_quantity
                     ELSE 0
                   END)  AS dec_net
        FROM   web_sales,
               warehouse,
               date_dim,
               time_dim,
               ship_mode
        WHERE  ws_warehouse_sk = w_warehouse_sk
               AND ws_sold_date_sk = d_date_sk
               AND ws_sold_time_sk = t_time_sk
               AND ws_ship_mode_sk = sm_ship_mode_sk
               AND d_year = 1998
               AND t_time BETWEEN 7249 AND 7249 + 28800
               AND sm_carrier IN ( 'ZOUROS', 'ZHOU' )
        GROUP  BY w_warehouse_name,
                  w_warehouse_sq_ft,
                  w_city,
                  w_county,
                  w_state,
                  w_country,
                  d_year
        UNION ALL
        SELECT w_warehouse_name,
               w_warehouse_sq_ft,
               w_city,
               w_county,
               w_state,
               w_country,
               'ZOUROS'
               || ','
               || 'ZHOU' AS ship_carriers,
               d_year    AS year1,
               Sum(CASE
                     WHEN d_moy = 1 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS jan_sales,
               Sum(CASE
                     WHEN d_moy = 2 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS feb_sales,
               Sum(CASE
                     WHEN d_moy = 3 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS mar_sales,
               Sum(CASE
                     WHEN d_moy = 4 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS apr_sales,
               Sum(CASE
                     WHEN d_moy = 5 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS may_sales,
               Sum(CASE
                     WHEN d_moy = 6 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS jun_sales,
               Sum(CASE
                     WHEN d_moy = 7 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS jul_sales,
               Sum(CASE
                     WHEN d_moy = 8 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS aug_sales,
               Sum(CASE
                     WHEN d_moy = 9 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS sep_sales,
               Sum(CASE
                     WHEN d_moy = 10 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS oct_sales,
               Sum(CASE
                     WHEN d_moy = 11 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS nov_sales,
               Sum(CASE
                     WHEN d_moy = 12 THEN cs_ext_sales_price * cs_quantity
                     ELSE 0
                   END)  AS dec_sales,
               Sum(CASE
                     WHEN d_moy = 1 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS jan_net,
               Sum(CASE
                     WHEN d_moy = 2 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS feb_net,
               Sum(CASE
                     WHEN d_moy = 3 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS mar_net,
               Sum(CASE
                     WHEN d_moy = 4 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS apr_net,
               Sum(CASE
                     WHEN d_moy = 5 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS may_net,
               Sum(CASE
                     WHEN d_moy = 6 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS jun_net,
               Sum(CASE
                     WHEN d_moy = 7 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS jul_net,
               Sum(CASE
                     WHEN d_moy = 8 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS aug_net,
               Sum(CASE
                     WHEN d_moy = 9 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS sep_net,
               Sum(CASE
                     WHEN d_moy = 10 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS oct_net,
               Sum(CASE
                     WHEN d_moy = 11 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS nov_net,
               Sum(CASE
                     WHEN d_moy = 12 THEN cs_net_paid * cs_quantity
                     ELSE 0
                   END)  AS dec_net
        FROM   catalog_sales,
               warehouse,
               date_dim,
               time_dim,
               ship_mode
        WHERE  cs_warehouse_sk = w_warehouse_sk
               AND cs_sold_date_sk = d_date_sk
               AND cs_sold_time_sk = t_time_sk
               AND cs_ship_mode_sk = sm_ship_mode_sk
               AND d_year = 1998
               AND t_time BETWEEN 7249 AND 7249 + 28800
               AND sm_carrier IN ( 'ZOUROS', 'ZHOU' )
        GROUP  BY w_warehouse_name,
                  w_warehouse_sq_ft,
                  w_city,
                  w_county,
                  w_state,
                  w_country,
                  d_year) x
GROUP  BY w_warehouse_name,
          w_warehouse_sq_ft,
          w_city,
          w_county,
          w_state,
          w_country,
          ship_carriers,
          year1
ORDER  BY w_warehouse_name
LIMIT 100
         Expected number of rows: 5
Actual number of rows from Drill: 5
         Number of matching rows: 4
          Number of rows missing: 1
       Number of rows unexpected: 1

These rows are not expected (first 10):
null	null	Fairview	Williamson County	TN	United States	ZOUROS,ZHOU	1998	5478281.649999999	5175474.01
3601736.340000001	8373195.390000001	4424175.600000001	5516117.37	3135432.2100000004	7614425.779999999
1.4999508999999996E7	5198635.529999999	1741449.64	0.0	null	null	null	null	null	null	null	null
null	null	null	null	6757258.66	8089452.660000001	4893972.33	1.1246446989999996E7	6567308.39
8734110.99	4256927.61	1.1798181320000002E7	2.2307786160000008E7	4910085.009999999	1457070.4000000001
0.0

These rows are missing (first 10):
null	null	Fairview	Williamson County	TN	United States	ZOUROS,ZHOU	1998	2.107754961E7	1.7189227050000004E7
1.7404323700000003E7	1.8908102909999996E7	1.6285662420000002E7	1.8500778700000003E7	1.7322239259999994E7
3.644408868999999E7	4.2712381879999995E7	3.5181388559999995E7	6.750091281E7	6.4034387639999956E7
null	null	null	null	null	null	null	null	null	null	null	null	2.155171713E7	1.912509567E7	1.6574806870000001E7
2.0705973540000007E7	1.6545966379999999E7	2.048446054E7	1.712386645E7	3.929974493999999E7
4.763222143000001E7	3.615190025999999E7	7.358634230000001E7	6.968475126999998E7 (1 time(s))
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query40.sql
Query:
-- start query 40 in stream 0 using template query40.tpl
SELECT
                w_state ,
                i_item_id ,
                Sum(
                CASE
                                WHEN (
                                                                Cast(d_date AS DATE) <
Cast ('2002-06-01' AS DATE)) THEN cs_sales_price - COALESCE(cr_refunded_cash,0)
                                ELSE 0
                END) AS sales_before ,
                Sum(
                CASE
                                WHEN (
                                                                Cast(d_date AS DATE) >=
Cast ('2002-06-01' AS DATE)) THEN cs_sales_price - COALESCE(cr_refunded_cash,0)
                                ELSE 0
                END) AS sales_after
FROM            catalog_sales
LEFT OUTER JOIN catalog_returns
ON              (
                                cs_order_number = cr_order_number
                AND             cs_item_sk = cr_item_sk) ,
                warehouse ,
                item ,
                date_dim
WHERE           i_current_price BETWEEN 0.99 AND             1.49
AND             i_item_sk = cs_item_sk
AND             cs_warehouse_sk = w_warehouse_sk
AND             cs_sold_date_sk = d_date_sk
AND             d_date BETWEEN (Cast ('2002-06-01' AS DATE) - INTERVAL '30' day) AND     
       (
                                cast ('2002-06-01' AS date) + INTERVAL '30' day)
GROUP BY        w_state,
                i_item_id
ORDER BY        w_state,
                i_item_id
LIMIT 100
         Expected number of rows: 100
Actual number of rows from Drill: 100
         Number of matching rows: 59
          Number of rows missing: 41
       Number of rows unexpected: 41

These rows are not expected (first 10):
TN	AAAAAAAAAFOAAAAA	180.96	66.76
TN	AAAAAAAACHDEAAAA	0.0	49.51
TN	AAAAAAAACKEBAAAA	85.37	0.0
TN	AAAAAAAAAIKBAAAA	0.0	59.209999999999994
TN	AAAAAAAACKCAAAAA	76.77000000000001	0.0
TN	AAAAAAAACBBBAAAA	269.48	44.96
TN	AAAAAAAABKMCAAAA	166.61	1.21
TN	AAAAAAAAALJAAAAA	0.0	96.77
TN	AAAAAAAAAAKBAAAA	3.07	0.0
TN	AAAAAAAACJNCAAAA	0.0	47.89

These rows are missing (first 10):
TN	AAAAAAAAAFOAAAAA	192.45	66.76 (1 time(s))
TN	AAAAAAAAAPFDAAAA	0.0	1.18 (1 time(s))
TN	AAAAAAAACHDEAAAA	48.33	49.51 (1 time(s))
TN	AAAAAAAAAIKBAAAA	0.0	59.269999999999996 (1 time(s))
TN	AAAAAAAACBBBAAAA	325.43999999999994	44.96 (1 time(s))
TN	AAAAAAAABKMCAAAA	208.69	1.21 (1 time(s))
TN	AAAAAAAAALJAAAAA	0.0	255.05 (1 time(s))
TN	AAAAAAAAAAKBAAAA	59.42	0.0 (1 time(s))
TN	AAAAAAAABDCEAAAA	311.01	117.17 (1 time(s))
TN	AAAAAAAACGPCAAAA	120.58	316.33 (1 time(s))
Timeout Failures
----------------------------------------------------------------------------------------------------------------
Summary
----------------------------------------------------------------------------------------------------------------
Execution Failures:
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query21.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query24.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query99.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query26.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query88.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query79.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query97.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query75.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query7.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query59.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query46.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query68.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query30.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query71.sql
Verification Failures
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query73.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query62.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query66.sql
/root/drill-test-framework/framework/resources/Advanced/tpcds/tpcds_sf1/original/parquet/query40.sql
Timeout Failures
----------------------------------------------------------------------------------------------------------------

Passing tests: 29
Execution Failures: 14
VerificationFailures: 4
Timeouts: 0
Canceled: 0
{noformat}

> TPCDS queries are running out of memory when hash join is disabled
> ------------------------------------------------------------------
>
>                 Key: DRILL-4190
>                 URL: https://issues.apache.org/jira/browse/DRILL-4190
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>    Affects Versions: 1.3.0, 1.4.0, 1.5.0
>            Reporter: Victoria Markman
>            Priority: Blocker
>
> TPCDS queries with the latest 1.4.0 release when hash join is disabled:
> 22 queries fail with out of memory 
> 2 wrong results (I did not validate the nature of wrong result yet)
> Only query97.sql is a legitimate failure: we don't support full outer join with the merge
join.
> It is important to understand what has changed between 1.2.0 and 1.4.0 that made these
tests not runnable with the same configuration. 
> Same tests with the same drill configuration pass in 1.2.0 release.
> (I hope I did not make a mistake somewhere in my cluster setup :))
> {code}
> 0: jdbc:drill:schema=dfs> select * from sys.version;
> +-----------------+-------------------------------------------+---------------------------------------------------------------------+----------------------------+--------------+----------------------------+
> |     version     |                 commit_id                 |                     
     commit_message                            |        commit_time         | build_email
 |         build_time         |
> +-----------------+-------------------------------------------+---------------------------------------------------------------------+----------------------------+--------------+----------------------------+
> | 1.4.0-SNAPSHOT  | b9068117177c3b47025f52c00f67938e0c3e4732  | DRILL-4165 Add a precondition
for size of merge join record batch.  | 08.12.2015 @ 01:25:34 UTC  | Unknown      | 08.12.2015
@ 03:36:25 UTC  |
> +-----------------+-------------------------------------------+---------------------------------------------------------------------+----------------------------+--------------+----------------------------+
> 1 row selected (2.211 seconds)
> Execution Failures:
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query50.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query33.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query74.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query68.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query34.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query21.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query46.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query91.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query59.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query3.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query66.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query84.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query97.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query19.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query96.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query43.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query2.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query60.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query79.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query73.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query45.sql
> Verification Failures
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query52.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query40.sql
> Timeout Failures
> ----------------------------------------------------------------------------------------------------------------
> Passing tests: 3
> Execution Failures: 22
> VerificationFailures: 2
> Timeouts: 0
> Canceled: 0
> {code}
> {code}
> 0: jdbc:drill:schema=dfs> select * from sys.version;
> +-----------+----------------+-------------+-------------+------------+
> | commit_id | commit_message | commit_time | build_email | build_time |
> +-----------+----------------+-------------+-------------+------------+
> | f1100a79b4e4fbb1b58b35b0230edff137588777 | DRILL-3947: Use setSafe() for date, time,
timestamp types while populating pruning vector (other types were already using setSafe).
| 19.10.2015 @ 16:02:00 UTC | Unknown | 19.10.2015 @ 16:25:21 UTC |
> +-----------+----------------+-------------+-------------+------------+
> 1 row selected (2.79 seconds)
> PASS (1.543 min) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query68.sql
(connection: 1681915178)
> PASS (29.36 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query34.sql
(connection: 1681915178)
> PASS (3.311 min) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query21.sql
(connection: 1681915178)
> PASS (1.447 min) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query46.sql
(connection: 1681915178)
> PASS (34.53 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query76.sql
(connection: 1681915178)
> PASS (47.13 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query91.sql
(connection: 1681915178)
> PASS (1.151 min) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query59.sql
(connection: 1681915178)
> PASS (32.29 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query3.sql
(connection: 1681915178)
> PASS (1.939 min) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query66.sql
(connection: 1681915178)
> PASS (19.26 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query84.sql
(connection: 1681915178)
> PASS (1.243 min) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query40.sql
(connection: 1681915178)
> [#37] Query failed:
> oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException:
Full outer join not currently supported
> [Error Id: 9a400ac2-3f1d-428c-9dc6-5f556cb520aa on atsqa4-133.qa.lab:31010]
>         at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:118)
>         at oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:110)
>         at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:47)
>         at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:32)
>         at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:61)
>         at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:233)
>         at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:205)
>         at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
>         at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>         at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>         at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
>         at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>         at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>         at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
>         at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>         at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>         at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
>         at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>         at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>         at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>         at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>         at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>         at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
>         at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
>         at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>         at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>         at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>         at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>         at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
> EXECUTION_FAILURE (2.814 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query97.sql
(connection: 1681915178)
> PASS (57.04 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query19.sql
(connection: 1681915178)
> PASS (24.01 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query96.sql
(connection: 1681915178)
> PASS (28.77 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query43.sql
(connection: 1681915178)
> PASS (1.833 min) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query93.sql
(connection: 1681915178)
> PASS (38.84 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
(connection: 1681915178)
> PASS (55.82 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query2.sql
(connection: 1681915178)
> PASS (1.308 min) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query60.sql
(connection: 1681915178)
> PASS (1.116 min) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query79.sql
(connection: 1681915178)
> PASS (27.79 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query73.sql
(connection: 1681915178)
> PASS (39.85 s) /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query45.sql
(connection: 1681915178)
> {code}
> *Cluster configuration:*
> - 4 nodes
> - 48 GB direct memory
> - 10GB memory allocated to sort
> - timeout setup for the framework = 600 seconds
> - queries were executed one at a time
> *System settings:*
> {code}
> 0: jdbc:drill:schema=dfs> select * from sys.options where status like '%CHANGED%';
> +-------------------------------------------+----------+---------+----------+--------------+-------------+-----------+------------+
> |                   name                    |   kind   |  type   |  status  |   num_val
   | string_val  | bool_val  | float_val  |
> +-------------------------------------------+----------+---------+----------+--------------+-------------+-----------+------------+
> | planner.enable_decimal_data_type          | BOOLEAN  | SYSTEM  | CHANGED  | null  
      | null        | true      | null       |
> | planner.enable_hashjoin                   | BOOLEAN  | SYSTEM  | CHANGED  | null  
      | null        | false     | null       |
> | planner.memory.max_query_memory_per_node  | LONG     | SYSTEM  | CHANGED  | 10737418240
 | null        | null      | null       |
> +-------------------------------------------+----------+---------+----------+--------------+-------------+-----------+------------+
> 3 rows selected (3.464 seconds)
> {code}
> TPCDS queries that were executed from the public test framework: 
> ./run.sh -s Advanced/tpcds/tpcds_sf100/original -g smoke -t 600
> More details shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message