impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (JIRA)" <>
Subject [jira] [Resolved] (IMPALA-5161) TPC-DS Q78 with MEM_LIMIT=10GB fails with "Repartitioning did not reduce the size of a spilled partition" on BufferPool dev branch
Date Mon, 17 Apr 2017 18:04:41 GMT


Tim Armstrong resolved IMPALA-5161.
    Resolution: Duplicate

> TPC-DS Q78 with MEM_LIMIT=10GB fails with "Repartitioning did not reduce the size of
a spilled partition" on BufferPool dev branch
> ----------------------------------------------------------------------------------------------------------------------------------
>                 Key: IMPALA-5161
>                 URL:
>             Project: IMPALA
>          Issue Type: Sub-task
>    Affects Versions: Impala 2.9.0
>            Reporter: Mostafa Mokhtar
>            Assignee: Tim Armstrong
>            Priority: Critical
>         Attachments: tpcds_q78_memory_repartitioning_didnt_reduce_num_scanner_threads-1.txt,
> I set the memory limit to 10GB which is fairly low for Q78 and it failed with:
> {code}
> Query (id=8d463de3722df36f:a2dae93a00000000):
>   Summary:
>     Session ID: b54cda644686da93:f1c20afc188e0fa0
>     Session Type: BEESWAX
>     Start Time: 2017-04-04 10:46:17.099246000
>     End Time: 2017-04-04 11:06:04.438426000
>     Query Type: QUERY
>     Query State: EXCEPTION
>     Query Status: 
> Cannot perform aggregation at node with id 30. Repartitioning did not reduce the size
of a spilled partition. Repartitioning level 7. Number of rows 1.
>     Impala Version: impalad version 2.9.0-SNAPSHOT RELEASE (build 07021cc117c56e4bbc0744b342fee6f5a8cce964)
>     User: mmokhtar
>     Connected User: mmokhtar
>     Delegated User: 
>     Network Address:
>     Default Db: tpcds_10000_parquet
>     Sql Statement: with ws as (select d_year AS ws_sold_year, ws_item_sk, ws_bill_customer_sk
ws_customer_sk, sum(ws_quantity) ws_qty, sum(ws_wholesale_cost) ws_wc, sum(ws_sales_price)
ws_sp from web_sales left join web_returns on wr_order_number=ws_order_number and ws_item_sk=wr_item_sk
join date_dim on ws_sold_date_sk = d_date_sk where wr_order_number is null group by d_year,
ws_item_sk, ws_bill_customer_sk ), cs as (select d_year AS cs_sold_year, cs_item_sk, cs_bill_customer_sk
cs_customer_sk, sum(cs_quantity) cs_qty, sum(cs_wholesale_cost) cs_wc, sum(cs_sales_price)
cs_sp from catalog_sales left join catalog_returns on cr_order_number=cs_order_number and
cs_item_sk=cr_item_sk join date_dim on cs_sold_date_sk = d_date_sk where cr_order_number is
null group by d_year, cs_item_sk, cs_bill_customer_sk ), ss as (select d_year AS ss_sold_year,
ss_item_sk, ss_customer_sk, sum(ss_quantity) ss_qty, sum(ss_wholesale_cost) ss_wc, sum(ss_sales_price)
ss_sp from store_sales left join store_returns on sr_ticket_number=ss_ticket_number and ss_item_sk=sr_item_sk
join date_dim on ss_sold_date_sk = d_date_sk where sr_ticket_number is null group by d_year,
ss_item_sk, ss_customer_sk ) select ss_item_sk, round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2)
ratio, ss_qty store_qty, ss_wc store_wholesale_cost, ss_sp store_sales_price, coalesce(ws_qty,0)+coalesce(cs_qty,0)
other_chan_qty, coalesce(ws_wc,0)+coalesce(cs_wc,0) other_chan_wholesale_cost, coalesce(ws_sp,0)+coalesce(cs_sp,0)
other_chan_sales_price from ss left join ws on (ws_sold_year=ss_sold_year and ws_item_sk=ss_item_sk
and ws_customer_sk=ss_customer_sk) left join cs on (cs_sold_year=ss_sold_year and cs_item_sk=cs_item_sk
and cs_customer_sk=ss_customer_sk) where coalesce(ws_qty,0)>0 and coalesce(cs_qty, 0)>0
and ss_sold_year=2000 order by ss_item_sk, ss_qty desc, ss_wc desc, ss_sp desc, other_chan_qty,
other_chan_wholesale_cost, other_chan_sales_price, round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2)
limit 100
>     Coordinator:
>     Query Options (non default): MEM_LIMIT=10737418240
> {code}

This message was sent by Atlassian JIRA

View raw message