drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3732) Drill leaks memory if external sort hits out of disk space exception
Date Wed, 16 Sep 2015 17:54:45 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790800#comment-14790800
] 

ASF GitHub Bot commented on DRILL-3732:
---------------------------------------

Github user adeneche commented on a diff in the pull request:

    https://github.com/apache/drill/pull/147#discussion_r39662668
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
---
    @@ -514,13 +516,20 @@ public BatchGroup mergeAndSpill(LinkedList<BatchGroup> batchGroups)
throws Schem
             outputContainer.setRecordCount(count);
             newGroup.addBatch(outputContainer);
           }
    +      injector.injectChecked(context.getExecutionControls(), INTERRUPTION_WHILE_SPILLING,
IOException.class);
           newGroup.closeOutputStream();
    -      for (BatchGroup group : batchGroupList) {
    -        group.cleanup();
    -      }
    -      hyperBatch.clear();
    +      threw = false;
    --- End diff --
    
    `threw = false;` needs to be after any statement that could throw. I will add this in
it's comment


> Drill leaks memory if external sort hits out of disk space exception
> --------------------------------------------------------------------
>
>                 Key: DRILL-3732
>                 URL: https://issues.apache.org/jira/browse/DRILL-3732
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>    Affects Versions: 1.2.0
>            Reporter: Victoria Markman
>            Assignee: Aman Sinha
>            Priority: Critical
>             Fix For: 1.2.0
>
>         Attachments: drillbit.log
>
>
> Ran into it when running CTAS with partition by.
> Here is what reproduction looks like:
> {code}
> 0: jdbc:drill:schema=dfs> create table store_sales_4(ss_item_sk, ss_customer_sk, ss_cdemo_sk,
ss_hdemo_sk, s_sold_date_sk, ss_promo_sk) partition by (ss_promo_sk) as
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . >     case when columns[2] = '' then cast(null as varchar(100))
else cast(columns[2] as varchar(100)) end,
> . . . . . . . . . . . . >     case when columns[3] = '' then cast(null as varchar(100))
else cast(columns[3] as varchar(100)) end,
> . . . . . . . . . . . . >     case when columns[4] = '' then cast(null as varchar(100))
else cast(columns[4] as varchar(100)) end, 
> . . . . . . . . . . . . >     case when columns[5] = '' then cast(null as varchar(100))
else cast(columns[5] as varchar(100)) end, 
> . . . . . . . . . . . . >     case when columns[0] = '' then cast(null as varchar(100))
else cast(columns[0] as varchar(100)) end, 
> . . . . . . . . . . . . >     case when columns[8] = '' then cast(null as varchar(100))
else cast(columns[8] as varchar(100)) end
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . >          `store_sales.dat` ss     
> . . . . . . . . . . . . > ;
> Error: SYSTEM ERROR: IllegalStateException: Failure while closing accountor.  Expected
private and shared pools to be set to initial values.  However, one or more were not.  Stats
are
> 	zone	init	allocated	delta 
> 	private	10000000	9680512	319488 
> 	shared	10000000	10000000	0.
> Fragment 1:21
> [Error Id: bd0d7d59-8693-476b-8671-70f0b2e7a176 on atsqa4-133.qa.lab:31010] (state=,code=0)
> {code}
> Setup:
>     single node
>     8GB direct memory
>     4GB heap memory
>    store_sales.dat is a file from TPCDS SF100
> drillbit.log attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message