drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhishek Girish (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-3913) Possible memory leak during CTAS using 30 TB TPC-H dataset
Date Wed, 07 Oct 2015 23:52:26 GMT

     [ https://issues.apache.org/jira/browse/DRILL-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Abhishek Girish updated DRILL-3913:
    Attachment: sys.memory.txt

Attached is Sys memory snapshot at the beginning of execution of the failing CTAS query (while
writing table 8 of 8)

> Possible memory leak during CTAS using 30 TB TPC-H dataset
> ----------------------------------------------------------
>                 Key: DRILL-3913
>                 URL: https://issues.apache.org/jira/browse/DRILL-3913
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Flow
>    Affects Versions: 1.2.0
>         Environment: 47 nodes configured with 32 GB Drill Direct memory
>            Reporter: Abhishek Girish
>         Attachments: create_table_sf30000.txt, query_profile.json, sys.memory.txt
> 8 CTAS queries were executed sequentially to write TPC-H text data into Parquet. After
successfully writing a few tables, CTAS failed with OOM.
> Restarting Drillbits fixed the problem and re-run of pending CTAS queries completed.
This process was done twice in-order to complete all 8 tables to be written. Overall source
was 30TB in size. 
> Queries attached. Query profile for one of the CTAS which failed is attached. Logs indicated
that the Drillbit was out of Direct Memory. 
> Can share more details as required.

This message was sent by Atlassian JIRA

View raw message