drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-2100) Drill not deleting spooling files
Date Thu, 21 Apr 2016 13:13:25 GMT

    [ https://issues.apache.org/jira/browse/DRILL-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251872#comment-15251872
] 

ASF GitHub Bot commented on DRILL-2100:
---------------------------------------

Github user adeneche commented on a diff in the pull request:

    https://github.com/apache/drill/pull/454#discussion_r60576483
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
---
    @@ -223,7 +227,21 @@ public void close() {
             if (mSorter != null) {
               mSorter.clear();
             }
    -
    +        for(Iterator iter = this.currSpillDirs.iterator(); iter.hasNext(); iter.remove())
{
    +            Path path = (Path)iter.next();
    +            try {
    +                if (fs != null && path != null && fs.exists(path)) {
    +                    if (fs.delete(path, true)) {
    +                        fs.cancelDeleteOnExit(path);
    +                    }
    +                }
    +            } catch (IOException e) {
    +                // since this is meant to be used in a batches's cleanup, we don't propagate
the exception
    +                final String message = "Unable to delete spill directory \"" + path +
"\"";
    +                System.err.println(message + '.');
    --- End diff --
    
    I don't think printing to stderr is a good idea, logging a warning should be enough


> Drill not deleting spooling files
> ---------------------------------
>
>                 Key: DRILL-2100
>                 URL: https://issues.apache.org/jira/browse/DRILL-2100
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>    Affects Versions: 0.8.0
>            Reporter: Abhishek Girish
>            Assignee: Deneche A. Hakim
>             Fix For: 1.7.0
>
>
> Currently, after forcing queries to use an external sort by switching off hash join/agg
causes spill-to-disk files accumulating. 
> This causes issues with disk space availability when the spill is configured to be on
the local file system (/tmp/drill). Also not optimal when configured to use DFS (custom).

> Drill must clean up all temporary files created after a query completes or after a drillbit
restart. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message