drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venki Korukanti (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-2730) Use different paths for ExternalSort spill files
Date Tue, 14 Apr 2015 00:34:12 GMT

    [ https://issues.apache.org/jira/browse/DRILL-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14493365#comment-14493365
] 

Venki Korukanti commented on DRILL-2730:
----------------------------------------

Looks good. In utility method, we try to create the same formatted string {{queryId/major_fragment_1/minor_fragment_3/operator_4}}
for every spill file. I think it is better to create once in  constructor and use it like
{{new Path(new Path(dirs.next(), formattedStringCreatedInCtor), spillNumber)}}. We can get
rid of the utility method getFileName().

> Use different paths for ExternalSort spill files
> ------------------------------------------------
>
>                 Key: DRILL-2730
>                 URL: https://issues.apache.org/jira/browse/DRILL-2730
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>            Reporter: Steven Phillips
>            Assignee: Venki Korukanti
>             Fix For: 0.9.0
>
>         Attachments: DRILL-2730.patch
>
>
> Currently, all of the spill files for a drillbit end up in the same directory (or directories
if multiple are configured). There is no separation based on query or fragment. This makes
it very difficult to explore the spill files via the filesystem.
> We should create a directory structure that will organize the files more effectively,
create a directory tree with query id, major fragment id, minor fragment id, and operator
id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message