drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5274) Exception thrown in Drillbit shutdown in UDF cleanup code
Date Sat, 18 Feb 2017 20:11:44 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15873319#comment-15873319
] 

Paul Rogers commented on DRILL-5274:
------------------------------------

Example case. Query:

{code}
select * from (select * from dfs.`/drill/testdata/250wide.tbl`
    order by columns[0])d where d.columns[0] = 'ljdfhwuehnoiueyf';
{code}

The data file is a large file consisting of a single column 250 characters wide.

Batch as seen by the external sort:

{code}
ExternalSortBatch - Actual batch schema & sizes {
  T0¦¦columns(std col. size: 54, actual col. size: 0, total size: 263942, vector size: 0,
data size: 0, row capacity: 1023, density: 0)
  EXPR$1(std col. size: 54, actual col. size: 250, total size: 282624, vector size: 278528,
data size: 255750, row capacity: 4095, density: 92)
  Records: 1023, Total size: 548614, Row width:538, Density:92}
{code}

The result is that the sort must buffer two copies of the data. Since this is an 18 GB input
file, the sort must buffer (then spill) 36 GB of data. Note that there is actually no column
alias or expression on the column.

> Exception thrown in Drillbit shutdown in UDF cleanup code
> ---------------------------------------------------------
>
>                 Key: DRILL-5274
>                 URL: https://issues.apache.org/jira/browse/DRILL-5274
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.10
>            Reporter: Paul Rogers
>            Assignee: Arina Ielchiieva
>            Priority: Minor
>
> I ran a very simple query: a single-line text file in an embedded Drillbit. The UDF directory
was placed in /tmp. During the run, the directory was deleted. On Drillbit shutdown, the following
occurred:
> {code}
> 25328 DEBUG [main] [org.apache.drill.exec.server.Drillbit] - Shutdown begun.
> 26344 INFO [pool-1-thread-2] [org.apache.drill.exec.rpc.data.DataServer] - closed eventLoopGroup
io.netty.channel.nio.NioEventLoopGroup@7d1c0d85 in 1007 ms
> 26345 INFO [pool-1-thread-1] [org.apache.drill.exec.rpc.user.UserServer] - closed eventLoopGroup
io.netty.channel.nio.NioEventLoopGroup@7cdb3b56 in 1008 ms
> 26345 INFO [pool-1-thread-1] [org.apache.drill.exec.service.ServiceEngine] - closed userServer
in 1009 ms
> 26345 INFO [pool-1-thread-2] [org.apache.drill.exec.service.ServiceEngine] - closed dataPool
in 1009 ms
> 26356 WARN [main] [org.apache.drill.exec.server.Drillbit] - Failure on close()
> java.lang.IllegalArgumentException: /tmp/drill/udf/udf/local does not exist
> 	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1637) ~[commons-io-2.4.jar:2.4]
> 	at org.apache.drill.exec.expr.fn.FunctionImplementationRegistry.close(FunctionImplementationRegistry.java:469)
~[classes/:na]
> 	at org.apache.drill.exec.server.DrillbitContext.close(DrillbitContext.java:209) ~[classes/:na]
> 	at org.apache.drill.exec.work.WorkManager.close(WorkManager.java:152) ~[classes/:na]
> 	at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) ~[classes/:na]
> 	at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) ~[classes/:na]
> 	at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:171) ~[classes/:na]
> ...
> {code}
> The following patch makes the problem go away, but I'm not sure if the above is an indication
of deeper problems.
> {code}
> public class FunctionImplementationRegistry implements FunctionLookupContext, AutoCloseable
{
>   ...
>   public void close() {
>     if (deleteTmpDir) {
>       ...
>     } else {
>       try {
>         File dir = new File(localUdfDir.toUri().getPath());
>         if (dir.exists()) {
>           FileUtils.cleanDirectory(dir);
>         }
>       ...
>     }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message