drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-5163) External sort on Mac creates a separate child process per spill via HDFS FS
Date Sat, 24 Dec 2016 22:02:58 GMT
Paul Rogers created DRILL-5163:
----------------------------------

             Summary: External sort on Mac creates a separate child process per spill via
HDFS FS
                 Key: DRILL-5163
                 URL: https://issues.apache.org/jira/browse/DRILL-5163
             Project: Apache Drill
          Issue Type: Improvement
    Affects Versions: 1.8.0
            Reporter: Paul Rogers
            Priority: Minor


The external sort operator spills to disk. Spill files are created and written using the HDFS
file system. For performance, HDFS uses native libraries to access the file system. These
native libraries are not available on the Mac. As a result, some operations are implemented
using a shower, Java-only path. One of these operations (need details) is implemented by forking
a child process.

When run in a debugger on the Mac, the behavior shows up as the furious creation and deletion
of threads to manage the child processes: one per spill. Because of this behavior, performance
of external sort is slow. Of course, no production code uses Drill on a Mac, so this is more
of a nuisance than a real bug, which is why it is marked as an improvement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message