drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhishek Girish (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-3230) Local file system plug-in must be disabled in distributed mode
Date Mon, 01 Jun 2015 17:32:17 GMT
Abhishek Girish created DRILL-3230:
--------------------------------------

             Summary: Local file system plug-in must be disabled in distributed mode
                 Key: DRILL-3230
                 URL: https://issues.apache.org/jira/browse/DRILL-3230
             Project: Apache Drill
          Issue Type: Bug
          Components: Client - HTTP
            Reporter: Abhishek Girish
            Assignee: Jacques Nadeau


The local file system plug-in (The "file:///" connection string in dfs storage plug-in) does
not behave as expected for both CTAS and querying files, when Drill is configured with distributed
mode (multiple drill-bits across nodes). 

In case of CTAS, parquet files will be written to a specific node's local file system, depending
on which Drill-bit the client connects to. And if the table is moderate to large in size,
Drill may process them in a distributed manner and write data into more than one node - data
is partitioned into different nodes. 

In case of queries, it could be confusing again, as the behavior will depend on which drill-bit
the client connects to. Hence the behavior seen would be inconsistent - queries would return
only partial results, which depend on the drillbit connected to.

My suggestion would be that the local file system plugin be disabled with distributed mode.
With multiple drill bits and a centralized plugin for local file system, consistent behavior
cannot be expected. 

It should be either disabled when distributed mode is detected or we could add support for
multiple namespaces (using IP of nodes) with local file systems (might still not fix all issues).
Or may be there could be other ways to resolve this, which I might be overlooking or not aware
of. 

There have been many issues seen on the user ML, where inconsistent behaviors have been observed
by users.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message