incubator-drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sudheesh Katkam <skat...@maprtech.com>
Subject Re: Cannot add an HDFS storage plugin
Date Thu, 31 Jul 2014 16:29:21 GMT
This information is taken from https://issues.apache.org/jira/browse/DRILL-1075

Here’s a sample configuration for HDFS:

{
   "type" : "file",
   "enabled" : true,
   "connection" : "hdfs://10.10.30.156:8020/",
   “workspaces" : {
       “root : {
          "location" : "/user/root/drill",
          "writable" : true,
          "storageformat" : "null"
       }
   },
  “formats" : {
      “json” : {
         "type" : “json"
      }
   }
}

Make sure these packages are in the class path (changes based on installation):
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-annotations-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-auth-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-common-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.0.0-cdh4.7.0.jar

And point to the correct Zookeeper in drill-override.conf
drill.exec : { cluster-id: "working_cdh_drill" zk.connect: "10.10.30.156:2181” }

There is an open issue to add the packages mentioned above to the class path automatically:
https://issues.apache.org/jira/browse/DRILL-1160 

Thank you,
Sudheesh Katkam

On Jul 31, 2014, at 7:47 AM, Bert M. <bgmlist@hotmail.com> wrote:

> 
> Neeraja Rentachintala <nrentachintala@...> writes:
> 
>> 
>> Are you able to query the local file system successfully.?
>> The doc
>> 
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+to+Data+Sources
> #ConnectingtoD
> ataSources-QueryingaFileSystem
>> has info on the sample hdfs connection under 'querying file system'.
>> Is this not working for you.
>> 
>> On Wed, Jul 30, 2014 at 11:49 AM, Bert M <bgmlist@...> wrote:
>> 
>>> Hello - I get a "Unable to create/ update plugin:" error on any attempts
>>> to add an HDFS storage destination via the Storage-plugin configuration UI.
>>> An example config is:
>>> {  "type": "file",  "enabled": true,  "connection":
>>> "hdfs://ip-xxx-yyy-zzz.ec2.internal:8020",}
>>> 
>>> I've also tried a plain "connection": "hdfs:///" variation too, without
>>> much success.This is with both, the binary distribution for drill-M2 as
>>> well as a compiled version from the top of the tree, deployed on a CDH5.1
>>> stack.
>>> I couldn't find any docs or examples beyond the simple one in the wiki
>>> which shows the second plain hdfs:/// variation.  Could someone help point
>>> me in the right direction?
>>> Thanks,B
>> 
> 
> Yes, I'm able to create or modify storage configs with local filesystems
> and use them successfully. The problem is that I cannot create/update 
> a storage config to use an hdfs location (and yes, that 
> hdfs location does work correctly, both directly and in other 
> hadoop based applications). 
> 
> 
> This behavior is exhibited with the example config from the doc too, viz.:
> {
>  "type": "file",
>  "enabled": true,
>  "connection": "hdfs:///",
>    "workspaces": {
>    "root": {
>      "location": "/",
>      "writable": false,
>      "storageformat": null
>     }
>   }
> }
> 
> 
> This works fine with "connection": "file:///", but the following error is 
> generated with the  "connection": "hdfs:///"
> 
> 2014-07-30 18:39:36,007 [qtp353753254-71] DEBUG 
> o.a.d.e.server.rest.StorageResources - Unable to create/ update plugin: zfs
> 
> 
> Are any other settings required to enable HDFS access in Drill?
> 
> Thanks,
> B
> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message