drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neeraja (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-1290) Creating storage plugin for "hdfs:///" failed with "Unable to create/ update plugin: myhdfs"
Date Tue, 12 Aug 2014 20:12:12 GMT

    [ https://issues.apache.org/jira/browse/DRILL-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094600#comment-14094600
] 

Neeraja commented on DRILL-1290:
--------------------------------

Can you refer to the configuration mentioned in https://issues.apache.org/jira/browse/DRILL-1075
to see if it helps.
Here is a quick snippet from the bug.
----------
Here’s a sample configuration for HDFS:

{
   "type" : "file",
   "enabled" : true,
   "connection" : "hdfs://10.10.30.156:8020/",
   “workspaces" : {
       “root : {
          "location" : "/user/root/drill",
          "writable" : true,
          "storageformat" : "null"
       }
   },
  “formats" : {
      “json” : {
         "type" : “json"
      }
   }
}

Make sure these packages are in the class path (changes based on installation):
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-annotations-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-auth-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-common-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.0.0-cdh4.7.0.jar

And point to the correct Zookeeper in drill-override.conf
drill.exec : { cluster-id: "working_cdh_drill" zk.connect: "10.10.30.156:2181” }

There is an open issue to add the packages mentioned above to the class path automatically:
https://issues.apache.org/jira/browse/DRILL-1160

> Creating storage plugin for "hdfs:///" failed with "Unable to create/ update plugin:
myhdfs"
> --------------------------------------------------------------------------------------------
>
>                 Key: DRILL-1290
>                 URL: https://issues.apache.org/jira/browse/DRILL-1290
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Metadata
>    Affects Versions: 0.4.0
>         Environment: OS: Centos 6.4
> HDFS: CDH5.1
> Drill: 0.4.0
>            Reporter: Hao Zhu
>
> In web GUI, I can successfully create a new storage plugin named "myhdfs" using "file:///":
> {code}
> {
>   "type": "file",
>   "enabled": true,
>   "connection": "file:///",
>   "workspaces": {
>     "root": {
>       "location": "/",
>       "writable": false,
>       "storageformat": null
>     },
>     "tmp": {
>       "location": "/tmp",
>       "writable": true,
>       "storageformat": "csv"
>     }
>   },
>   "formats": {
>     "psv": {
>       "type": "text",
>       "extensions": [
>         "tbl"
>       ],
>       "delimiter": "|"
>     },
>     "csv": {
>       "type": "text",
>       "extensions": [
>         "csv"
>       ],
>       "delimiter": ","
>     },
>     "tsv": {
>       "type": "text",
>       "extensions": [
>         "tsv"
>       ],
>       "delimiter": "\t"
>     },
>     "parquet": {
>       "type": "parquet"
>     },
>     "json": {
>       "type": "json"
>     }
>   }
> }
> {code}
> However if I try to change "file:///" to "hdfs:///" to point to HDFS other than local
file system, drill log errors out "[qtp416200645-67] DEBUG o.a.d.e.server.rest.StorageResources
- Unable to create/ update plugin: myhdfs".
> {code}
> {
>   "type": "file",
>   "enabled": true,
>   "connection": "hdfs:///",
>   "workspaces": {
>     "root": {
>       "location": "/",
>       "writable": false,
>       "storageformat": null
>     },
>     "tmp": {
>       "location": "/tmp",
>       "writable": true,
>       "storageformat": "csv"
>     }
>   },
>   "formats": {
>     "psv": {
>       "type": "text",
>       "extensions": [
>         "tbl"
>       ],
>       "delimiter": "|"
>     },
>     "csv": {
>       "type": "text",
>       "extensions": [
>         "csv"
>       ],
>       "delimiter": ","
>     },
>     "tsv": {
>       "type": "text",
>       "extensions": [
>         "tsv"
>       ],
>       "delimiter": "\t"
>     },
>     "parquet": {
>       "type": "parquet"
>     },
>     "json": {
>       "type": "json"
>     }
>   }
> }
> {code}
> On my cluster, I am using CDH5 hdfs, and it all client configurations are valid. For
example, on the drillbit server:
> {code}
> [root@hdm ~]# hdfs dfs -ls /
> Found 3 items
> drwxr-xr-x   - hbase hbase               0 2014-08-04 22:55 /hbase
> drwxrwxrwt   - hdfs  supergroup          0 2014-07-31 16:31 /tmp
> drwxr-xr-x   - hdfs  supergroup          0 2014-07-11 12:06 /user
> {code}
> Is there anything wrong with the storage plugin syntax for HDFS?
> If so, can drill log prints more debug info to show the reason why it failed?
> Thanks.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message