drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Katti (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (DRILL-1075) can not create hdfs as connection type in storage engine : server throws http 500 error
Date Mon, 21 Jul 2014 20:32:39 GMT

    [ https://issues.apache.org/jira/browse/DRILL-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069228#comment-14069228
] 

Amit Katti edited comment on DRILL-1075 at 7/21/14 8:31 PM:
------------------------------------------------------------

I just finished testing DRILL on vanilla Hadoop and CDH4 and CDH5 and it is working successfully.

For Drill to work on Hadoop, you need to provide the connection as "connection" : "hdfs://10.10.30.156:8020/"
The storage plugin for dfs is below
{code}
{
  "type" : "file",
  "enabled" : true,
  "connection" : "hdfs://10.10.30.156:8020/",
  "workspaces" : {
    "root" : {
      "location" : "/user/root/drill",
      "writable" : true,
      "storageformat" : "null"
    },
    "tmp" : {
      "location" : "/tmp",
      "writable" : true,
      "storageformat" : "csv"
    },
    "drillTestDir" : {
      "location" : "/drill/testdata/",
      "writable" : false,
      "storageformat" : "parquet"
    }
  },
  "formats" : {
    "psv" : {
      "type" : "text",
      "extensions" : [ "tbl" ],
      "delimiter" : "|"
    },
    "csv" : {
      "type" : "text",
      "extensions" : [ "csv" ],
      "delimiter" : ","
    },
    "tsv" : {
      "type" : "text",
      "extensions" : [ "tsv" ],
      "delimiter" : "t"
    },
    "parquet" : {
      "type" : "parquet"
    },
    "json" : {
      "type" : "json"
    }
  }
}
{code}

For this to work you need to have the below mentioned jars in the Drill Classpath (location
might change depending on installation):
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-annotations-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-auth-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-common-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.0.0-cdh4.7.0.jar

There is a separate Drill Jira open for ensuring this is configured correctly at DRILL-1160
https://issues.apache.org/jira/browse/DRILL-1160


was (Author: amitskatti):
I just finished testing DRILL on vanilla Hadoop and CDH4 and CDH5 and it is working successfully.

For Drill to work on Hadoop, you need to provide the connection as "connection" : "hdfs://10.10.30.156:8020/"
The storage plugin for dfs is below
{code}
{
  "type" : "file",
  "enabled" : true,
  "connection" : "hdfs://10.10.30.156:8020/",
  "workspaces" : {
    "root" : {
      "location" : "/user/root/drill",
      "writable" : true,
      "storageformat" : "null"
    },
    "tmp" : {
      "location" : "/tmp",
      "writable" : true,
      "storageformat" : "csv"
    },
    "drillTestDir" : {
      "location" : "/drill/testdata/",
      "writable" : false,
      "storageformat" : "parquet"
    }
  },
  "formats" : {
    "psv" : {
      "type" : "text",
      "extensions" : [ "tbl" ],
      "delimiter" : "|"
    },
    "csv" : {
      "type" : "text",
      "extensions" : [ "csv" ],
      "delimiter" : ","
    },
    "tsv" : {
      "type" : "text",
      "extensions" : [ "tsv" ],
      "delimiter" : "t"
    },
    "parquet" : {
      "type" : "parquet"
    },
    "json" : {
      "type" : "json"
    }
  }
}
{code}

For this to work you need to have the below mentioned jars in the Drill Classpath:
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-annotations-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-auth-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-common-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.0.0-cdh4.7.0.jar

There is a separate Drill Jira open for ensuring this is configured correctly at DRILL-1160
https://issues.apache.org/jira/browse/DRILL-1160

> can not create hdfs as connection type in storage engine : server throws http 500 error
> ---------------------------------------------------------------------------------------
>
>                 Key: DRILL-1075
>                 URL: https://issues.apache.org/jira/browse/DRILL-1075
>             Project: Apache Drill
>          Issue Type: Bug
>            Reporter: Vivian Summers
>            Assignee: Sudheesh Katkam
>            Priority: Critical
>
> Server at 8047 throws:
> HTTP ERROR 500
> Problem accessing /storage/config/update. Reason:
>     Request failed.
> configure file:
> {
>   "type" : "file",
>   "enabled" : true,
>   "connection" : "hdfs:///",
>   "workspaces" : {
>     "root" : {
>       "location" : "/",
>       "writable" : false,
>       "storageformat" : null
>     },
>     "default" : {
>       "location" : "/user/root",
>       "writable" : true,
>       "storageformat" : null
>     },
>     "tmp" : {
>       "location" : "/tmp",
>       "writable" : true,
>       "storageformat" : "csv"
>     }
>   },
>   "formats" : {
>     "psv" : {
>       "type" : "text",
>       "extensions" : [ "tbl" ],
>       "delimiter" : "|"
>     },
>     "csv" : {
>       "type" : "text",
>       "extensions" : [ "csv" ],
>       "delimiter" : ","
>     },
>     "tsv" : {
>       "type" : "text",
>       "extensions" : [ "tsv" ],
>       "delimiter" : "\t"
>     },
>     "parquet" : {
>       "type" : "parquet"
>     },
>     "json" : {
>       "type" : "json"
>     }
>   }
> }



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message