drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Altekruse <ja...@dremio.com>
Subject Re: Drill v1.6 and s3n connection
Date Wed, 20 Apr 2016 15:19:43 GMT
I don't believe there is any way in which a particular bucket has a
property of being s3, s3n or s3a. As I understand it, this only change the
client library that is used to interface with S3. We have included the jars
necessary for s3a with Drill, which is the newest and most performant
option available.

I need to open a doc JIRA for this, but there is one way in which the s3
experience was improved recently to prevent the need to restart Drill to
add your S3 credentials. When you create a connection to an S3 bucket, you
can now specify your credentials in a property named "config" in the
storage plugin. This allows you to set any filesystem properties, which we
previously was only possible to set with a core-site.xml file on the
classpath when starting Drill.

Example:
{
  "type": "file",
  "enabled": true,
  "connection": "s3a://address.of.your.bucket/",
  "config": {
    "fs.s3a.access.key": "<PUT YOUR ACCESS KEY HERE>",
    "fs.s3a.secret.key": "<PUT YOUR SECRET KEY HERE>"
  },
  "workspaces": {
    "root": {
      "location": "/",
      "writable": false,
      "defaultInputFormat": null
    }
  },
  "formats": {
    "psv": {
      "type": "text",
      "extensions": [
        "tbl"
      ],
      "delimiter": "|"
    }, ...


Jason Altekruse
Software Engineer at Dremio
Apache Drill Committer

On Wed, Apr 20, 2016 at 7:40 AM, Nick Monetta <NickM@inrix.com> wrote:

> Hi,
> Does Drill v1.6 still support s3n connections or just s3a?
>
> I have a s3n S3 bucket that I'm trying to connect to and it will not work.
> My config is:
>
> {
>   "type": "file",
>   "enabled": true,
>   "connection": "s3n://inrixprod-tapp/",
>   "workspaces": {
>     "root": {
>       "location": "/",
>       "writable": false,
>       "defaultInputFormat": null
>     },
>
> Nick Monetta | INRIX |nickm@inrix.com |Movement Intelligence |
> www.inrix.com  | mobile +1 646-248-4105 |
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message