drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bridget Bevens <bbev...@maprtech.com>
Subject Re: Drill v1.6 and s3n connection
Date Wed, 20 Apr 2016 18:28:51 GMT
Created DRILL-4621 <https://issues.apache.org/jira/browse/DRILL-4621> to
track doc change request.

Thanks,
Bridget

On Wed, Apr 20, 2016 at 10:10 AM, Jason Altekruse <jason@dremio.com> wrote:

> It looks like a number of doc pages can be improved by referencing some
> changes made recently.
>
> With the inclusion of the needed jars for s3a with Drill, there is no
> longer a need to download jets3t [1]. In addition to setting your
> credentials, this option for allowing more concurrent connections
> (necessary to allow reads of wider parquet files) can also be set in this
> block instead of a core-site.xml file [2].
>
> This config block can actually be used to set any filesystem properties.
> Some of these are custom to a particular filesystem like S3, but a number
> of them are used by a variety of implementations of the HDFS interface. Any
> properties like these [3] should be able to be set in this config block.
>
> [1] -
> https://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/
> [2] -
>
> https://drill.apache.org/docs/s3-storage-plugin/#quering-parquet-format-files-on-s3
> [3] -
>
> https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/core-default.xml
>
> Jason Altekruse
> Software Engineer at Dremio
> Apache Drill Committer
>
> On Wed, Apr 20, 2016 at 9:52 AM, Abhishek Girish <
> abhishek.girish@gmail.com>
> wrote:
>
> > Thanks Jason! I hadn't noticed the config property for S3. I tried this
> out
> > now, and feel it is a lot more easier now.
> >
> > And yes, we should definitely update the docs. There have been quite a
> few
> > threads related to S3 config.
> >
> > On Wed, Apr 20, 2016 at 8:19 AM, Jason Altekruse <jason@dremio.com>
> wrote:
> >
> > > I don't believe there is any way in which a particular bucket has a
> > > property of being s3, s3n or s3a. As I understand it, this only change
> > the
> > > client library that is used to interface with S3. We have included the
> > jars
> > > necessary for s3a with Drill, which is the newest and most performant
> > > option available.
> > >
> > > I need to open a doc JIRA for this, but there is one way in which the
> s3
> > > experience was improved recently to prevent the need to restart Drill
> to
> > > add your S3 credentials. When you create a connection to an S3 bucket,
> > you
> > > can now specify your credentials in a property named "config" in the
> > > storage plugin. This allows you to set any filesystem properties, which
> > we
> > > previously was only possible to set with a core-site.xml file on the
> > > classpath when starting Drill.
> > >
> > > Example:
> > > {
> > >   "type": "file",
> > >   "enabled": true,
> > >   "connection": "s3a://address.of.your.bucket/",
> > >   "config": {
> > >     "fs.s3a.access.key": "<PUT YOUR ACCESS KEY HERE>",
> > >     "fs.s3a.secret.key": "<PUT YOUR SECRET KEY HERE>"
> > >   },
> > >   "workspaces": {
> > >     "root": {
> > >       "location": "/",
> > >       "writable": false,
> > >       "defaultInputFormat": null
> > >     }
> > >   },
> > >   "formats": {
> > >     "psv": {
> > >       "type": "text",
> > >       "extensions": [
> > >         "tbl"
> > >       ],
> > >       "delimiter": "|"
> > >     }, ...
> > >
> > >
> > > Jason Altekruse
> > > Software Engineer at Dremio
> > > Apache Drill Committer
> > >
> > > On Wed, Apr 20, 2016 at 7:40 AM, Nick Monetta <NickM@inrix.com> wrote:
> > >
> > > > Hi,
> > > > Does Drill v1.6 still support s3n connections or just s3a?
> > > >
> > > > I have a s3n S3 bucket that I'm trying to connect to and it will not
> > > work.
> > > > My config is:
> > > >
> > > > {
> > > >   "type": "file",
> > > >   "enabled": true,
> > > >   "connection": "s3n://inrixprod-tapp/",
> > > >   "workspaces": {
> > > >     "root": {
> > > >       "location": "/",
> > > >       "writable": false,
> > > >       "defaultInputFormat": null
> > > >     },
> > > >
> > > > Nick Monetta | INRIX |nickm@inrix.com |Movement Intelligence |
> > > > www.inrix.com  | mobile +1 646-248-4105 |
> > > >
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message