Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <BY2PR07MB2325EDEE42E0B54FEDABB72FB56C0@BY2PR07MB2325.namprd07.prod.outlook.com>
References: 
 <BY2PR07MB2325C37D431D36BF8EC1DEC1B56C0@BY2PR07MB2325.namprd07.prod.outlook.com>
 <CAC27z=OLH0J3a2nS+BAz+ZnXUV1Yya0m1xhPpCS2uXH6rY=E0w@mail.gmail.com>
 <CAKiyyaG2sSe-zJiomaQmT+jV3rqfpagJOw5iPpr=ecEJhvof2Q@mail.gmail.com>
 <BY2PR07MB2325EDEE42E0B54FEDABB72FB56C0@BY2PR07MB2325.namprd07.prod.outlook.com>
From: Ufuk Celebi <uce@apache.org>
Date: Wed, 20 Apr 2016 12:20:28 +0200
Message-ID: 
 <CAKiyyaGBMyioh5897vm9LP8O+73083vB7o3cJMfC4gTdXx-17g@mail.gmail.com>
Subject: Re: Flink + S3
To: user@flink.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Wed, Apr 20, 2016 at 1:35 AM, Michael-Keith Bernard
<mkbernard@opentable.com> wrote:
> We're running on self-managed EC2 instances (and we'll eventually have a =
mirror cluster in our colo). The provided documentation notes that for Hado=
op 2.6, we'd need such-and-such version of hadoop-aws and guice on the CP. =
If I wanted to instead use Hadoop 2.7, which versions of those dependencies=
 should I get? And how can I look that up myself? The pom file for hadoop-a=
ws[1] doesn't mention a specific dependency on Guice, so I'm curious how th=
e author of that documentation knew exactly the dependencies and versions r=
equired.

Hey Michael-Keith,

I think you meant Guava and not Guice.

How to determine, which dependencies you need is quite a mess at the
moment. It depends on a combination of 3 things:
1) the dependencies of hadoop-aws [1],
2) which S3 file system you use (in case of the docs
org.apache.hadoop.fs.s3native.NativeS3FileSystem) [2],
3) what Flink shades away in its Hadoop dependencies [3]

1) hadoop-aws depends on hadoop-common (and other packages).
hadoop-common is already part of Flink (including the fs.FileSystem
classes etc.)
2) NativeS3FileSystem uses dependencies from hadoop-common like
FileSystem and from hadoop-aws like Jets3tNativeFileSystemStore. The
hadoop-common stuff is part of Flink and Jets3tNativeFileSystemStore
is part of hadoop-aws. The big issue here is that other S3 FS
implementations might work with the aws-java-sdk packages of
hadoop-aws.
3) Flink shades Hadoop's Guava dependency away and that's why you need
to add it manually to the CP.

So, if you go for the suggested NativeS3FileSystem, you end up needing
hadoop-aws and Guava. Of course, this might change in future versions
of Flink and/or Hadoop. I will update the docs for the different
versions of Flink and Hadoop for now and hope that this will help. :-(

The easiest solution in the future would be that Flink comes with
hadoop-aws, but I don't think that this is going to happen.

=E2=80=93 Ufuk

[1] http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.6.0
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/aws.h=
tml#provide-s3-filesystem-dependency
[3] https://github.com/apache/flink/blob/master/flink-shaded-hadoop/pom.xml