Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B78A919285 for ; Sat, 30 Apr 2016 22:16:42 +0000 (UTC) Received: (qmail 49703 invoked by uid 500); 30 Apr 2016 22:16:38 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 49612 invoked by uid 500); 30 Apr 2016 22:16:38 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 49601 invoked by uid 99); 30 Apr 2016 22:16:38 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Apr 2016 22:16:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id C6030C0B3A for ; Sat, 30 Apr 2016 22:16:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.798 X-Spam-Level: ** X-Spam-Status: No, score=2.798 tagged_above=-999 required=6.31 tests=[FSL_HELO_BARE_IP_2=1.499, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 7kkhLKldeGFP for ; Sat, 30 Apr 2016 22:16:34 +0000 (UTC) Received: from relayvx11b.securemail.intermedia.net (relayvx11b.securemail.intermedia.net [64.78.52.184]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 2C0565FACD for ; Sat, 30 Apr 2016 22:16:34 +0000 (UTC) Received: from securemail.intermedia.net (localhost [127.0.0.1]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by emg-ca-1-1.localdomain (Postfix) with ESMTPS id 2DF5A53E7B; Sat, 30 Apr 2016 15:16:33 -0700 (PDT) Subject: Re: S3 Hadoop FileSystems MIME-Version: 1.0 x-echoworx-msg-id: d6a81735-8436-4742-8ede-ba8250c2ba39 x-echoworx-emg-received: Sat, 30 Apr 2016 15:16:33.127 -0700 x-echoworx-message-code-hashed: 1f2a0bf781f0767530c9c2e4840fb7a5db49c6cc0a7f4485a15c15da4e39d43b x-echoworx-action: delivered Received: from 10.254.155.14 ([10.254.155.14]) by emg-ca-1-1 (JAMES SMTP Server 2.3.2) with SMTP ID 720; Sat, 30 Apr 2016 15:16:33 -0700 (PDT) Received: from MBX080-W4-CO-2.exch080.serverpod.net (unknown [10.224.117.102]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by emg-ca-1-1.localdomain (Postfix) with ESMTPS id DCA5F53E7B; Sat, 30 Apr 2016 15:16:32 -0700 (PDT) Received: from MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) by MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) with Microsoft SMTP Server (TLS) id 15.0.1130.7; Sat, 30 Apr 2016 15:16:32 -0700 Received: from MBX080-W4-CO-2.exch080.serverpod.net ([10.224.117.102]) by mbx080-w4-co-2.exch080.serverpod.net ([10.224.117.102]) with mapi id 15.00.1130.005; Sat, 30 Apr 2016 15:16:32 -0700 From: Chris Nauroth To: Elliot West , "user@hadoop.apache.org" Thread-Topic: S3 Hadoop FileSystems Thread-Index: AQHRoUXD7BrR7nK7gkatoXQqLdITep+h6A4AgAExSYA= Date: Sat, 30 Apr 2016 22:16:31 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [73.254.66.69] x-source-routing-agent: Processed Content-Type: multipart/alternative; boundary="_000_D34A7C5F41DD7cnaurothhortonworkscom_" --_000_D34A7C5F41DD7cnaurothhortonworkscom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I have some more information regarding MD5 verification with s3a. It turns= out that s3a does have the MD5 verification. It's just not visible from r= eading the s3a code, because the MD5 verification is performed entirely wit= hin the AWS SDK library dependency. If you're interested in more details o= n how this works, or if you want to follow any further discussion on this t= opic, then please take a look at the comments on HADOOP-13076. --Chris Nauroth From: Chris Nauroth > Date: Friday, April 29, 2016 at 9:03 PM To: Elliot West >, "user@hadoop.a= pache.org" > Subject: Re: S3 Hadoop FileSystems Hello Elliot, The current state of support for the various S3 file system implementations= within the Apache Hadoop community can be summed up as follows: s3: Soon to be deprecated, not actively maintained, appears to not work rel= iably at all in recent versions. s3n: Not yet on its way to deprecation, but also not actively maintained. s3a: This is seen as the direction forward for S3 integration, so this is w= here Hadoop contributors are currently focusing their energy. Regarding interoperability with EMR, I can't speak from any of my own exper= ience on how to achieve this. We know that EMR runs custom code different = from what you'll see in the Apache repos. I think that creates a risk for = interop. My only suggestion would be to experiment and make sure to test a= ny of your interop scenarios end-to-end very thoroughly. As you noticed, s3n no longer has a 5 GB limitation. Issue HADOOP-9454 int= roduced support for files larger than 5 GB by using multi-part upload. Thi= s patch was released in Apache Hadoop 2.4.0. Regarding lack of MD5 verification in s3a, I believe that is just an oversi= ght, not an intentional design choice. I filed HADOOP-13076 to track addin= g this feature in s3a. --Chris Nauroth From: Elliot West > Date: Thursday, April 28, 2016 at 5:01 AM To: "user@hadoop.apache.org" > Subject: S3 Hadoop FileSystems Hello, I'm working on a project that moves data from HDFS file systems into S3 for= analysis with Hive on EMR. Recently I've become quite confused with the st= ate of play regarding the different FileSystems: s3, s3n, and s3a. For my u= se case I require the following: * Support for the transfer of very large files. * MD5 checks on copy operations to provide data verification. * Excellent compatibility within an EMR/Hive environment. To move data between clusters it would seem that current versions of the Na= tiveS3FileSystem are my best bet; It appears that only s3n provides MD5 che= cking. It is often cited that s3n does not support files over 5GB = but I can find no indication of such a limitation in the source code, in fa= ct I see that it switches over to multi-part upload for larger files. So, has this limitation been removed in s3n? Within EMR Amazon appear to recommend s3, support s3n, and advise against s= 3a. So yet again s3n would appear to win out here too? = I assume that the s3n implementation available in EMR is different to that = in Apache Hadoop? I find it hard to imagine that AWS would use JetS3t inste= ad of their own AWS Java client, but perhaps they do? Finally, could I use NativeS3FileSystem to perform the actual transfer on m= y Apache Hadoop cluster but then rewrite the table locations in my EMR Hive= metastore to use the s3:// protocol prefix? Could that work? I'd appreciate any light that can be shed on these questions, and any advic= e regarding my reasoning behind my proposal to use s3n for this particular = use case. Thanks, Elliot. --_000_D34A7C5F41DD7cnaurothhortonworkscom_ Content-Type: text/html; charset="us-ascii" Content-ID: <08D8FACE688FC2469F083F5326107D71@exch080.serverpod.net> Content-Transfer-Encoding: quoted-printable
I have some more information regarding MD5 verification with s3a. &nbs= p;It turns out that s3a does have the MD5 verification.  It's just not= visible from reading the s3a code, because the MD5 verification is perform= ed entirely within the AWS SDK library dependency.  If you're interested in more details on how this works, or if you wa= nt to follow any further discussion on this topic, then please take a look = at the comments on HADOOP-13076.

--Chris Nauroth

From: Chris Nauroth <cnauroth@hortonworks.com>
Date: Friday, April 29, 2016 at 9:0= 3 PM
To: Elliot West <teabot@gmail.com>, "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Re: S3 Hadoop FileSystems<= br>

Hello Elliot,

The current state of support for the various S3 file system implementa= tions within the Apache Hadoop community can be summed up as follows:

s3: Soon to be deprecated, not actively maintained, appears to not wor= k reliably at all in recent versions.
s3n: Not yet on its way to deprecation, but also not actively maintain= ed.
s3a: This is seen as the direction forward for S3 integration, so this= is where Hadoop contributors are currently focusing their energy.

Regarding interoperability with EMR, I can't speak from any of my own = experience on how to achieve this.  We know that EMR runs custom code = different from what you'll see in the Apache repos.  I think that crea= tes a risk for interop.  My only suggestion would be to experiment and make sure to test any of your interop scenarios= end-to-end very thoroughly.

As you noticed, s3n no longer has a 5 GB limitation.  Issue HADOO= P-9454 introduced support for files larger than 5 GB by using multi-part up= load.  This patch was released in Apache Hadoop 2.4.0.

Regarding lack of MD5 verification in s3a, I believe that is just an o= versight, not an intentional design choice.  I filed HADOOP-13076 to t= rack adding this feature in s3a.

--Chris Nauroth

From: Elliot West <teabot@gmail.com>
Date: Thursday, April 28, 2016 at 5= :01 AM
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: S3 Hadoop FileSystems

Hello,

I'm working on a project that moves data from HDFS file systems into S= 3 for analysis with Hive on EMR. Recently I've become quite confused with t= he state of play regarding the different FileSystems: s3, s3n, and s3a. For= my use case I require the following:
  • Support for the transfer of very large files.
  • MD5 checks on cop= y operations to provide data verification.
  • Excellent compatibility = within an EMR/Hive environment.
To move data between clusters it would seem that current versions of t= he NativeS3FileSystem are my best bet; It appears that only s3n provides MD5 checking. It is often cited that s3n does not support = files over 5GB but I can find no indication of such a limitation in the sou= rce code, in fact I see that it switches over to multi-part upload for larger files. So, has this limit= ation been removed in s3n?

Within EMR Amazon appear to recommend s3, support s3n, and advise against s3a. So yet again s3n wou= ld appear to win out here too? I assume that the s3n implementation availab= le in EMR is different to that in Apache Hadoop? I find it hard to imagine = that AWS would use JetS3t instead of their own AWS Java client, but perhaps they do?

Finally, could I use NativeS3FileSy= stem to perform the actual transfer on my Apache Hadoop cluster but = then rewrite the table locations in my EMR Hive metastore to use the s3:// protocol prefix? Could that= work?

I'd appreciate any light that can be shed on these questions, and any = advice regarding my reasoning behind my proposal to use s3n for this partic= ular use case.

Thanks,

Elliot.


 --_000_D34A7C5F41DD7cnaurothhortonworkscom_--