Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 20857 invoked from network); 13 May 2009 21:01:39 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 13 May 2009 21:01:39 -0000 Received: (qmail 44563 invoked by uid 500); 13 May 2009 21:01:36 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 44081 invoked by uid 500); 13 May 2009 21:01:35 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 43946 invoked by uid 99); 13 May 2009 21:01:35 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 May 2009 21:01:35 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [209.177.145.93] (HELO prxy.net) (209.177.145.93) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 May 2009 21:01:24 +0000 X-Scanned-By: RAE MPP/ClamAV http://raeinternet.com/mpp X-Scanned-By: This message was scanned by MPP Free Edition (www.messagepartners.com)! Received: from [174.144.224.31] (account ken@krugler.org HELO [173.116.109.31]) by prxy.net (CommuniGate Pro SMTP 4.2.10) with ESMTP-TLS id 104437785 for core-user@hadoop.apache.org; Wed, 13 May 2009 14:01:32 -0700 Mime-Version: 1.0 Message-Id: In-Reply-To: References: Date: Wed, 13 May 2009 13:58:36 -0700 To: core-user@hadoop.apache.org From: Ken Krugler Subject: Re: HDFS to S3 copy problems Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Virus-Checked: Checked by ClamAV on apache.org Hi Tom, >Ian - Thanks for the detailed analysis. It was these issues that lead >me to create a temporary file in NativeS3FileSystem in the first >place. I think we can get NativeS3FileSystem to report progress >though, see https://issues.apache.org/jira/browse/HADOOP-5814. > >Ken - I can't see why you would be getting that error. Does it work >with hadoop fs, but not hadoop distcp? Neither worked, unfortunately. See my original email re my first attempt using hadoop fs - the result was similar to when I tried hadoop distcp. Seemed to be working, then hung. Eventually I copied the files to my master node's local disk, then our ops guy used s3cmd to push the files to S3. -- Ken >On Sat, May 9, 2009 at 6:48 AM, Nowland, Ian wrote: >> Hi Tom, >> >> Not creating a temp file is the ideal as it saves you from having >>to "waste" using the local hard disk by writing an output file just >>before uploading same to Amazon S3. There are a few problems though: >> >> 1) Amazon S3 PUTs need the file length up front. You could use a >>chunked POST, but then you have the disadvantage of having to >>Base64 encode all your data, increasing bandwidth usage, and also >>you still have the next problems; >> >> 2) You would still want to have MD5 checking. In Amazon S3 both >>PUT and POST require the MD5 to be supplied before the contents. To >>work around this then you would have to upload the object without >>MD5, then check its metadata to make sure the MD5 is correct, then >>delete it if it is not. This is all possible, but would be >>difficult to make bulletproof, whereas in the current version, if >>the MD5 is different the PUT fails atomically and you can easily >>just retry. >> >> 3) Finally, you would have to be careful in reducers that output >>only very rarely. If there is too big a gap between data being >>uploaded through the socket, then S3 may determine the connection >>has timed out, closing the connection and meaning your task has to >>rerun (perhaps just to hit the same problem again). >> >> All of this means that the current solution may be best for now as >>far as general upload. The best I think we can so is fix the fact >>that the task is not progressed in close(). The best way I can see >>to do this is introducing a new interface say called >>ExtendedClosable which defines a close(Progressable p) method. >>Then, have the various clients of FileSystem output streams (e.g. >>Distcp, TextOutputFormat) test if their DataOutputStream supports >>the interface, and if so call this in preference to the default. In >>the case of NativeS3FileSystem then, this method spins up a thread >>to keep the Progressable updated as the upload progresses. >> >> As an additional optimization to Distcp, where the source file >>already exists we could have some extended interface say >>ExtendedWriteFileSystem that has a create() method that takes the >>MD5 and the file size, then test for this interface in the Distcp >>mapper call the extended method. The trade off here is the fact >>that the checksum HDFS stored is not the MD5 needed by S3, and so >>two (perhaps distributed) reads would be needed so the tradeoff is >>these two distributed reads vs a distributed read and a local write >>then local read. >> >> What do you think? >> >> Cheers, >> Ian Nowland >> Amazon.com >> >> -----Original Message----- >> From: Tom White [mailto:tom@cloudera.com] >> Sent: Friday, May 08, 2009 1:36 AM >> To: core-user@hadoop.apache.org >> Subject: Re: HDFS to S3 copy problems >> >> Perhaps we should revisit the implementation of NativeS3FileSystem so >> that it doesn't always buffer the file on the client. We could have an >> option to make it write directly to S3. Thoughts? >> >> Regarding the problem with HADOOP-3733, you can work around it by >> setting fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey in your >> hadoop-site.xml. >> >> Cheers, >> Tom >> >> On Fri, May 8, 2009 at 1:17 AM, Andrew Hitchcock wrote: >>> Hi Ken, >>> >>> S3N doesn't work that well with large files. When uploading a file to > >> S3, S3N saves it to local disk during write() and then uploads to S3 >>> during the close(). Close can take a long time for large files and it >>> doesn't report progress, so the call can time out. >>> >>> As a work around, I'd recommend either increasing the timeout or >>> uploading the files by hand. Since you only have a few large files, >>> you might want to copy the files to local disk and then use something >>> like s3cmd to upload them to S3. >>> >>> Regards, >>> Andrew >>> >>> On Thu, May 7, 2009 at 4:42 PM, Ken Krugler >>> wrote: >>>> Hi all, >>>> >>>> I have a few large files (4 that are 1.8GB+) I'm trying to copy >>>>from HDFS to >>>> S3. My micro EC2 cluster is running Hadoop 0.19.1, and has one master/two >>>> slaves. >>>> > >>> I first tried using the hadoop fs -cp command, as in: >>>> >>>> hadoop fs -cp output// s3n://// >>>> >>>> This seemed to be working, as I could walk the network traffic spike, and >>>> temp files were being created in S3 (as seen with CyberDuck). >>>> >>>> But then it seemed to hang. Nothing happened for 30 minutes, so >>>>I killed the >>>> command. > >>> >>>> Then I tried using the hadoop distcp command, as in: >>>> >>>> hadoop distcp hdfs://:50001/// s3://:>>> key>@// >>>> >>>> This failed, because my secret key has a '/' in it >>>> (http://issues.apache.org/jira/browse/HADOOP-3733) >>>> >>>> Then I tried using hadoop distcp with the s3n URI syntax: >>>> >>>> hadoop distcp hdfs://:50001/// s3n://// >>>> >>>> Similar to my first attempt, it seemed to work. Lots of network activity, >>>> temp files being created, and in the terminal I got: >>>> >>>> 09/05/07 18:36:11 INFO mapred.JobClient: Running job: >>>>job_200905071339_0004 >>>> 09/05/07 18:36:12 INFO mapred.JobClient: map 0% reduce 0% >>>> 09/05/07 18:36:30 INFO mapred.JobClient: map 9% reduce 0% >>>> 09/05/07 18:36:35 INFO mapred.JobClient: map 14% reduce 0% >>>> 09/05/07 18:36:38 INFO mapred.JobClient: map 20% reduce 0% >>>> >>>> But again it hung. No network traffic, and eventually it dumped out: >>>> >>>> 09/05/07 18:52:34 INFO mapred.JobClient: Task Id : >>>> attempt_200905071339_0004_m_000001_0, Status : FAILED >>>> Task attempt_200905071339_0004_m_000001_0 failed to report status for 601 >>>> seconds. Killing! >>>> 09/05/07 18:53:02 INFO mapred.JobClient: Task Id : >>>> attempt_200905071339_0004_m_000004_0, Status : FAILED >>>> Task attempt_200905071339_0004_m_000004_0 failed to report status for 602 >>>> seconds. Killing! >>>> 09/05/07 18:53:06 INFO mapred.JobClient: Task Id : >>>> attempt_200905071339_0004_m_000002_0, Status : FAILED >>>> Task attempt_200905071339_0004_m_000002_0 failed to report status for 602 >>>> seconds. Killing! >>>> 09/05/07 18:53:09 INFO mapred.JobClient: Task Id : >>>> attempt_200905071339_0004_m_000003_0, Status : FAILED >>>> Task attempt_200905071339_0004_m_000003_0 failed to report status for 601 >>>> seconds. Killing! >>>> > >>> In the task GUI, I can see the same tasks failing, and being >restarted. But >>>> the restarted tasks seem to be just hanging w/o doing anything. >>>> >>>> Eventually one of the tasks made a bit more progress, but then it finally >>>> died with: >>>> > >>> Copy failed: java.io.IOException: Job failed! >>>> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) >>>> at org.apache.hadoop.tools.DistCp.copy(DistCp.java:647) >>>> at org.apache.hadoop.tools.DistCp.run(DistCp.java:844) >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>>> at org.apache.hadoop.tools.DistCp.main(DistCp.java:871) >>>> >>>> So - any thoughts on what's going wrong? >>>> >>>> Thanks, >>>> >>>> -- Ken >>>> -- >>>> Ken Krugler >>>> +1 530-210-6378 >>>> >>> >> -- Ken Krugler +1 530-210-6378