Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E2E1EE806 for ; Sat, 29 Dec 2012 10:29:55 +0000 (UTC) Received: (qmail 75527 invoked by uid 500); 29 Dec 2012 10:29:50 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 75148 invoked by uid 500); 29 Dec 2012 10:29:50 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 75110 invoked by uid 99); 29 Dec 2012 10:29:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Dec 2012 10:29:48 +0000 X-ASF-Spam-Status: No, hits=3.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,UNPARSEABLE_RELAY X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [98.138.229.69] (HELO nm33-vm5.bullet.mail.ne1.yahoo.com) (98.138.229.69) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Dec 2012 10:29:41 +0000 Received: from [98.138.90.56] by nm33.bullet.mail.ne1.yahoo.com with NNFMP; 29 Dec 2012 10:29:19 -0000 Received: from [98.138.84.36] by tm9.bullet.mail.ne1.yahoo.com with NNFMP; 29 Dec 2012 10:29:19 -0000 Received: from [127.0.0.1] by smtp104.mail.ne1.yahoo.com with NNFMP; 29 Dec 2012 10:29:19 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1356776959; bh=Pl0bvIHRCfdGb9jLJPGJcDfP9bCvlKFmfIFp5i8oppg=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:From:To:References:In-Reply-To:Subject:Date:Message-ID:MIME-Version:Content-Type:X-Mailer:Thread-Index:Content-Language; b=rAzMzzOU0tWLYx08/KaOe3fSqhuBxf9ZNkAoivJOjJwH8UX4IA5b+4Achs0KXsQFB/7Qv7LVWsX8h/QHUgIra/FRiJWFIHdV5PDjh+RfwDdxk0AnSOmTRGCqTl1JcH2QYZOZf6FWWPP99jiopLhoaiZ1o3HMFLI0b9yGz8Ux+ZA= X-Yahoo-Newman-Id: 232559.19850.bm@smtp104.mail.ne1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: .KyWX1kVM1mOeE.mB0qj6EN6NTcPLlc1Libg06FapBsZL7l eaApUzdwdwQCEgu4.A0IyBXT2J0hz3TAhndQQxg6itf9Y8sYFn.FWBVoaFHx oEtPFu3nD8SYW.mZ9opPqItQMdIUXucFnRCfWtUT_XASVU7smu8q9bzEvkQf 0qn7ZXdHvmWyaqflhwRiBW00ee4CWe_tWZaT2gMYSiJB8UUnluYkVnCY4nh. ox1KN.YVuO3VDPQhgsGMJaJzOoejXnrnt3JobAfD4nNVgt3gVPnjRtyRAKWu LuTRX2DGgpgEyZxdL8zi.NlgnPjZoS8hl_1cu._MYrAwYV_8.UkqX_3uQawD U5kTnuaO.s8ZV.4BpZqSwq01D6IEzdJWPwjntZN8tcmrKwlJBV3CCeIY.CrT ThD32cg.k_FXZ_3F1.0BFzhPt6XFn_bdUFp.PVx3bG14Gc4qODA6txz7KqGW 2lyI.1oVKz8mwnL29wY2kwVVtDaW5QVXbLizvDSQzRjewcHe2_VCJDcH9fPY CnNhnPuy0urwDraONb1QWEiVY9LBWX2VJMUGCCnkqTLjNmZ5FYhuClk5A4qM 6iWLXj8ySSfDu8AsAb8O_42wWPdXkwFrs_3wUfHXdLw-- X-Yahoo-SMTP: k2gD1GeswBAV_JFpZm8dmpTCwr4ufTKOyA-- Received: from sattelite (davidparks21@113.161.75.108 with login) by smtp104.mail.ne1.yahoo.com with SMTP; 29 Dec 2012 02:29:19 -0800 PST From: "David Parks" To: References: In-Reply-To: Subject: RE: Fastest way to transfer files Date: Sat, 29 Dec 2012 17:29:08 +0700 Message-ID: <006c01cde5af$574f8880$05ee9980$@yahoo.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_006D_01CDE5EA.03B90EE0" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQIyk+ZRqsZOUta2rFif8X6Sqs/1SAJsf0jtl1LSCoA= Content-Language: en-us X-Virus-Checked: Checked by ClamAV on apache.org This is a multipart message in MIME format. ------=_NextPart_000_006D_01CDE5EA.03B90EE0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Here=E2=80=99s an example of running distcp (actually in this case = s3distcp, but it=E2=80=99s about the same, just new DistCp()) from java: =20 ToolRunner.run(getConf(), new S3DistCp(), new String[] { "--src", "/src/dir/", "--srcPattern", ".*(itemtable)-r-[0-9]*.*", "--dest", "s3://yourbucket/results/",=20 "--s3Endpoint", "s3.amazonaws.com" }); =20 =20 =20 From: Joep Rottinghuis [mailto:jrottinghuis@gmail.com]=20 Sent: Saturday, December 29, 2012 2:51 PM To: user@hadoop.apache.org Cc: user@hadoop.apache.org; hdfs-user@hadoop.apache.org Subject: Re: Fastest way to transfer files =20 Not sure why you are implying a contradiction when you say: "... distcp = is useful _but_ you want to do 'it' in java..." =20 First of all distcp _is_ written in Java. You can call distcp or any other MR job from Java just fine. =20 Cheers, =20 Joep Sent from my iPhone On Dec 28, 2012, at 12:01 PM, burakkk wrote: Hi, I have two different hdfs cluster. I need to transfer files between = these environments. What's the fastest way to transfer files for that = situation?=20 =20 I've researched about it. I found distcp command. It's useful but I want = to do in java so is there any way to do this? =20 Is there any way to transfer files chunk by chunk from one hdfs cluster = to another one or is there any way to implement a process using chunks = without whole file? =20 Thanks Best Regards... =20 --=20 BURAK ISIKLI | http://burakisikli.wordpress.com =20 ------=_NextPart_000_006D_01CDE5EA.03B90EE0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Here=E2=80=99s an example of running distcp (actually in this case = s3distcp, but it=E2=80=99s about the same, just new DistCp()) from = java:

 

ToolRunner.run(getConf(), new<= /b> = S3DistCp(), new<= /b> = String[] {

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 "--src= ", = =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = "/src/= dir/",

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 "--src= Pattern",=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 ".*(it= emtable)-r-[0-9]*.*",

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 "--des= t", = =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = "s3://= yourbucket/results/", =

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 "--s3E= ndpoint",=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 "s3.am= azonaws.com"=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 });

 

 

 

From:= = Joep Rottinghuis [mailto:jrottinghuis@gmail.com]
Sent: = Saturday, December 29, 2012 2:51 PM
To: = user@hadoop.apache.org
Cc: user@hadoop.apache.org; = hdfs-user@hadoop.apache.org
Subject: Re: Fastest way to = transfer files

 

Not = sure why you are implying a contradiction when you say: "... distcp = is useful _but_ you want to do 'it' in = java..."

 

First of all distcp _is_ written in = Java.

You can call distcp = or any other MR job from Java just fine.

 

Cheers,

 

Joep

Sent from my = iPhone


On Dec 28, 2012, at 12:01 PM, burakkk = <burak.isikli@gmail.com> = wrote:

Hi,

I have two = different hdfs cluster. I need to transfer files between these = environments. What's the fastest way to transfer files for that = situation? 

 

I've researched about it. I found distcp command. It's = useful but I want to do in java so is there any way to do = this?

 

Is there any way to transfer files chunk by chunk from = one hdfs cluster to another one or is there any way to implement a = process using chunks without whole file?

 

Thanks

Best Regards...

= ------=_NextPart_000_006D_01CDE5EA.03B90EE0--