Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of sfhadoop@gmail.com designates
 209.85.213.174 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAHfzKEqo76iw48FzQLiSR7r8gy6XdH8CdMZvVuiaarKTqCZK4A@mail.gmail.com>
References: 
 <CAHfzKEqo76iw48FzQLiSR7r8gy6XdH8CdMZvVuiaarKTqCZK4A@mail.gmail.com>
Date: Fri, 19 Jun 2015 23:45:39 -0700
Message-ID: 
 <CA+tdLApqrF_Sgcw7_8y0V+dZnyoD5RUm8u9vSqQ0jadbGDxT-A@mail.gmail.com>
Subject: Re: copy data from one hadoop cluster to another hadoop cluster +
 cant use distcp
From: SF Hadoop <sfhadoop@gmail.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=90e6ba613b3a8c61f40518ed634a

--90e6ba613b3a8c61f40518ed634a
Content-Type: text/plain; charset=UTF-8

Really depends on your requirements for the format of the data.

The easiest way I can think of is to "stream" batches of data into a pub
sub system that the target system can access and then consume.

Verify each batch and then ditch them.

You can throttle the size of the intermediary infrastructure based on your
batches.

Seems the most efficient approach.

On Thursday, June 18, 2015, Divya Gehlot <divya.htconex@gmail.com> wrote:

> Hi,
> I need to copy data from first hadoop cluster to second hadoop cluster.
> I cant access second hadoop cluster from first hadoop cluster due to some
> security issue.
> Can any point me how can I do apart from distcp command.
> For instance
> Cluster 1 secured zone -> copy hdfs data  to -> cluster 2 in non secured
> zone
>
>
>
> Thanks,
> Divya
>
>
>

--90e6ba613b3a8c61f40518ed634a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Really depends on your requirements for the format of the data.<div><br></d=
iv><div>The easiest way I can think of is to &quot;stream&quot; batches of =
data into a pub sub system that the target system can access and then consu=
me.</div><div><br></div><div>Verify each batch and then ditch them.=C2=A0<s=
pan></span></div><div><br></div><div>You can throttle the size of the inter=
mediary infrastructure based on your batches.=C2=A0</div><div><br></div><di=
v>Seems the most efficient approach.=C2=A0<br><br>On Thursday, June 18, 201=
5, Divya Gehlot &lt;<a href=3D"mailto:divya.htconex@gmail.com">divya.htcone=
x@gmail.com</a>&gt; wrote:<br><blockquote class=3D"gmail_quote" style=3D"ma=
rgin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"lt=
r"><div><div>Hi,<br></div>I need to copy data from first hadoop cluster to =
second hadoop cluster.<br></div><div>I cant access second hadoop cluster fr=
om first hadoop cluster due to some security issue.<br></div><div>Can any p=
oint me how can I do apart from distcp command.<br></div><div>For instance =
<br></div><div>Cluster 1 secured zone -&gt; copy hdfs data=C2=A0 to -&gt; c=
luster 2 in non secured zone <br>=C2=A0<br></div><div><br><br></div><div>Th=
anks,<br></div><div>Divya <br></div><div><br><br></div></div>
</blockquote></div>

--90e6ba613b3a8c61f40518ed634a--