Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of nitinpawar432@gmail.com
 designates 209.85.220.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CANiuQZcR1yLShm-B0tuAJ96vPqJPYCqYXruCZUhNk_Zz5f6D9A@mail.gmail.com>
References: 
 <CA+NDPedbVxB8985gvzi2xL7TB33TsBtTDGfdRO+13gnda0uuRw@mail.gmail.com>
	<CANiuQZcR1yLShm-B0tuAJ96vPqJPYCqYXruCZUhNk_Zz5f6D9A@mail.gmail.com>
Date: Fri, 25 Jan 2013 13:14:18 +0530
Message-ID: 
 <CAORpBsiFaFVMyUS2xS0_DQYkC6XoeujSf9jrDKrXiLQZFde2GQ@mail.gmail.com>
Subject: Re: Copy files from remote folder to HDFS
From: Nitin Pawar <nitinpawar432@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf307c9c7c4dbcde04d4181883

--20cf307c9c7c4dbcde04d4181883
Content-Type: text/plain; charset=ISO-8859-1

if this is one time activity

then just download hadoop binaries from apache
replace the hdfs-site.xml and core-site.xml  with the one you have on
hadoop cluster
allow this machine to connect with hadoop cluster
then you can just do it with hadoop command line scripts.


On Fri, Jan 25, 2013 at 1:01 PM, Mahesh Balija
<balijamahesh.mca@gmail.com>wrote:

> Hi Panshul,
>
>              I am also working on similar requirement, one approach is,
> mount your remote folder on your hadoop master node.
>              And simply write a shell script to copy the files to HDFS
> using crontab.
>
>              I believe Flume is literally a wrong choice as Flume is  a
> data collection and aggregation framework and NOT a file transfer tool and
> may NOT be a good choice when you actually want to copy the files as-is
> onto your cluster (NOT 100% sure as I am also working on that).
>
> Thanks,
> Mahesh Balija,
> CalsoftLabs.
>
> On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <ouchwhisper@gmail.com>wrote:
>
>> Hello,
>>
>> I am trying to copy files, Json files from a remote folder - (a folder on
>> my local system, Cloudfiles folder or a folder on S3 server) to the HDFS of
>> a cluster running at a remote location.
>> The job submitting Application is based on Spring Hadoop.
>>
>> Can someone please suggest or point me in the right direction for best
>> option to achieve the above task:
>> 1. Use Spring Integration data pipelines to poll the folders for files
>> and copy them to the HDFS as they arrive in the source folder. - I have
>> tried to implement the solution in Spring Data book, but it does not run -
>> no idea what is wrong as it does not generate logs.
>>
>> 2. Use some other script method to transfer files.
>>
>> Main requirement, I need to transfer files from a remote folder to HDFS
>> everyday at a fixed time for processing in the hadoop cluster. These files
>> are collecting from various sources in the remote folders.
>>
>> Please suggest an efficient approach. I have been searching and finding a
>> lot of approaches but unable to decide what will work best. As this
>> transfer needs to be as fast as possible.
>> The files to be transferred will be almost 10 GB of Json files not more
>> than 6kb each file.
>>
>> Thanking You,
>>
>>
>> --
>>  Regards,
>> Ouch Whisper
>> 010101010101
>>
>
>


-- 
Nitin Pawar

--20cf307c9c7c4dbcde04d4181883
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">if this is one time activity=A0<div><br></div><div>then ju=
st download hadoop binaries from apache</div><div style>replace the hdfs-si=
te.xml and core-site.xml =A0with the one you have on hadoop cluster=A0</div=
><div style>
allow this machine to connect with hadoop cluster=A0</div><div style>then y=
ou can just do it with hadoop command line scripts.=A0</div></div><div clas=
s=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Fri, Jan 25, 2013 a=
t 1:01 PM, Mahesh Balija <span dir=3D"ltr">&lt;<a href=3D"mailto:balijamahe=
sh.mca@gmail.com" target=3D"_blank">balijamahesh.mca@gmail.com</a>&gt;</spa=
n> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi Panshul,<br><br>=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0 I am also working on similar requirement, one approach is, mou=
nt your remote folder on your hadoop master node.<br>
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 And simply write a shell script to cop=
y the files to HDFS using crontab. <br>
=A0<br>=A0 =A0 =A0 =A0 =A0 =A0=A0 I believe Flume is literally a wrong choi=
ce as Flume is=A0 a data collection and aggregation framework and NOT a fil=
e transfer tool and may NOT be a good choice when you actually want to copy=
 the files as-is onto your cluster (NOT 100% sure as I am also working on t=
hat). <br>

<br>Thanks,<br>Mahesh Balija,<br>CalsoftLabs.<br><br><div class=3D"gmail_qu=
ote"><div class=3D"im">On Fri, Jan 25, 2013 at 6:39 AM, Panshul Whisper <sp=
an dir=3D"ltr">&lt;<a href=3D"mailto:ouchwhisper@gmail.com" target=3D"_blan=
k">ouchwhisper@gmail.com</a>&gt;</span> wrote:<br>

</div><div><div class=3D"h5"><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr=
">Hello,<div><br></div><div>I am trying to copy files, Json files from a re=
mote folder - (a folder on my local system, Cloudfiles folder or a folder o=
n S3 server) to the HDFS of a cluster running at a remote location.=A0</div=
>


<div>The job submitting Application is based on Spring Hadoop.</div><div><b=
r></div><div>Can someone please suggest or point me in the right direction =
for best option to achieve the above task:</div><div>
1. Use Spring Integration data pipelines to poll the folders for files and =
copy them to the HDFS as they arrive in the source folder. - I have tried t=
o implement the solution in Spring Data book, but it does not run - no idea=
 what is wrong as it does not generate logs.</div>


<div><br></div><div>2. Use some other script method to transfer files.</div=
><div><br></div><div>Main requirement, I need to transfer files from a remo=
te folder to HDFS everyday at a fixed time for processing in the hadoop clu=
ster. These files are collecting from various sources in the remote folders=
.</div>


<div><br></div><div>Please suggest an efficient approach. I have been searc=
hing and finding a lot of approaches but unable to decide what will work be=
st. As this transfer needs to be as fast as possible.=A0</div>
<div>The files to be=A0transferred=A0will be almost 10 GB of Json files not=
 more than 6kb each file.</div><div><br></div><div>Thanking You,</div><span=
><font color=3D"#888888"><div><br></div><div><br></div><div>-- <br>
<div dir=3D"ltr">
<div>Regards,</div>Ouch Whisper<div>010101010101</div></div>
</div></font></span></div>
</blockquote></div></div></div><br>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>Nitin Pawar<=
br>
</div>

--20cf307c9c7c4dbcde04d4181883--