Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "David Parks" <davidparks21@yahoo.com>
To: <user@hadoop.apache.org>
References: 
 <CAKdN-T0QS9A=K8NHg=Ttjd_PKn4oAuUy1CEC2a=nwBQ7oyP7zw@mail.gmail.com>
 <CAOcnVr0xJ=cdzqjZvVrw8P2KvO-3WXn7eLZm_k=GToT3QFk=4w@mail.gmail.com>
In-Reply-To: 
 <CAOcnVr0xJ=cdzqjZvVrw8P2KvO-3WXn7eLZm_k=GToT3QFk=4w@mail.gmail.com>
Subject: RE: Uploading file to HDFS
Date: Fri, 19 Apr 2013 14:02:15 +0700
Message-ID: <118701ce3ccb$d5a1c720$80e55560$@yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="GB2312"
Content-Transfer-Encoding: quoted-printable
Thread-Index: AQExkd0iOiq3R7BZItIyRlktUC87pwKlyGOFmgFDFLA=
Content-Language: en-us

I think the problem here is that he doesn't have Hadoop installed on =
this
other location so there's no Hadoop DFS client to do the put directly =
into
HDFS on, he would normally copy the file to one of the nodes in the =
cluster
where the client files are installed. I've had the same problem =
recently.

I've tried setting up dfs-hdfs-proxy, though I must say that it's been
crashing when I try to put modest to large files through it (but I've =
got a
thread going with the developer on that issue). That, or one of the =
other
remote mount options might work well.=20

https://www.google.com/url?sa=3Dt&rct=3Dj&q=3D&esrc=3Ds&source=3Dweb&cd=3D=
1&cad=3Drja&ved=3D
0CDQQFjAA&url=3Dhttp%3A%2F%2Fwiki.apache.org%2Fhadoop%2FMountableHDFS&ei=3D=
T-pwU
Y74A8jPrQfYooHoBw&usg=3DAFQjCNEQbxmrMGKAETj3FPEw3Lr1PBHz-w&sig2=3D4JpEzZ_=
8IAyJ-N
PofSRmMg&bvm=3Dbv.45373924,d.bmk

You could also install Hadoop on the box that has the 2TB file (I =
realize
that you might not control it or want to do that depending on the
configuration).

A remote NFS mount that you can access from one of the Hadoop boxes... ?

Split up the file into smaller pieces?

There are some ideas. I'd love to hear your final solution as I've also =
been
having fits getting into HDFS from outside the Hadoop environment.  I =
wish
it natively supported NFS mounts or some light weight/easy to install =
remote
DFS tools.

Dave


-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]=20
Sent: Friday, April 19, 2013 1:40 PM
To: <user@hadoop.apache.org>
Subject: Re: Uploading file to HDFS

Can you not simply do a fs -put from the location where the 2 TB file
currently resides? HDFS should be able to consume it just fine, as the
client chunks them into fixed size blocks.

On Fri, Apr 19, 2013 at 10:05 AM, =B3=AC=BC=B6=C8=FB=D1=C7=C8=CB =
<sheldom@gmail.com> wrote:
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna=20
> upload 2TB file to HDFS.How can I put the file to the namenode and =
upload
to HDFS?


--
Harsh J