hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Renault <orena...@hortonworks.com>
Subject Re: Uploading file to HDFS
Date Mon, 22 Apr 2013 08:37:40 GMT
You should be able to do it using WebHDFS.

WebHDFS is a REST API so there is no need to have hadoop installed on the
client where the file is located. YOu can find an example on how to copy a
file at the following URL.
http://hadoop.apache.org/docs/r1.0.4/webhdfs.html#CREATE

Olivier


On 19 April 2013 11:01, Wellington Chevreuil <wellington.chevreuil@gmail.com
> wrote:

> Can't you use flume for that?
>
>
> 2013/4/19 David Parks <davidparks21@yahoo.com>
>
>> I just realized another trick you might trying. The Hadoop dfs client can
>> read input from STDIN, you could use netcat to pipe the stuff across to
>> HDFS without hitting the hard drive, I haven’t tried it, but here’s what I
>> would think might work:****
>>
>> ** **
>>
>> On the Hadoop box, open a listening port and feed that to the HDFS
>> command:****
>>
>> nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt****
>>
>> ** **
>>
>> On the remote server:****
>>
>> cat my_big_2tb_file > nc 10.1.1.1 2342****
>>
>> ** **
>>
>> I haven’t tried it yet, but in theory this would work. I just happened to
>> test out the hdfs dfs command reading from stdin. You might have to correct
>> the above syntax, I just wrote it off the top of my head.****
>>
>> ** **
>>
>> Dave****
>>
>> ** **
>>
>> ** **
>>
>> *From:* 超级塞亚人 [mailto:sheldom@gmail.com]
>> *Sent:* Friday, April 19, 2013 11:35 AM
>> *To:* user@hadoop.apache.org
>> *Subject:* Uploading file to HDFS****
>>
>> ** **
>>
>> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
>> upload 2TB file to HDFS.How can I put the file to the namenode and upload
>> to HDFS? ****
>>
>
>


-- 
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>

Mime
View raw message