hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: copy chunk of hadoop output
Date Wed, 20 Feb 2013 15:55:04 GMT
But be careful.

hadoop fs -cat will retrieve the entire file and last only when it
will have retrieve the last bytes you are looking for.

If your file is many GB big, it will take a lot of time for this
command to complete and will put some pressure on your network.

JM

2013/2/19, jamal sasha <jamalshasha@gmail.com>:
> Awesome thanks :)
>
>
> On Tue, Feb 19, 2013 at 2:14 PM, Harsh J <harsh@cloudera.com> wrote:
>
>> You can instead use 'fs -cat' and the 'head' coreutil, as one example:
>>
>> hadoop fs -cat 100-byte-dfs-file | head -c 5 > 5-byte-local-file
>>
>> On Wed, Feb 20, 2013 at 3:38 AM, jamal sasha <jamalshasha@gmail.com>
>> wrote:
>> > Hi,
>> >   I was wondering in the following command:
>> >
>> > bin/hadoop dfs -copyToLocal hdfspath localpath
>> > can we have specify to copy not full but like xMB's of file to local
>> drive?
>> >
>> > Is something like this possible
>> > Thanks
>> > Jamal
>>
>>
>>
>> --
>> Harsh J
>>
>

Mime
View raw message