Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAOcnVr2-m+n5912XOkT3HMki+QYzq3+hSBh11+yygiDnPw5Psw@mail.gmail.com>
References: 
 <CACb0Fn7kuLROZD6SUPd9e68rLJg9_ydKsRPHkk352hJjy70HtA@mail.gmail.com>
	<CAOcnVr3mc_eHyUjXtpwLwAxCQVwmSnfq-L9XRBEryYzJY7J_kA@mail.gmail.com>
	<CACb0Fn7-jogEE7iHrej91LQ3Tm7t+4yOEEQPt3VfdCHD1v9Z+g@mail.gmail.com>
	<CAPQV63X+RRRcwK=sc547jQ_baqpp4+z=z8PX5=_ViG8K9wk-pQ@mail.gmail.com>
	<CAOcnVr2-m+n5912XOkT3HMki+QYzq3+hSBh11+yygiDnPw5Psw@mail.gmail.com>
Date: Wed, 20 Feb 2013 14:44:55 -0500
Message-ID: 
 <CAPQV63Vq1gkWTPXYn8kpGiXFfXLac0R4xeLRs7-KwWMSEBMCvg@mail.gmail.com>
Subject: Re: copy chunk of hadoop output
From: Jean-Marc Spaggiari <jean-marc@spaggiari.org>
To: user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Harsh,

My bad.

I read the example quickly and I don't know why I tought you used tail
and not head.

head will work perfectly. But tail will not since it will need to read
the entier file. My comment was for tail, not for head, and therefore
not application to the example you gave.


hadoop fs -cat 100-byte-dfs-file | tail -c 5 > 5-byte-local-file

Will have to download the entire file.

Is there a way to "jump" into a certain position in a file and "cat" from t=
here?

JM

2013/2/20, Harsh J <harsh@cloudera.com>:
> Hi JM,
>
> I am not sure how "dangerous" it is, since we're using a pipe here,
> and as you yourself note, it will only last as long as the last bytes
> have been got and then terminate.
>
> The -cat process will terminate because the
> process we're piping to will terminate first after it reaches its goal
> of -c <N bytes>; so certainly the "-cat" program will not fetch the
> whole file down but it may fetch a few bytes extra over communication
> due to use of read buffers (the extra data won't be put into the target
> file, and get discarded).
>
> We can try it out and observe the "clienttrace" logged
> at the DN at the end of the -cat's read. Here's an example:
>
> I wrote a 1.6~ MB file into a file called "foo.jar", see "bytes"
> below, its ~1.58 MB:
>
> 2013-02-20 23:55:19,777 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:58785, dest: /127.0.0.1:50010, bytes: 1658314, op:
> HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_915204057_1, offset: 0,
> srvID: DS-1092147940-192.168.2.1-50010-1349279636946, blockid:
> BP-1461691939-192.168.2.1-1349279623549:blk_2568668834545125596_73870,
> duration: 192289000
>
> I ran the command "hadoop fs -cat foo.jar | head -c 5 > foo.xml" to
> store first 5 bytes onto a local file:
>
> Asserting that post command we get 5 bytes:
> =E2=9E=9C  ~ wc -c foo.xml
>        5 foo.xml
>
> Asserting that DN didn't IO-read the whole file, see the read op below
> and its "bytes" parameter, its only about 193 KB, not the whole block
> of 1.58 MB we wrote earlier:
>
> 2013-02-21 00:01:32,437 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:50010, dest: /127.0.0.1:58802, bytes: 198144, op:
> HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-1698829178_1, offset: 0,
> srvID: DS-1092147940-192.168.2.1-50010-1349279636946, blockid:
> BP-1461691939-192.168.2.1-1349279623549:blk_2568668834545125596_73870,
> duration: 19207000
>
> I don't see how this is anymore dangerous than doing a
> -copyToLocal/-get, which retrieves the whole file anyway?
>
> On Wed, Feb 20, 2013 at 9:25 PM, Jean-Marc Spaggiari
> <jean-marc@spaggiari.org> wrote:
>> But be careful.
>>
>> hadoop fs -cat will retrieve the entire file and last only when it
>> will have retrieve the last bytes you are looking for.
>>
>> If your file is many GB big, it will take a lot of time for this
>> command to complete and will put some pressure on your network.
>>
>> JM
>>
>> 2013/2/19, jamal sasha <jamalshasha@gmail.com>:
>>> Awesome thanks :)
>>>
>>>
>>> On Tue, Feb 19, 2013 at 2:14 PM, Harsh J <harsh@cloudera.com> wrote:
>>>
>>>> You can instead use 'fs -cat' and the 'head' coreutil, as one example:
>>>>
>>>> hadoop fs -cat 100-byte-dfs-file | head -c 5 > 5-byte-local-file
>>>>
>>>> On Wed, Feb 20, 2013 at 3:38 AM, jamal sasha <jamalshasha@gmail.com>
>>>> wrote:
>>>> > Hi,
>>>> >   I was wondering in the following command:
>>>> >
>>>> > bin/hadoop dfs -copyToLocal hdfspath localpath
>>>> > can we have specify to copy not full but like xMB's of file to local
>>>> drive?
>>>> >
>>>> > Is something like this possible
>>>> > Thanks
>>>> > Jamal
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>
>
>
> --
> Harsh J
>