Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C0472CE80 for ; Mon, 17 Mar 2014 17:09:55 +0000 (UTC) Received: (qmail 88508 invoked by uid 500); 17 Mar 2014 17:09:47 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 88231 invoked by uid 500); 17 Mar 2014 17:09:47 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 88098 invoked by uid 99); 17 Mar 2014 17:09:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Mar 2014 17:09:46 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of tucu@cloudera.com designates 209.85.220.42 as permitted sender) Received: from [209.85.220.42] (HELO mail-pa0-f42.google.com) (209.85.220.42) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Mar 2014 17:09:41 +0000 Received: by mail-pa0-f42.google.com with SMTP id fb1so5994355pad.15 for ; Mon, 17 Mar 2014 10:09:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:references:from:content-type:in-reply-to :message-id:date:to:content-transfer-encoding:mime-version; bh=PmnMiA2dUwxvVGghg6DyiwPxX4xzTDCLqwqDJFEeExg=; b=gWQmji9iaASShdSOl6nY0TFgQyQ7AW7yXBC7sbvwY7TvgEdlHQ3l+5f1qbFZ5aozTw uf7DDHZdse6OpsFWoP6lA9ACEn5yD6Wsw4Z98SC+xa1OC/hyMZH4LnXcJOPpPi+OALiB /GhmNTVBWlKzbe+hdGly3TMJ8ixCPgLjjMjNXcn/UHcQ48lxNADTcI77LxPSjY/RPHbc IUNXccEJp4xX8jYuoYijFD4hWiw/21EOd4Nteq1xCRdQQ+RJMBZI3rHy1nCmUkSdbDCF XVRm8V89K89+nGasX+m7oE8WoTzGcZO/UwXR182w2zL4t71a6TxXazfIBhmRGrmxZKQ8 2Qnw== X-Gm-Message-State: ALoCoQkpxZ7cYFbDnXfnVg2s0dU9GjMenqcrpQfIlkB8hUcy681qDY4sZ26jqJdmQfljNPUMc5rj X-Received: by 10.66.243.131 with SMTP id wy3mr26879864pac.32.1395076161528; Mon, 17 Mar 2014 10:09:21 -0700 (PDT) Received: from [192.168.2.102] (70-36-143-86.dsl.dynamic.sonic.net. [70.36.143.86]) by mx.google.com with ESMTPSA id te2sm74474382pac.25.2014.03.17.10.09.20 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 17 Mar 2014 10:09:20 -0700 (PDT) Subject: Re: Data Locality and WebHDFS References: <56681234-BFCC-4293-A9D9-05547F53F9B3@cloudera.com> From: Alejandro Abdelnur Content-Type: multipart/alternative; boundary=Apple-Mail-8A5AF459-1016-4882-BECD-B39FD632B08E X-Mailer: iPhone Mail (11D167) In-Reply-To: <56681234-BFCC-4293-A9D9-05547F53F9B3@cloudera.com> Message-Id: Date: Mon, 17 Mar 2014 10:09:19 -0700 To: "user@hadoop.apache.org" Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (1.0) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-8A5AF459-1016-4882-BECD-B39FD632B08E Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable actually, i am wrong, the webhdfs rest call has an offset.=20 Alejandro (phone typing) > On Mar 17, 2014, at 10:07, Alejandro Abdelnur wrote: >=20 > dont recall how skips are handled in webhdfs, but i would assume that you'= ll get to the first block As usual, and the skip is handled by the DN servin= g the file (as webhdfs doesnot know at open that you'll skip) >=20 > Alejandro > (phone typing) >=20 >> On Mar 17, 2014, at 9:47, RJ Nowling wrote: >>=20 >> Hi Alejandro, >>=20 >> The WebHDFS API allows specifying an offset and length for the request. I= f I specify an offset that start in the second block for a file (thus skippi= ng the first block all together), will the namenode still direct me to a dat= anode with the first block or will it direct me to a namenode with the secon= d block? I.e., am I assured data locality only on the first block of the fi= le (as you're saying) or on the first block I am accessing? >>=20 >> If it is as you say, then I may want to reach out the WebHDFS developers a= nd see if they would be interested in the additional functionality. >>=20 >> Thank you, >> RJ >>=20 >>=20 >>> On Mon, Mar 17, 2014 at 2:40 AM, Alejandro Abdelnur w= rote: >>> I may have expressed myself wrong. You don't need to do any test to see h= ow locality works with files of multiple blocks. If you are accessing a file= of more than one block over webhdfs, you only have assured locality for the= first block of the file. >>>=20 >>> Thanks. >>>=20 >>>=20 >>>> On Sun, Mar 16, 2014 at 9:18 PM, RJ Nowling wrote:= >>>> Thank you, Mingjiang and Alejandro. >>>>=20 >>>> This is interesting. Since we will use the data locality information f= or scheduling, we could "hack" this to get the data locality information, at= least for the first block. As Alejandro says, we'd have to test what happe= ns for other data blocks -- e.g., what if, knowing the block sizes, we reque= st the second or third block? >>>>=20 >>>> Interesting food for thought! I see some experiments in my future! =20= >>>>=20 >>>> Thanks! >>>>=20 >>>>=20 >>>>> On Sun, Mar 16, 2014 at 10:14 PM, Alejandro Abdelnur wrote: >>>>> well, this is for the first block of the file, the rest of the file (b= locks being local or not) are streamed out by the same datanode. for small f= iles (one block) you'll get locality, for large files only the first block, a= nd by chance if other blocks are local to that datanode.=20 >>>>>=20 >>>>>=20 >>>>> Alejandro >>>>> (phone typing) >>>>>=20 >>>>>> On Mar 16, 2014, at 18:53, Mingjiang Shi wrote: >>>>>>=20 >>>>>> According to this page: http://hortonworks.com/blog/webhdfs-%E2%80%93= -http-rest-access-to-hdfs/ >>>>>>> Data Locality: The file read and file write calls are redirected to t= he corresponding datanodes. It uses the full bandwidth of the Hadoop cluster= for streaming data. >>>>>>>=20 >>>>>>> A HDFS Built-in Component: WebHDFS is a first class built-in compone= nt of HDFS. It runs inside Namenodes and Datanodes, therefore, it can use al= l HDFS functionalities. It is a part of HDFS =E2=80=93 there are no addition= al servers to install >>>>>>>=20 >>>>>>=20 >>>>>> So it looks like the data locality is built-into webhdfs, client will= be redirected to the data node automatically.=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>> On Mon, Mar 17, 2014 at 6:07 AM, RJ Nowling wro= te: >>>>>>> Hi all, >>>>>>>=20 >>>>>>> I'm writing up a Google Summer of Code proposal to add HDFS support t= o Disco, an Erlang MapReduce framework. =20 >>>>>>>=20 >>>>>>> We're interested in using WebHDFS. I have two questions: >>>>>>>=20 >>>>>>> 1) Does WebHDFS allow querying data locality information? >>>>>>>=20 >>>>>>> 2) If the data locality information is known, can data on specific d= ata nodes be accessed via Web HDFS? Or do all Web HDFS requests have to go t= hrough a single server? >>>>>>>=20 >>>>>>> Thanks, >>>>>>> RJ >>>>>>>=20 >>>>>>> --=20 >>>>>>> em rnowling@gmail.com >>>>>>> c 954.496.2314 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> --=20 >>>>>> Cheers >>>>>> -MJ >>>>=20 >>>>=20 >>>>=20 >>>> --=20 >>>> em rnowling@gmail.com >>>> c 954.496.2314 >>>=20 >>>=20 >>>=20 >>> --=20 >>> Alejandro >>=20 >>=20 >>=20 >> --=20 >> em rnowling@gmail.com >> c 954.496.2314 --Apple-Mail-8A5AF459-1016-4882-BECD-B39FD632B08E Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
actually, i am wrong, the webhdfs rest= call has an offset. 

Alejandro
(phone typing)

On Mar 17, 2014, at 10:07, Alejandro Abdelnur <tucu@cloudera.com> wrote:

dont rec= all how skips are handled in webhdfs, but i would assume that you'll get to t= he first block As usual, and the skip is handled by the DN serving the file (= as webhdfs doesnot know at open that you'll skip)

Alejandro
(phone typing)

On Mar 17, 2014, at 9:47, RJ Nowling <rnowling@gmail.com> wrote:

<= /div>
Hi Alejandro,

The WebHDFS API allows s= pecifying an offset and length for the request.  If I specify an offset= that start in the second block for a file (thus skipping the first block al= l together), will the namenode still direct me to a datanode with the first b= lock or will it direct me to a namenode with the second block?  I.e., a= m I assured data locality only on the first block of the file (as you're say= ing) or on the first block I am accessing?

If it is as you say, then I may want to reach out the We= bHDFS developers and see if they would be interested in the additional funct= ionality.

Thank you,
RJ


On Mon, Mar 17= , 2014 at 2:40 AM, Alejandro Abdelnur <tucu@cloudera.com> wrot= e:
I may have expressed myself w= rong. You don't need to do any test to see how locality works with files of m= ultiple blocks. If you are accessing a file of more than one block over webh= dfs, you only have assured locality for the first block of the file.

Thanks.


On Sun, Mar 16, 2014 at 9:18 PM, R= J Nowling <rnowling@gmail.com> wrote:
Thank you, Mingjiang and Alej= andro.

This is interesting.  Since we will use the d= ata locality information for scheduling, we could "hack" this to get the dat= a locality information, at least for the first block.  As Alejandro say= s, we'd have to test what happens for other data blocks -- e.g., what if, kn= owing the block sizes, we request the second or third block?

Interesting food for thought!  I see some experimen= ts in my future!  

Thanks!


On Sun, Mar 16, 2014 at 10:14 PM, Alejandro Abdelnur <<= a href=3D"mailto:tucu@cloudera.com" target=3D"_blank">tucu@cloudera.com&= gt; wrote:
well, this is for the f= irst block of the file, the rest of the file (blocks being local or not) are= streamed out by the same datanode. for small files (one block) you'll get l= ocality, for large files only the first block, and by chance if other blocks= are local to that datanode. 


Alejandro
(phone typing)

On Mar 16, 2014, at 18:53, Mingjiang Shi <mshi@gopivotal.com> wrote:

According to this page: http://hortonworks.com/blog/webhdfs-%E2%80%93-http-rest-access-to-h= dfs/

Data Locality: The file read and file write calls=20 are redirected to the corresponding datanodes. It uses the full=20 bandwidth of the Hadoop cluster for streaming data.

A HDFS Built-in Component: WebHDFS is a first class=20 built-in component of HDFS. It runs inside Namenodes and Datanodes,=20 therefore, it can use all HDFS functionalities. It is a part of HDFS =E2=80=93= =20 there are no additional servers to install


So it looks like the data lo= cality is built-into webhdfs, client will be redirected to the data node aut= omatically.




On Mon, Mar 17, 2014 at 6:07 AM, RJ Nowling <rnowling@gmail.com&g= t; wrote:
Hi all,

I'= m writing up a Google Summer of Code proposal to add HDFS support to Disco, a= n Erlang MapReduce framework.  

We're interested in using WebHDFS.  I have two ques= tions:

1) Does WebHDFS allow querying data locality information= ?

2) If the data locality information is known, can= data on specific data nodes be accessed via Web HDFS?  Or do all Web H= DFS requests have to go through a single server?

Thanks,



--
Ch= eers
-MJ



--
em rnowling@gmail.com
c 954.496.2314



--
Alejandro



--
em rnowling@gmail.com
c 954.496.2314 = --Apple-Mail-8A5AF459-1016-4882-BECD-B39FD632B08E--