Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7241A10A22 for ; Mon, 3 Jun 2013 22:59:28 +0000 (UTC) Received: (qmail 69656 invoked by uid 500); 3 Jun 2013 22:59:23 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 69559 invoked by uid 500); 3 Jun 2013 22:59:23 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 69552 invoked by uid 99); 3 Jun 2013 22:59:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jun 2013 22:59:23 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of john.lilley@redpoint.net designates 206.225.164.233 as permitted sender) Received: from [206.225.164.233] (HELO hub021-nj-8.exch021.serverdata.net) (206.225.164.233) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jun 2013 22:59:16 +0000 Received: from MBX021-E3-NJ-2.exch021.domain.local ([10.240.4.78]) by HUB021-NJ-8.exch021.domain.local ([10.240.4.117]) with mapi id 14.03.0123.003; Mon, 3 Jun 2013 15:58:55 -0700 From: John Lilley To: "user@hadoop.apache.org" , Mahmood Naderan Subject: RE: HDFS interfaces Thread-Topic: HDFS interfaces Thread-Index: AQHOYIWH/253W3gB3UqeSNSZunP845kkmKQA Date: Mon, 3 Jun 2013 22:58:55 +0000 Message-ID: <869970D71E26D7498BDAC4E1CA92226B658B9E1C@MBX021-E3-NJ-2.exch021.domain.local> References: <1370282952.23582.YahooMailNeo@web163006.mail.bf1.yahoo.com> In-Reply-To: <1370282952.23582.YahooMailNeo@web163006.mail.bf1.yahoo.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [173.160.43.61] Content-Type: multipart/alternative; boundary="_000_869970D71E26D7498BDAC4E1CA92226B658B9E1CMBX021E3NJ2exch_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_869970D71E26D7498BDAC4E1CA92226B658B9E1CMBX021E3NJ2exch_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Mahmood, It is the in the FileSystem interface. http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.h= tml#getFileBlockLocations(org.apache.hadoop.fs.Path, long, long) This by itself is not sufficient for application programmers to make good u= se of data locality. It must be combined with a method of logically splitt= ing the input data along block boundaries, and of launching tasks on worker= nodes that are close to the data splits. MapReduce does both of these thi= ngs internally along with the file-format input classes. For an applicatio= n to do so directly, see the new YARN-based interfaces ApplicationMaster an= d ResourceManager. These are however very new and there is little document= ation or examples. john From: Mahmood Naderan [mailto:nt_mahmood@yahoo.com] Sent: Monday, June 03, 2013 12:09 PM To: user@hadoop.apache.org Subject: HDFS interfaces Hello, It is stated in the "HDFS architecture guide" (https://hadoop.apache.org/do= cs/r1.0.4/hdfs_design.html) that HDFS provides interfaces for applications to move themselves closer to wher= e the data is located. What are these interfaces and where they are in the source code? Is there a= ny manual for the interfaces? Regards, Mahmood --_000_869970D71E26D7498BDAC4E1CA92226B658B9E1CMBX021E3NJ2exch_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Mahmood,

 <= /p>

It is the in the FileSyst= em interface.

http://hadoop.apache.org/docs/current/api/org/apac= he/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.Pat= h, long, long)

 <= /p>

This by itself is not suf= ficient for application programmers to make good use of data locality. = ; It must be combined with a method of logically splitting the input data along block boundaries, and of launching tasks on worker nodes = that are close to the data splits.  MapReduce does both of these thing= s internally along with the file-format input classes.  For an applica= tion to do so directly, see the new YARN-based interfaces ApplicationMaster and ResourceManager.  These are however = very new and there is little documentation or examples.

 <= /p>

john

 <= /p>

From: Mahmood = Naderan [mailto:nt_mahmood@yahoo.com]
Sent: Monday, June 03, 2013 12:09 PM
To: user@hadoop.apache.org
Subject: HDFS interfaces

 

Hello,

It is stated in the "HDFS architecture gu= ide" (https://hadoop.apache.org/docs/r1.0.4/hdfs_design.html) that

 

HDFS provides interfaces for applications t= o move themselves closer to where the data is located.

 

What are these interfaces and where they are i= n the source code? Is there any manual for the interfaces?

 

Regards,
Mahmood

--_000_869970D71E26D7498BDAC4E1CA92226B658B9E1CMBX021E3NJ2exch_--