hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mahmood Naderan <nt_mahm...@yahoo.com>
Subject Re: HDFS interfaces
Date Tue, 04 Jun 2013 06:00:37 GMT
There are many instances of getFileBlockLocations in hadoop/fs. Can you explain which one is
the main?

>It must be combined with a method of logically splitting the input data along block boundaries,
and of launching tasks on worker nodes that >are close to the data splits 
Is this a user level task of system level task? 



Regards,
Mahmood



________________________________
 From: John Lilley <john.lilley@redpoint.net>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>; Mahmood Naderan <nt_mahmood@yahoo.com>

Sent: Tuesday, June 4, 2013 3:28 AM
Subject: RE: HDFS interfaces
 


 
Mahmood,
 
It is the in the FileSystem interface.
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.Path,
long, long)
 
This by itself is not sufficient for application programmers to make good use of data locality. 
It must be combined with a method of logically splitting the input data along block boundaries,
and of launching tasks on worker nodes that are close to the data splits.  MapReduce does
both of these things internally along with the file-format input classes.  For an application
to do so directly, see the new YARN-based interfaces ApplicationMaster and ResourceManager. 
These are however very new and there is little documentation or examples.
 
john
 
From:Mahmood Naderan [mailto:nt_mahmood@yahoo.com] 
Sent: Monday, June 03, 2013 12:09 PM
To: user@hadoop.apache.org
Subject: HDFS interfaces
 
Hello,
It is stated in the "HDFS architecture guide" (https://hadoop.apache.org/docs/r1.0.4/hdfs_design.html)
that 
 
HDFS provides interfaces for applications to move themselves closer to where the data is located.

 
What are these interfaces and where they are in the source code? Is there any manual for the
interfaces?
 
Regards,
Mahmood
Mime
View raw message