hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arjun Bakshi <baksh...@mail.uc.edu>
Subject Building custom block placement policy. What is srcPath?
Date Thu, 24 Jul 2014 17:41:24 GMT
Hi,

I want to write a block placement policy that takes the size of the file 
being placed into account. Something like what is done in CoHadoop or 
BEEMR paper. I have the following questions:

1- What is srcPath in chooseTarget? Is it the path to the original 
un-chunked file, or it is a path to a single block, or something else? I 
added some code to blockplacementpolicydefault to print out the value of 
srcPath but the results look odd.

2- Will a simple new File(srcPath) will do?

3- I've spent time looking at hadoop source code. I can't find a way to 
go from srcPath in chooseTarget to a file size. Every function I think 
can do it, in FSNamesystem, FSDirectory, etc., is either non-public, or 
cannot be called from inside the blockmanagement package or 
blockplacement class.

How do I go from srcPath in blockplacement class to size of the file 
being placed?

Thank you,

AB

Mime
View raw message