hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Dost <jd...@ucsd.edu>
Subject Re: Feature request to provide DFSInputStream subclassing mechanism
Date Fri, 09 Aug 2013 19:52:35 GMT
Hi Steve,

Thanks again for your in depth replies, we found your comments quite 
useful.  A few responses inline:

On 8/9/13 10:31 AM, Steve Loughran wrote:
> On 8 August 2013 21:51, Matevz Tadel <mtadel@ucsd.edu> wrote:
>> We already do fallback to xrootd on open failures from our application
>> framework ... the job gets redirected to xrootd proxy which downloads the
>> whole file and serves data as the job asks for it. The changes described by
>> Jeff are an optimization for cases when a single block becomes unavailable.
> Like other said, there's work on heterogenous storage. Maybe you could make
> sure that there is some handling there for block unavailablity events -then
> you can hook in that to handle it.
We discussed this a bit more and agree, the best way forward is to 
carefully watch the progress of this development, and we will look into 
how it will fit our needs.
>> Thanks also for the comments below, Jeff will appreciate them more as he
>> actually went through all the HDFS code and did all the changes in the
>> patch, I did the JNI + Xrootd part.
>> One more question ... if we'd converge on an acceptable solution (I find
>> it a bit hard at the moment), how long would it take for the changes to be
>> released and what release cycle would it end up in?
> all changes go into trunk, if it was go to into the 2.x line then it would
> be 2.3 at the earliest; 2.1 is going through its beta right now. If you
> work against 2.1 and report bugs now, that would help the beta and make it
> easier for you to have a private fork of 2.1.x with the extensions
In the long term this sounds like a reasonable approach.  For the short 
term, we will continue to work with OSG to include patches needed to 
support our use case, as their upcoming release will be based the 2.0 
branch we have already been experimenting with.  We will take into 
careful consideration the requirements you outlined that would make our 
patches reasonable candidates to be considered in the official Hadoop trunk.

Jeff Dost
UCSD Physics

View raw message