hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: Need to add fs shim to use QFS
Date Wed, 10 Oct 2012 19:21:51 GMT
On Wed, Oct 10, 2012 at 12:14 PM, Steve Loughran <stevel@hortonworks.com> wrote:
> On 10 October 2012 16:03, Thilee Subramaniam <thilee@quantcast.com> wrote:
>> Hi Steve,
>> Like Harsh said, HADOOP-8886 addresses removing KFS from apache tree.
>> But I interpret your suggestion as 'moving qfs.jar out of apache tree, and
>> keeping the jar in possibly a maven repo externally. The new fs shim for
>> QFS will pull this jar from the repo upon compilation etc.'.
> I think the main question is if its external, what actually needs to be
> done w.r.t. Hadoop's build itself. Is your goal to include the shim JAR in
> the normal Hadoop releases? As that's what I'm not sure is needed -no other
> non-HDFS filesystem needs that to work with Hadoop. What matters more is to
> hook a local Jenkins server to do the nightly builds and tests of hadoop
> branch-1 and trunk onto your filesystem, test out all the releases, and
> then file bug reports if there have been any regressions.

Good point Steve.  This touches on the larger issue of whether it
makes sense to host FS clients for other file systems in Hadoop
itself.  I agree with what I think you're getting which is - if we can
handle the testing and integration via external dependencies it would
probably be better to have the Hadoop client code live and ship as
part of the other projects since it's more likely to be maintained
there.  Perhaps start a DISCUSS thread on common-dev since this
pertains to other file systems aside from QFS?


View raw message