hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Need to add fs shim to use QFS
Date Thu, 11 Oct 2012 15:01:15 GMT
On 11 October 2012 00:34, Thilee Subramaniam <thilee@quantcast.com> wrote:

> My initial goal was to make Hadoop use QFS the same way it used KFS. Since
> Hadoop branch-1 had lib/kfs.xx.jar, I was expecting to include a
> qfs.x.x.jar in the Hadoop release; my first patch was to use such jar. But
> now I see that Hadoop trunk links to external maven repos.
> It may be reasonable to link qfs.jar from an external source (I haven't
> yet figured out how to serve the maven repo from github for qfs.jar for
> Hadoop - any help on this will be appreciated).

The issue is not so much where qfs.jar comes from as where the
implementation of Hadoop FileSystem goes -the big question being: should it
live in your (OSS) codebase rather than Hadoop's? That would keep it 100%
in sync with your back end, give you the release schedule to suit you, and
ensure that bugreps end up in your bug reporting tools.

> This way your nightly
> builds will work and the tests can catch any qfs related regressions.

That regression testing is going to have be on QFS -which means your
infrastructure, real or virtual.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message