hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thilee Subramaniam <thi...@quantcast.com>
Subject Re: Need to add fs shim to use QFS
Date Wed, 10 Oct 2012 23:34:28 GMT

On 10/10/12 12:14 PM, "Steve Loughran" <stevel@hortonworks.com> wrote:

>On 10 October 2012 16:03, Thilee Subramaniam <thilee@quantcast.com> wrote:
>> Hi Steve,
>> Like Harsh said, HADOOP-8886 addresses removing KFS from apache tree.
>> But I interpret your suggestion as 'moving qfs.jar out of apache tree,
>> keeping the jar in possibly a maven repo externally. The new fs shim for
>> QFS will pull this jar from the repo upon compilation etc.'.
>I think the main question is if its external, what actually needs to be
>done w.r.t. Hadoop's build itself. Is your goal to include the shim JAR in
>the normal Hadoop releases? As that's what I'm not sure is needed -no
>non-HDFS filesystem needs that to work with Hadoop. What matters more is
>hook a local Jenkins server to do the nightly builds and tests of hadoop
>branch-1 and trunk onto your filesystem, test out all the releases, and
>then file bug reports if there have been any regressions.

My initial goal was to make Hadoop use QFS the same way it used KFS. Since
Hadoop branch-1 had lib/kfs.xx.jar, I was expecting to include a
qfs.x.x.jar in the Hadoop release; my first patch was to use such jar. But
now I see that Hadoop trunk links to external maven repos.

It may be reasonable to link qfs.jar from an external source (I haven't
yet figured out how to serve the maven repo from github for qfs.jar for
Hadoop - any help on this will be appreciated). This way your nightly
builds will work and the tests can catch any qfs related regressions.

View raw message