hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject hbase-testing-utils [Was: HBase 0.98.x dependency problems]
Date Fri, 27 Jun 2014 19:51:55 GMT
I am looking into this  hbase-testing-util module and don't see any use of it,
at least not within the hbase itself. The module doesn't have any source code
and doesn't provide any functionality for the Hbase itself.

The only relevant ticket I found is HBASE-9699, which only specifies one use
of this module, namely https://github.com/elliottneilclark/hbase-downstreamer,
that seems to be a "Fake downstream project used figuring what is required
when depending on hbase client and minicluster". The explcit use of test
artifacts in non-test scopes are a big no-no in my opinion. I don't think we
need to repeat the same mistake that some of the Hadoop's modules are making.

I'd like someone with better understanding of the subject to shed the light on
the adoption of this module, and possible modifications of it? If the whole
purpose of it is to provide a set of jar files than perhaps a simple assembly
will suffice?

Thanks in advance,
  Cos

On Fri, Jun 27, 2014 at 12:27PM, Konstantin Boudnik wrote:
> Thanks a bunch Ted - it really opened my eye on that's the issue (you can tell
> I am still a noob).
> 
> Cos
> 
> On Fri, Jun 27, 2014 at 05:04AM, Ted Yu wrote:
> > Thanks for finding this issue, Cos.
> > 
> > I opened HBASE-11422 and attached a patch there - there is no dependency on
> > 2.2.0 if -Dhadoop-two.version=2.3.0 is specified.
> > 
> > Cheers
> > 
> > 
> > On Thu, Jun 26, 2014 at 9:43 PM, Konstantin Boudnik <cos@apache.org> wrote:
> > 
> > > Weird... running
> > >   mvn install dependency:tree -Dhadoop-two.version=2.3.0
> > > keeps pulling in 2.2.0 for test artifacts. An easy way to reproduce it is
> > > to
> > > wipe out clean hadoop 2.2.0 artifacts from ~/.m2 as well as 0.98.2* (if
> > > installed) and then run the command above. Something like this can be seen:
> > >
> > > [INFO] +- org.apache.hbase:hbase-shell:jar:0.98.2:compile
> > > [INFO] |  +- org.apache.hbase:hbase-prefix-tree:jar:0.98.2:runtime
> > > [INFO] |  +- com.yammer.metrics:metrics-core:jar:2.1.2:compile
> > > [INFO] |  +- org.jruby:jruby-complete:jar:1.6.8:compile
> > > [INFO] |  +- org.apache.hadoop:hadoop-client:jar:2.3.0:compile
> > > [INFO] |  |  +-
> > > org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.3.0:compile
> > > [INFO] |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.3.0:compile
> > > [INFO] |  |  \-
> > > org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.3.0:compile
> > > [INFO] |  +- org.apache.hadoop:hadoop-hdfs:jar:2.3.0:compile
> > > [INFO] |  |  \- commons-daemon:commons-daemon:jar:1.0.13:compile
> > > [INFO] |  \- org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.2.0:compile
> > >
> > > I have fixed a problem with site target, but still see this...
> > >
> > > Any ideas are very appreciated! Thanks.
> > >   Cos
> > >
> > >
> > > On Tue, Jun 17, 2014 at 08:33PM, Konstantin Boudnik wrote:
> > > > Thank you very much Ted - that helped beautifully! I think the build
> > > system
> > > > becomes a product on its own :)
> > > >
> > > > Regards,
> > > >   Cos
> > > >
> > > > On Tue, Jun 17, 2014 at 12:40PM, Ted Yu wrote:
> > > > > Have you tried the following command ?
> > > > >
> > > > > mvn dependency:tree -Dhadoop-two.version=2.3.0
> > > > >
> > > > > The output of the above has 2.3.0 as the dependency.
> > > > >
> > > > > Cheers
> > > > >
> > > > >
> > > > > On Tue, Jun 17, 2014 at 11:40 AM, Konstantin Boudnik <cos@apache.org>
> > > wrote:
> > > > >
> > > > > > Guys,
> > > > > >
> > > > > > I have noticed an interesting problem with HBase 0.98 line.
I have
> > > ended
> > > > > > up with
> > > > > > a crappy Hadoop 2.2.0 artifacts in my local M2 cache - don't
ask me
> > > how
> > > > > > that
> > > > > > happen ;( - which causes compilation problems in HBase. And
that
> > > brought
> > > > > > this
> > > > > > whole issue into the light. Here's the essence of it:
> > > > > >
> > > > > > while running for hbase-server module
> > > > > >   % mvn dependency:tree  -Dhadoop.version=2.3.0
> > > > > > I am getting a reference to hadoop 2.2.0
> > > > > >
> > > > > > [INFO] +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
> > > > > > ..
> > > > > > [INFO] +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
> > > > > > [INFO] +- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
> > > > > >
> > > > > > This only happens for 0.98. On master the reference goes to
2.4.0
> > > > > > The problem here is that -Dhadoop.version is being seemingly
> > > ignored, which
> > > > > > might lead to binaries with dependencies inconsistent with what
it
> > > was
> > > > > > built
> > > > > > against. I am not sure if this had came up before, but certainly
> > > would
> > > > > > appreciate the community's input, if any.
> > > > > >
> > > > > > Thanks a lot!
> > > > > > --
> > > > > > Take care,
> > > > > >   Cos
> > > > > >
> > > > > >
> > >

Mime
View raw message