spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: fixed hbase version in SparkBuild (spark-0.8)
Date Wed, 31 Jul 2013 06:13:12 GMT
FWIW it does get landed in lib_managed...

i beleive there's a post in CDH group with the same very problem i had ,
still unresolved/unadvised by Cloudera.


On Tue, Jul 30, 2013 at 10:07 PM, Konstantin Boudnik <cos@apache.org> wrote:

> Ok, that makes sense, although I thought examples might be a good part of
> the
> documentation for the beginners' references, etc. But I have problems
> modifying
> the assembly accordingly. Will do it the first thing in the morning.
>
> Thanks for the input.
>   Cos
>
> On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> > Basically the way to think of the assembly is that it should have
> libraries
> > that users' client programs need to run. These are core, repl (needed if
> > they use the shell), and likely bagel and streaming and mllib, though
> > originally we'd opted to leave those out. We are still deciding on that
> --
> > could go either way.
> >
> > Matei
> >
> > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <matei.zaharia@gmail.com>
> wrote:
> >
> > > Yeah, that is true. But the assembly shouldn't include the examples
> project at all IMO -- if it does now, we should remove it.
> > >
> > > Matei
> > >
> > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <cos@apache.org>
> wrote:
> > >
> > >> Matei,
> > >>
> > >> Hbase dependencies aren't actually included into the Maven assembly
> as of this
> > >> moment, because scope of hbase dependency in examples' module is
> "compile"; but
> > >> the assembly is only includes those with "runtime". Hence it is
> automatically
> > >> excluded.
> > >>
> > >> I believe, hbase is needed for examples during the execution time,
> and if so -
> > >> it would have to be fixed in the module. This will lead to need to
> exclude it
> > >> from the assembly, in turn.
> > >>
> > >> And of course... :)
> > >>
> > >>   s/putt/pull/
> > >>
> > >> Cos
> > >>
> > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> > >>> Yeah, and maybe we will want to change to Maven as the recommended
> tool for
> > >>> assembly building. I want to look into this more for the 0.8 release.
> > >>>
> > >>> Matei
> > >>>
> > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <cos@apache.org>
> wrote:
> > >>>
> > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> > >>>>> Let's at the very least make it configurable, but an even better
> thing will
> > >>>>> be to make sbt assembly not include it. I think the only thing
> that depends
> > >>>>> on HBase is the examples project, but unfortunately SBT puts
all
> its JARs in
> > >>>>> the lib_managed folder and just stupidly creates an assembly
by
> grouping
> > >>>>> those. The Maven build, for example, should not do that.
> > >>>>
> > >>>> It is very easy to exclude dependencies in Maven assembly, like
it
> is done for
> > >>>> Hadoop. Lemme send out a putt request - a good finding indeed,
> Dmitriy, thank
> > >>>> you!
> > >>>>
> > >>>> Cos
> > >>>>
> > >>>>> Matei
> > >>>>>
> > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> wrote:
> > >>>>>
> > >>>>>> Hello,
> > >>>>>>
> > >>>>>> after couple of days(!) of trying to understand where i
get the
> > >>>>>> "NoSuchMethod" error, i traced it down to the fact that
0.8 now
> includes
> > >>>>>> hbase.
> > >>>>>>
> > >>>>>> While it is assumed that hadoop version is specified, hbase
> version is
> > >>>>>> fixed. This seem to create problem if hbase is used with
a
> particular
> > >>>>>> version of CDH hadoop client in the backend. (there's a
known
> compatibility
> > >>>>>> bug).
> > >>>>>>
> > >>>>>> wouldn't it make sense in this case to allow to declare
hbase
> version as
> > >>>>>> well, perhaps even tie it to the CDH version?
> > >>>>>>
> > >>>>>> At the very least i think it deserves a specific mention
in the
> header
> > >>>>>> section to provide opportunity to override, just like hadoop
> version does?
> > >>>>>>
> > >>>>>> Thanks.
> > >>>>>> -D
> > >>>>>
> > >>>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message