spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Burak Yavuz <brk...@gmail.com>
Subject Re: Ivy support in Spark vs. sbt
Date Thu, 04 Jun 2015 16:47:15 GMT
Hi Marcelo,

This is interesting. Can you please send me links to any failing builds if
you see that problem please. For now you can set a conf: `spark.jars.ivy`
to use a path except `~/.ivy2` for Spark.

Thanks,
Burak

On Thu, Jun 4, 2015 at 4:29 AM, Sean Owen <sowen@cloudera.com> wrote:

> I've definitely seen the "dependency path must be relative" problem,
> and fixed it by deleting the ivy cache, but I don't know more than
> this.
>
> On Thu, Jun 4, 2015 at 1:33 AM, Marcelo Vanzin <vanzin@cloudera.com>
> wrote:
> > Hey all,
> >
> > I've been bit by something really weird lately and I'm starting to think
> > it's related to the ivy support we have in Spark, and running unit tests
> > that use that code.
> >
> > The first thing that happens is that after running unit tests, sometimes
> my
> > sbt builds start failing with error saying something about "dependency
> path
> > must be relative" (sorry, don't have the exact error around). The
> dependency
> > path it prints is a "file:" URL.
> >
> > I have a feeling that this is because Spark uses Ivy 2.4 while sbt uses
> Ivy
> > 2.3, and those might be incompatible. So if they get mixed up, things can
> > break.
> >
> > The second is that sometimes unit tests fail with some weird error
> > downloading dependencies. When checking the ivy metadata in
> ~/.ivy2/cache,
> > the offending dependencies are pointing to my local maven repo (I have
> > "maven-local" as one of the entries in my ~/.sbt/repositories).
> >
> > My feeling in this case is that Spark's version of Ivy somehow doesn't
> > handle that case.
> >
> > So, long story short:
> >
> > - Has anyone run into either of these problems?
> > - Is it possible to set some env variable or something during tests to
> force
> > them to use their own directory instead of messing up and breaking my
> > ~/.ivy2?
> >
> >
> > --
> > Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
View raw message