spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krishna Sankar <ksanka...@gmail.com>
Subject Re: Can not build master
Date Sat, 04 Jul 2015 00:35:54 GMT
Patrick,
   I assume an RC3 will be out for folks like me to test the distribution.
As usual, I will run the tests when you have a new distribution.
Cheers
<k/>

On Fri, Jul 3, 2015 at 4:38 PM, Patrick Wendell <pwendell@gmail.com> wrote:

> Patch that added test-jar dependencies:
> https://github.com/apache/spark/commit/bfe74b34
>
> Patch that originally disabled dependency reduced poms:
>
> https://github.com/apache/spark/commit/984ad60147c933f2d5a2040c87ae687c14eb1724
>
> Patch that reverted the disabling of dependency reduced poms:
>
> https://github.com/apache/spark/commit/bc51bcaea734fe64a90d007559e76f5ceebfea9e
>
> On Fri, Jul 3, 2015 at 4:36 PM, Patrick Wendell <pwendell@gmail.com>
> wrote:
> > Okay I did some forensics with Sean Owen. Some things about this bug:
> >
> > 1. The underlying cause is that we added some code to make the tests
> > of sub modules depend on the core tests. For unknown reasons this
> > causes Spark to hit MSHADE-148 for *some* combinations of build
> > profiles.
> >
> > 2. MSHADE-148 can be worked around by disabling building of
> > "dependency reduced poms" because then the buggy code path is
> > circumvented. Andrew Or did this in a patch on the 1.4 branch.
> > However, that is not a tenable option for us because our *published*
> > pom files require dependency reduction to substitute in the scala
> > version correctly for the poms published to maven central.
> >
> > 3. As a result, Andrew Or reverted his patch recently, causing some
> > package builds to start failing again (but publishing works now).
> >
> > 4. The reason this is not detected in our test harness or release
> > build is that it is sensitive to the profiles enabled. The combination
> > of profiles we enable in the test harness and release builds do not
> > trigger this bug.
> >
> > The best path I see forward right now is to do the following:
> >
> > 1. Disable creation of dependency reduced poms by default (this
> > doesn't matter for people doing a package build) so typical users
> > won't have this bug.
> >
> > 2. Add a profile that re-enables that setting.
> >
> > 3. Use the above profile when publishing release artifacts to maven
> central.
> >
> > 4. Hope that we don't hit this bug for publishing.
> >
> > - Patrick
> >
> > On Fri, Jul 3, 2015 at 3:51 PM, Tarek Auel <tarek.auel@gmail.com> wrote:
> >> Doesn't change anything for me.
> >>
> >> On Fri, Jul 3, 2015 at 3:45 PM Patrick Wendell <pwendell@gmail.com>
> wrote:
> >>>
> >>> Can you try using the built in maven "build/mvn..."? All of our builds
> >>> are passing on Jenkins so I wonder if it's a maven version issue:
> >>>
> >>> https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Compile/
> >>>
> >>> - Patrick
> >>>
> >>> On Fri, Jul 3, 2015 at 3:14 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>> > Please take a look at SPARK-8781
> >>> > (https://github.com/apache/spark/pull/7193)
> >>> >
> >>> > Cheers
> >>> >
> >>> > On Fri, Jul 3, 2015 at 3:05 PM, Tarek Auel <tarek.auel@gmail.com>
> wrote:
> >>> >>
> >>> >> I found a solution, there might be a better one.
> >>> >>
> >>> >> https://github.com/apache/spark/pull/7217
> >>> >>
> >>> >> On Fri, Jul 3, 2015 at 2:28 PM Robin East <robin.east@xense.co.uk>
> >>> >> wrote:
> >>> >>>
> >>> >>> Yes me too
> >>> >>>
> >>> >>> On 3 Jul 2015, at 22:21, Ted Yu <yuzhihong@gmail.com>
wrote:
> >>> >>>
> >>> >>> This is what I got (the last line was repeated non-stop):
> >>> >>>
> >>> >>> [INFO] Replacing original artifact with shaded artifact.
> >>> >>> [INFO] Replacing
> >>> >>> /home/hbase/spark/bagel/target/spark-bagel_2.10-1.5.0-SNAPSHOT.jar
> >>> >>> with
> >>> >>>
> >>> >>>
> /home/hbase/spark/bagel/target/spark-bagel_2.10-1.5.0-SNAPSHOT-shaded.jar
> >>> >>> [INFO] Dependency-reduced POM written at:
> >>> >>> /home/hbase/spark/bagel/dependency-reduced-pom.xml
> >>> >>> [INFO] Dependency-reduced POM written at:
> >>> >>> /home/hbase/spark/bagel/dependency-reduced-pom.xml
> >>> >>>
> >>> >>> On Fri, Jul 3, 2015 at 1:13 PM, Tarek Auel <tarek.auel@gmail.com>
> >>> >>> wrote:
> >>> >>>>
> >>> >>>> Hi all,
> >>> >>>>
> >>> >>>> I am trying to build the master, but it stucks and prints
> >>> >>>>
> >>> >>>> [INFO] Dependency-reduced POM written at:
> >>> >>>> /Users/tarek/test/spark/bagel/dependency-reduced-pom.xml
> >>> >>>>
> >>> >>>> build command:  mvn -DskipTests clean package
> >>> >>>>
> >>> >>>> Do others have the same issue?
> >>> >>>>
> >>> >>>> Regards,
> >>> >>>> Tarek
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
View raw message