Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4A91B18E28 for ; Tue, 27 Oct 2015 12:15:06 +0000 (UTC) Received: (qmail 44416 invoked by uid 500); 27 Oct 2015 12:15:02 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 44331 invoked by uid 500); 27 Oct 2015 12:15:02 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 44321 invoked by uid 99); 27 Oct 2015 12:15:02 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Oct 2015 12:15:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id A3B8CC0E4D for ; Tue, 27 Oct 2015 12:15:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.902 X-Spam-Level: ** X-Spam-Status: No, score=2.902 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, MIME_QP_LONG_LINE=0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 0f87QizRS3jy for ; Tue, 27 Oct 2015 12:14:47 +0000 (UTC) Received: from mail-pa0-f49.google.com (mail-pa0-f49.google.com [209.85.220.49]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 2113D2314A for ; Tue, 27 Oct 2015 12:14:46 +0000 (UTC) Received: by pasz6 with SMTP id z6so221220760pas.2 for ; Tue, 27 Oct 2015 05:14:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=NUxtt2A1fRNU7TMxSFhuOI9SentVNji/oTOxWQ107Zk=; b=lcqqxrGNia7jnOI2k+27YNyh/64hqPdTczl9X6dWtGL4PNU5MzA+Oa/rBHBuDD0Fi4 CqGxUMgu4IYT7PtAsAgKTK9Pusu9UuuWdy8+jgeZnvM1gDyeXKsZ+yo0tPpNpvJJrjQc Litzvn+MtFrAY8IXlS+ozH1GDCYA/EUFrjwa9d2nNG0MUlO47DJclK00iBGxY7XUibR8 oJpflFKNoTj+vz8GrOTTT+7vH4opgtRHCxt8+/0TkWHikq1QQRf/oLCFTNBQlN7LherG wGYks8Ox8UTNVKVTg1CO+slO94uV/Xgkn0rHjZNIK7hxY7JoetPKQvdKbaj3/Sk14Bev DORQ== X-Received: by 10.68.136.10 with SMTP id pw10mr27976843pbb.89.1445948084672; Tue, 27 Oct 2015 05:14:44 -0700 (PDT) Received: from [192.168.0.11] (c-50-150-102-33.hsd1.ca.comcast.net. [50.150.102.33]) by smtp.gmail.com with ESMTPSA id qb7sm39635870pab.47.2015.10.27.05.14.43 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 27 Oct 2015 05:14:44 -0700 (PDT) Content-Type: multipart/alternative; boundary=Apple-Mail-474D6ECC-F634-452A-A844-7C417E916DF4 Mime-Version: 1.0 (1.0) Subject: Re: Maven build failed (Spark master) From: Ted Yu X-Mailer: iPhone Mail (13A344) In-Reply-To: Date: Tue, 27 Oct 2015 05:14:43 -0700 Cc: Yana Kadiyska , Sean Owen , user Content-Transfer-Encoding: 7bit Message-Id: <86DACDBE-3B41-4F3A-B32F-9581905D9D72@gmail.com> References: To: Kayode Odeyemi --Apple-Mail-474D6ECC-F634-452A-A844-7C417E916DF4 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Can you try the same command shown in the pull request ? Thanks > On Oct 27, 2015, at 12:40 AM, Kayode Odeyemi wrote: >=20 > Thank you. >=20 > But I'm getting same warnings and it's still preventing the archive from b= eing generated. >=20 > I've ran this on both OSX Lion and Ubuntu 12. Same error. No .gz file >=20 >> On Mon, Oct 26, 2015 at 9:10 PM, Ted Yu wrote: >> Looks like '-Pyarn' was missing in your command. >>=20 >>> On Mon, Oct 26, 2015 at 12:06 PM, Kayode Odeyemi wro= te: >>> I used this command which is synonymous to what you have: >>>=20 >>> ./make-distribution.sh --name spark-latest --tgz --mvn mvn -Dhadoop.vers= ion=3D2.6.0 -Phadoop-2.6 -Phive -Phive-thriftserver -DskipTests clean packag= e -U=20 >>>=20 >>> But I still see WARNINGS like this in the output and no .gz file created= : >>>=20 >>> cp: /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org= /display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/par= quet_partitioned/year=3D2015/month=3D10/day=3D26/.part-r-00005.gz.parquet.cr= c: No such file or directory >>> cp: /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org= /display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/par= quet_partitioned/year=3D2015/month=3D10/day=3D26/part-r-00005.gz.parquet: No= such file or directory >>> cp: /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org= /display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/par= quet_partitioned/year=3D2015/month=3D9: No such file or directory >>> cp: /usr/local/spark-latest/dist/python/test_support/sql/parquet_partiti= oned/year=3D2015/month=3D9: unable to copy extended attributes to /usr/local= /spark-latest/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER= /Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/y= ear=3D2015/month=3D9: No such file or directory >>> cp: /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org= /display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/par= quet_partitioned/year=3D2015/month=3D9/day=3D1: No such file or directory >>> cp: /usr/local/spark-latest/dist/python/test_support/sql/parquet_partiti= oned/year=3D2015/month=3D9/day=3D1: unable to copy extended attributes to /u= sr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org/display/M= AVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_parti= tioned/year=3D2015/month=3D9/day=3D1: No such file or directory >>> cp: /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org= /display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/par= quet_partitioned/year=3D2015/month=3D9/day=3D1/.part-r-00007.gz.parquet.crc:= No such file or directory >>>=20 >>>> On Mon, Oct 26, 2015 at 8:58 PM, Ted Yu wrote: >>>> If you use the command shown in: >>>> https://github.com/apache/spark/pull/9281 >>>>=20 >>>> You should have got the following: >>>>=20 >>>> ./dist/python/test_support/sql/parquet_partitioned/year=3D2014/month=3D= 9/day=3D1/part-r-00008.gz.parquet >>>> ./dist/python/test_support/sql/parquet_partitioned/year=3D2015/month=3D= 9/day=3D1/part-r-00007.gz.parquet >>>> ./dist/python/test_support/sql/parquet_partitioned/year=3D2015/month=3D= 10/day=3D25/part-r-00004.gz.parquet >>>> ./dist/python/test_support/sql/parquet_partitioned/year=3D2015/month=3D= 10/day=3D25/part-r-00002.gz.parquet >>>> ./dist/python/test_support/sql/parquet_partitioned/year=3D2015/month=3D= 10/day=3D26/part-r-00005.gz.parquet >>>>=20 >>>>> On Mon, Oct 26, 2015 at 11:47 AM, Kayode Odeyemi w= rote: >>>>> I see a lot of stuffs like this after the a successful maven build: >>>>>=20 >>>>> cp: /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.o= rg/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/p= arquet_partitioned/year=3D2014/month=3D9/day=3D1/ >>>>> part-r-00008.gz.parquet: No such file or directory >>>>>=20 >>>>> Seems it fails when it tries to package the build as an archive. >>>>>=20 >>>>> I'm using the latest code on github master. >>>>>=20 >>>>> Any ideas please? >>>>>=20 >>>>>> On Mon, Oct 26, 2015 at 6:20 PM, Yana Kadiyska wrote: >>>>>> In 1.4 ./make_distribution produces a .tgz file in the root directory= (same directory that make_distribution is in) >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>> On Mon, Oct 26, 2015 at 8:46 AM, Kayode Odeyemi w= rote: >>>>>>> Hi, >>>>>>>=20 >>>>>>> The ./make_distribution task completed. However, I can't seem to loc= ate the >>>>>>> .tar.gz file. >>>>>>>=20 >>>>>>> Where does Spark save this? or should I just work with the dist dire= ctory? >>>>>>>=20 >>>>>>>> On Fri, Oct 23, 2015 at 4:23 PM, Kayode Odeyemi = wrote: >>>>>>>> I saw this when I tested manually (without ./make-distribution) >>>>>>>>=20 >>>>>>>> Detected Maven Version: 3.2.2 is not in the allowed range 3.3.3. >>>>>>>>=20 >>>>>>>> So I simply upgraded maven to 3.3.3. >>>>>>>>=20 >>>>>>>> Resolved. Thanks >>>>>>>>=20 >>>>>>>>> On Fri, Oct 23, 2015 at 3:17 PM, Sean Owen wr= ote: >>>>>>>>> This doesn't show the actual error output from Maven. I have a str= ong >>>>>>>>> guess that you haven't set MAVEN_OPTS to increase the memory Maven= can >>>>>>>>> use. >>>>>>>>>=20 >>>>>>>>> On Fri, Oct 23, 2015 at 6:14 AM, Kayode Odeyemi wrote: >>>>>>>>> > Hi, >>>>>>>>> > >>>>>>>>> > I can't seem to get a successful maven build. Please see command= output >>>>>>>>> > below: >>>>>>>>> > >>>>>>>>> > bash-3.2$ ./make-distribution.sh --name spark-latest --tgz --mvn= mvn >>>>>>>>> > -Dhadoop.version=3D2.7.0 -Phadoop-2.7 -Phive -Phive-thriftserver= -DskipTests >>>>>>>>> > clean package >>>>>>>>> > +++ dirname ./make-distribution.sh >>>>>>>>> > ++ cd . >>>>>>>>> > ++ pwd >>>>>>>>> > + SPARK_HOME=3D/usr/local/spark-latest >>>>>>>>> > + DISTDIR=3D/usr/local/spark-latest/dist >>>>>>>>> > + SPARK_TACHYON=3Dfalse >>>>>>>>> > + TACHYON_VERSION=3D0.7.1 >>>>>>>>> > + TACHYON_TGZ=3Dtachyon-0.7.1-bin.tar.gz >>>>>>>>> > + >>>>>>>>> > TACHYON_URL=3Dhttps://github.com/amplab/tachyon/releases/downloa= d/v0.7.1/tachyon-0.7.1-bin.tar.gz >>>>>>>>> > + MAKE_TGZ=3Dfalse >>>>>>>>> > + NAME=3Dnone >>>>>>>>> > + MVN=3D/usr/local/spark-latest/build/mvn >>>>>>>>> > + (( 12 )) >>>>>>>>> > + case $1 in >>>>>>>>> > + NAME=3Dspark-latest >>>>>>>>> > + shift >>>>>>>>> > + shift >>>>>>>>> > + (( 10 )) >>>>>>>>> > + case $1 in >>>>>>>>> > + MAKE_TGZ=3Dtrue >>>>>>>>> > + shift >>>>>>>>> > + (( 9 )) >>>>>>>>> > + case $1 in >>>>>>>>> > + MVN=3Dmvn >>>>>>>>> > + shift >>>>>>>>> > + shift >>>>>>>>> > + (( 7 )) >>>>>>>>> > + case $1 in >>>>>>>>> > + break >>>>>>>>> > + '[' -z /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Conte= nts/Home ']' >>>>>>>>> > + '[' -z /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Conte= nts/Home ']' >>>>>>>>> > ++ command -v git >>>>>>>>> > + '[' /usr/bin/git ']' >>>>>>>>> > ++ git rev-parse --short HEAD >>>>>>>>> > + GITREV=3D487d409 >>>>>>>>> > + '[' '!' -z 487d409 ']' >>>>>>>>> > + GITREVSTRING=3D' (git revision 487d409)' >>>>>>>>> > + unset GITREV >>>>>>>>> > ++ command -v mvn >>>>>>>>> > + '[' '!' /usr/bin/mvn ']' >>>>>>>>> > ++ mvn help:evaluate -Dexpression=3Dproject.version -Dhadoop.ver= sion=3D2.7.0 >>>>>>>>> > -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests clean packag= e >>>>>>>>> > ++ grep -v INFO >>>>>>>>> > ++ tail -n 1 >>>>>>>>> > + VERSION=3D'[ERROR] [Help 1] >>>>>>>>> > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionEx= ception' >>>>>>>>> > >>>>>>>>> > Same output error with JDK 7 >>>>>>>>> > >>>>>>>>> > Appreciate your help. >>>>>>>>> > >>>>>>>>> > >>>>>>>=20 >>>=20 >=20 >=20 --Apple-Mail-474D6ECC-F634-452A-A844-7C417E916DF4 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Can you try the same comman= d shown in the pull request ?

Thanks

= On Oct 27, 2015, at 12:40 AM, Kayode Odeyemi <dreyemi@gmail.com> wrote:

Thank you.

But I'm getting sa= me warnings and it's still preventing the archive from being generated.

I've ran this on both OSX Lion and Ubuntu 12. Same erro= r. No .gz file

On Mon, Oct 26, 2015 at 9:10 PM, Ted Yu <yuzhihong@gmail.com> wrote:
Looks like '-Pyarn' was missing in your command.

On Mon, Oct 26, 2015 at 12:06 PM, Kayode Odeyemi <dreyemi@gmail.com>= wrote:
I used this= command which is synonymous to what you have:

./make-dis= tribution.sh --name spark-latest --tgz --mvn mvn -Dhadoop.version=3D2.6.0 -P= hadoop-2.6 -Phive -Phive-thriftserver -DskipTests clean package -U 
=

But I still see WARNINGS like this in the output a= nd no .gz file created:

cp:= /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-= bin-spark-latest/python/test_support/sql/parquet_partitioned/year=3D2015/mon= th=3D10/day=3D26/part-r-00005.gz.parquet: No such file or directory
cp: /usr/local/spark-latest/dist/python/test_support/s= ql/parquet_partitioned/year=3D2015/month=3D9: unable to copy extended attrib= utes to /usr/local/spark-latest/spark-[WARNING] See http://= docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/tes= t_support/sql/parquet_partitioned/year=3D2015/month=3D9: No such file or= directory
cp: /usr/local/spark-late= st/dist/python/test_support/sql/parquet_partitioned/year=3D2015/month=3D9/da= y=3D1: unable to copy extended attributes to /usr/local/spark-latest/spark-[= WARNING] See http://docs.codehaus.org/display/MAVEN= USER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partition= ed/year=3D2015/month=3D9/day=3D1: No such file or directory

On Mon, Oct 26, 2015 at 8:58 PM, Ted Yu <yuzhihong@gmail.com> wrote:
If you u= se the command shown in:
https://github.com/apache/spark/pull/9281
=
You should have got the following:

./di= st/python/test_support/sql/parquet_partitioned/year=3D2014/month=3D9/day=3D1= /part-r-00008.gz.parquet
./dist/python/test_support/sql/parquet_pa= rtitioned/year=3D2015/month=3D9/day=3D1/part-r-00007.gz.parquet
./= dist/python/test_support/sql/parquet_partitioned/year=3D2015/month=3D10/day=3D= 25/part-r-00004.gz.parquet
./dist/python/test_support/sql/parquet_= partitioned/year=3D2015/month=3D10/day=3D25/part-r-00002.gz.parquet
./dist/python/test_support/sql/parquet_partitioned/year=3D2015/month=3D10/= day=3D26/part-r-00005.gz.parquet

On Mon, Oct 26, 2015 a= t 11:47 AM, Kayode Odeyemi <dreyemi@gmail.com> wrote:
I see a lot of stuffs like this af= ter the a successful maven build:

part-r-000= 08.gz.parquet: No such file or directory

Seem= s it fails when it tries to package the build as an archive.
<= br>
I'm using the latest code on github master.

Any ideas please?

On Mon, Oct 26, 2015 at 6:20 PM, Yana Kadiyska <y= ana.kadiyska@gmail.com> wrote:
In 1.4 ./make_distr= ibution produces a .tgz file in the root directory (same directory that = ;make_distribution is in)

On Mon, Oct 26, 2015 at 8:46 AM, Kayode Odeyemi <= span dir=3D"ltr"><= dreyemi@gmail.com> wrote:
Hi,

The ./make_distribution task completed.= However, I can't seem to locate the
.tar.gz file.

<= /div>
Where does Spark save this? or should I just work with the dist di= rectory?

On Fri, Oct 23, 2015 at 4:23 PM, Kayode Odeyemi <= dreyemi@gmail.com= > wrote:
I saw th= is when I tested manually (without ./make-distribution)

D= etected Maven Version: 3.2.2 is not in the allowed range 3.3.3.

So I simply upgraded maven to 3.3.3.

Resolved. Thanks

On Fri, Oct 23, 2015 at 3:17 PM, Sean Owen <sowen@cloudera= .com> wrote:
This doesn't sho= w the actual error output from Maven. I have a strong
guess that you haven't set MAVEN_OPTS to increase the memory Maven can
use.

On Fri, Oct 23, 2015 at 6:14 AM, Kayode Odeyemi <dreyemi@gmail.com> wrote:
> Hi,
>
> I can't seem to get a successful maven build. Please see command output=
> below:
>
> bash-3.2$ ./make-distribution.sh --name spark-latest --tgz --mvn mvn > -Dhadoop.version=3D2.7.0 -Phadoop-2.7 -Phive -Phive-thriftserver -Dskip= Tests
> clean package
> +++ dirname ./make-distribution.sh
> ++ cd .
> ++ pwd
> + SPARK_HOME=3D/usr/local/spark-latest
> + DISTDIR=3D/usr/local/spark-latest/dist
> + SPARK_TACHYON=3Dfalse
> + TACHYON_VERSION=3D0.7.1
> + TACHYON_TGZ=3Dtachyon-0.7.1-bin.tar.gz
> +
> TACHYON_URL=3D= https://github.com/amplab/tachyon/releases/download/v0.7.1/tachyon-0.7.1-bin= .tar.gz
> + MAKE_TGZ=3Dfalse
> + NAME=3Dnone
> + MVN=3D/usr/local/spark-latest/build/mvn
> + ((  12  ))
> + case $1 in
> + NAME=3Dspark-latest
> + shift
> + shift
> + ((  10  ))
> + case $1 in
> + MAKE_TGZ=3Dtrue
> + shift
> + ((  9  ))
> + case $1 in
> + MVN=3Dmvn
> + shift
> + shift
> + ((  7  ))
> + case $1 in
> + break
> + '[' -z /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Hom= e ']'
> + '[' -z /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Hom= e ']'
> ++ command -v git
> + '[' /usr/bin/git ']'
> ++ git rev-parse --short HEAD
> + GITREV=3D487d409
> + '[' '!' -z 487d409 ']'
> + GITREVSTRING=3D' (git revision 487d409)'
> + unset GITREV
> ++ command -v mvn
> + '[' '!' /usr/bin/mvn ']'
> ++ mvn help:evaluate -Dexpression=3Dproject.version -Dhadoop.version=3D= 2.7.0
> -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests clean package
> ++ grep -v INFO
> ++ tail -n 1
> + VERSION=3D'[ERROR] [Help 1]
> http://cwiki.apache.org/co= nfluence/display/MAVEN/MojoExecutionException'
>
> Same output error with JDK 7
>
> Appreciate your help.
>
>










= --Apple-Mail-474D6ECC-F634-452A-A844-7C417E916DF4--