spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kayode Odeyemi <drey...@gmail.com>
Subject Re: Maven build failed (Spark master)
Date Tue, 27 Oct 2015 14:03:46 GMT
Ted, I switched to this:

./make-distribution.sh --name spark-latest --tgz -Dhadoop.version=2.6.0
-Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn -DskipTests clean package -U

Same error. No .gz file. Here's the bottom output log:

+ rm -rf /home/emperor/javaprojects/spark/dist
+ mkdir -p /home/emperor/javaprojects/spark/dist/lib
+ echo 'Spark [WARNING] See
http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin (git revision
3689beb) built for Hadoop [WARNING] See
http://docs.codehaus.org/display/MAVENUSER/Shade+Pl
+ echo 'Build flags: -Dhadoop.version=2.6.0' -Phadoop-2.6 -Phive
-Phive-thriftserver -Pyarn -DskipTests clean package -U
+ cp
/home/emperor/javaprojects/spark/assembly/target/scala-2.10/spark-assembly-1.6.0-SNAPSHOT-hadoop2.6.0.jar
/home/emperor/javaprojects/spark/dist/lib/
+ cp
/home/emperor/javaprojects/spark/examples/target/scala-2.10/spark-examples-1.6.0-SNAPSHOT-hadoop2.6.0.jar
/home/emperor/javaprojects/spark/dist/lib/
+ cp
/home/emperor/javaprojects/spark/network/yarn/target/scala-2.10/spark-1.6.0-SNAPSHOT-yarn-shuffle.jar
/home/emperor/javaprojects/spark/dist/lib/
+ mkdir -p /home/emperor/javaprojects/spark/dist/examples/src/main
+ cp -r /home/emperor/javaprojects/spark/examples/src/main
/home/emperor/javaprojects/spark/dist/examples/src/
+ '[' 1 == 1 ']'
+ cp
/home/emperor/javaprojects/spark/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar
/home/emperor/javaprojects/spark/lib_managed/jars/datanucleus-core-3.2.10.jar
/home/emperor/javaprojects
ed/jars/datanucleus-rdbms-3.2.9.jar
/home/emperor/javaprojects/spark/dist/lib/
+ cp /home/emperor/javaprojects/spark/LICENSE
/home/emperor/javaprojects/spark/dist
+ cp -r /home/emperor/javaprojects/spark/licenses
/home/emperor/javaprojects/spark/dist
+ cp /home/emperor/javaprojects/spark/NOTICE
/home/emperor/javaprojects/spark/dist
+ '[' -e /home/emperor/javaprojects/spark/CHANGES.txt ']'
+ cp -r /home/emperor/javaprojects/spark/data
/home/emperor/javaprojects/spark/dist
+ mkdir /home/emperor/javaprojects/spark/dist/conf
+ cp /home/emperor/javaprojects/spark/conf/docker.properties.template
/home/emperor/javaprojects/spark/conf/fairscheduler.xml.template
/home/emperor/javaprojects/spark/conf/log4j.properties
emperor/javaprojects/spark/conf/metrics.properties.template
/home/emperor/javaprojects/spark/conf/slaves.template
/home/emperor/javaprojects/spark/conf/spark-defaults.conf.template /home/em
ts/spark/conf/spark-env.sh.template
/home/emperor/javaprojects/spark/dist/conf
+ cp /home/emperor/javaprojects/spark/README.md
/home/emperor/javaprojects/spark/dist
+ cp -r /home/emperor/javaprojects/spark/bin
/home/emperor/javaprojects/spark/dist
+ cp -r /home/emperor/javaprojects/spark/python
/home/emperor/javaprojects/spark/dist
+ cp -r /home/emperor/javaprojects/spark/sbin
/home/emperor/javaprojects/spark/dist
+ cp -r /home/emperor/javaprojects/spark/ec2
/home/emperor/javaprojects/spark/dist
+ '[' -d /home/emperor/javaprojects/spark/R/lib/SparkR ']'
+ '[' false == true ']'
+ '[' true == true ']'
+ TARDIR_NAME='spark-[WARNING] See
http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest'
+ TARDIR='/home/emperor/javaprojects/spark/spark-[WARNING] See
http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest'
+ rm -rf '/home/emperor/javaprojects/spark/spark-[WARNING] See
http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest'
+ cp -r /home/emperor/javaprojects/spark/dist
'/home/emperor/javaprojects/spark/spark-[WARNING] See
http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest'
cp: cannot create directory
`/home/emperor/javaprojects/spark/spark-[WARNING] See
http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest':
No such file or directory


On Tue, Oct 27, 2015 at 2:14 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Can you try the same command shown in the pull request ?
>
> Thanks
>
> On Oct 27, 2015, at 12:40 AM, Kayode Odeyemi <dreyemi@gmail.com> wrote:
>
> Thank you.
>
> But I'm getting same warnings and it's still preventing the archive from
> being generated.
>
> I've ran this on both OSX Lion and Ubuntu 12. Same error. No .gz file
>
> On Mon, Oct 26, 2015 at 9:10 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> Looks like '-Pyarn' was missing in your command.
>>
>> On Mon, Oct 26, 2015 at 12:06 PM, Kayode Odeyemi <dreyemi@gmail.com>
>> wrote:
>>
>>> I used this command which is synonymous to what you have:
>>>
>>> ./make-distribution.sh --name spark-latest --tgz --mvn mvn
>>> -Dhadoop.version=2.6.0 -Phadoop-2.6 -Phive -Phive-thriftserver -DskipTests
>>> clean package -U
>>>
>>> But I still see WARNINGS like this in the output and no .gz file created:
>>>
>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/.part-r-00005.gz.parquet.crc:
>>> No such file or directory
>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet:
>>> No such file or directory
>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9:
>>> No such file or directory
>>> cp:
>>> /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9:
>>> unable to copy extended attributes to
>>> /usr/local/spark-latest/spark-[WARNING] See
>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9:
>>> No such file or directory
>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1:
>>> No such file or directory
>>> cp:
>>> /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1:
>>> unable to copy extended attributes to
>>> /usr/local/spark-latest/spark-[WARNING] See
>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1:
>>> No such file or directory
>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/.part-r-00007.gz.parquet.crc:
>>> No such file or directory
>>>
>>> On Mon, Oct 26, 2015 at 8:58 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>>
>>>> If you use the command shown in:
>>>> https://github.com/apache/spark/pull/9281
>>>>
>>>> You should have got the following:
>>>>
>>>>
>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/part-r-00008.gz.parquet
>>>>
>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/part-r-00007.gz.parquet
>>>>
>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00004.gz.parquet
>>>>
>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00002.gz.parquet
>>>>
>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet
>>>>
>>>> On Mon, Oct 26, 2015 at 11:47 AM, Kayode Odeyemi <dreyemi@gmail.com>
>>>> wrote:
>>>>
>>>>> I see a lot of stuffs like this after the a successful maven build:
>>>>>
>>>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/
>>>>> part-r-00008.gz.parquet: No such file or directory
>>>>>
>>>>> Seems it fails when it tries to package the build as an archive.
>>>>>
>>>>> I'm using the latest code on github master.
>>>>>
>>>>> Any ideas please?
>>>>>
>>>>> On Mon, Oct 26, 2015 at 6:20 PM, Yana Kadiyska <
>>>>> yana.kadiyska@gmail.com> wrote:
>>>>>
>>>>>> In 1.4 ./make_distribution produces a .tgz file in the root
>>>>>> directory (same directory that make_distribution is in)
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 26, 2015 at 8:46 AM, Kayode Odeyemi <dreyemi@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> The ./make_distribution task completed. However, I can't seem
to
>>>>>>> locate the
>>>>>>> .tar.gz file.
>>>>>>>
>>>>>>> Where does Spark save this? or should I just work with the dist
>>>>>>> directory?
>>>>>>>
>>>>>>> On Fri, Oct 23, 2015 at 4:23 PM, Kayode Odeyemi <dreyemi@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I saw this when I tested manually (without ./make-distribution)
>>>>>>>>
>>>>>>>> Detected Maven Version: 3.2.2 is not in the allowed range
3.3.3.
>>>>>>>>
>>>>>>>> So I simply upgraded maven to 3.3.3.
>>>>>>>>
>>>>>>>> Resolved. Thanks
>>>>>>>>
>>>>>>>> On Fri, Oct 23, 2015 at 3:17 PM, Sean Owen <sowen@cloudera.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> This doesn't show the actual error output from Maven.
I have a
>>>>>>>>> strong
>>>>>>>>> guess that you haven't set MAVEN_OPTS to increase the
memory Maven
>>>>>>>>> can
>>>>>>>>> use.
>>>>>>>>>
>>>>>>>>> On Fri, Oct 23, 2015 at 6:14 AM, Kayode Odeyemi <dreyemi@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> > Hi,
>>>>>>>>> >
>>>>>>>>> > I can't seem to get a successful maven build. Please
see command
>>>>>>>>> output
>>>>>>>>> > below:
>>>>>>>>> >
>>>>>>>>> > bash-3.2$ ./make-distribution.sh --name spark-latest
--tgz --mvn
>>>>>>>>> mvn
>>>>>>>>> > -Dhadoop.version=2.7.0 -Phadoop-2.7 -Phive -Phive-thriftserver
>>>>>>>>> -DskipTests
>>>>>>>>> > clean package
>>>>>>>>> > +++ dirname ./make-distribution.sh
>>>>>>>>> > ++ cd .
>>>>>>>>> > ++ pwd
>>>>>>>>> > + SPARK_HOME=/usr/local/spark-latest
>>>>>>>>> > + DISTDIR=/usr/local/spark-latest/dist
>>>>>>>>> > + SPARK_TACHYON=false
>>>>>>>>> > + TACHYON_VERSION=0.7.1
>>>>>>>>> > + TACHYON_TGZ=tachyon-0.7.1-bin.tar.gz
>>>>>>>>> > +
>>>>>>>>> > TACHYON_URL=
>>>>>>>>> https://github.com/amplab/tachyon/releases/download/v0.7.1/tachyon-0.7.1-bin.tar.gz
>>>>>>>>> > + MAKE_TGZ=false
>>>>>>>>> > + NAME=none
>>>>>>>>> > + MVN=/usr/local/spark-latest/build/mvn
>>>>>>>>> > + ((  12  ))
>>>>>>>>> > + case $1 in
>>>>>>>>> > + NAME=spark-latest
>>>>>>>>> > + shift
>>>>>>>>> > + shift
>>>>>>>>> > + ((  10  ))
>>>>>>>>> > + case $1 in
>>>>>>>>> > + MAKE_TGZ=true
>>>>>>>>> > + shift
>>>>>>>>> > + ((  9  ))
>>>>>>>>> > + case $1 in
>>>>>>>>> > + MVN=mvn
>>>>>>>>> > + shift
>>>>>>>>> > + shift
>>>>>>>>> > + ((  7  ))
>>>>>>>>> > + case $1 in
>>>>>>>>> > + break
>>>>>>>>> > + '[' -z
>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home
']'
>>>>>>>>> > + '[' -z
>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home
']'
>>>>>>>>> > ++ command -v git
>>>>>>>>> > + '[' /usr/bin/git ']'
>>>>>>>>> > ++ git rev-parse --short HEAD
>>>>>>>>> > + GITREV=487d409
>>>>>>>>> > + '[' '!' -z 487d409 ']'
>>>>>>>>> > + GITREVSTRING=' (git revision 487d409)'
>>>>>>>>> > + unset GITREV
>>>>>>>>> > ++ command -v mvn
>>>>>>>>> > + '[' '!' /usr/bin/mvn ']'
>>>>>>>>> > ++ mvn help:evaluate -Dexpression=project.version
>>>>>>>>> -Dhadoop.version=2.7.0
>>>>>>>>> > -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests
clean package
>>>>>>>>> > ++ grep -v INFO
>>>>>>>>> > ++ tail -n 1
>>>>>>>>> > + VERSION='[ERROR] [Help 1]
>>>>>>>>> >
>>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
>>>>>>>>> '
>>>>>>>>> >
>>>>>>>>> > Same output error with JDK 7
>>>>>>>>> >
>>>>>>>>> > Appreciate your help.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message