spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Nist <tsind...@gmail.com>
Subject Re: Maven build failed (Spark master)
Date Tue, 27 Oct 2015 16:39:58 GMT
I issued the same basic command and it worked fine.

RADTech-MBP:spark $ ./make-distribution.sh --name hadoop-2.6 --tgz -Pyarn
-Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -DskipTests

Which created: spark-1.6.0-SNAPSHOT-bin-hadoop-2.6.tgz in the root
directory of the project.

FWIW, the environment was an MBP with OS X 10.10.5 and Java:

java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)

-Todd

On Tue, Oct 27, 2015 at 12:17 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> I used the following command:
> make-distribution.sh --name custom-spark --tgz -Phadoop-2.4 -Phive
> -Phive-thriftserver -Pyarn
>
> spark-1.6.0-SNAPSHOT-bin-custom-spark.tgz was generated (with patch from
> SPARK-11348)
>
> Can you try above command ?
>
> Thanks
>
> On Tue, Oct 27, 2015 at 7:03 AM, Kayode Odeyemi <dreyemi@gmail.com> wrote:
>
>> Ted, I switched to this:
>>
>> ./make-distribution.sh --name spark-latest --tgz -Dhadoop.version=2.6.0
>> -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn -DskipTests clean package -U
>>
>> Same error. No .gz file. Here's the bottom output log:
>>
>> + rm -rf /home/emperor/javaprojects/spark/dist
>> + mkdir -p /home/emperor/javaprojects/spark/dist/lib
>> + echo 'Spark [WARNING] See
>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin (git revision
>> 3689beb) built for Hadoop [WARNING] See
>> http://docs.codehaus.org/display/MAVENUSER/Shade+Pl
>> + echo 'Build flags: -Dhadoop.version=2.6.0' -Phadoop-2.6 -Phive
>> -Phive-thriftserver -Pyarn -DskipTests clean package -U
>> + cp
>> /home/emperor/javaprojects/spark/assembly/target/scala-2.10/spark-assembly-1.6.0-SNAPSHOT-hadoop2.6.0.jar
>> /home/emperor/javaprojects/spark/dist/lib/
>> + cp
>> /home/emperor/javaprojects/spark/examples/target/scala-2.10/spark-examples-1.6.0-SNAPSHOT-hadoop2.6.0.jar
>> /home/emperor/javaprojects/spark/dist/lib/
>> + cp
>> /home/emperor/javaprojects/spark/network/yarn/target/scala-2.10/spark-1.6.0-SNAPSHOT-yarn-shuffle.jar
>> /home/emperor/javaprojects/spark/dist/lib/
>> + mkdir -p /home/emperor/javaprojects/spark/dist/examples/src/main
>> + cp -r /home/emperor/javaprojects/spark/examples/src/main
>> /home/emperor/javaprojects/spark/dist/examples/src/
>> + '[' 1 == 1 ']'
>> + cp
>> /home/emperor/javaprojects/spark/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar
>> /home/emperor/javaprojects/spark/lib_managed/jars/datanucleus-core-3.2.10.jar
>> /home/emperor/javaprojects
>> ed/jars/datanucleus-rdbms-3.2.9.jar
>> /home/emperor/javaprojects/spark/dist/lib/
>> + cp /home/emperor/javaprojects/spark/LICENSE
>> /home/emperor/javaprojects/spark/dist
>> + cp -r /home/emperor/javaprojects/spark/licenses
>> /home/emperor/javaprojects/spark/dist
>> + cp /home/emperor/javaprojects/spark/NOTICE
>> /home/emperor/javaprojects/spark/dist
>> + '[' -e /home/emperor/javaprojects/spark/CHANGES.txt ']'
>> + cp -r /home/emperor/javaprojects/spark/data
>> /home/emperor/javaprojects/spark/dist
>> + mkdir /home/emperor/javaprojects/spark/dist/conf
>> + cp /home/emperor/javaprojects/spark/conf/docker.properties.template
>> /home/emperor/javaprojects/spark/conf/fairscheduler.xml.template
>> /home/emperor/javaprojects/spark/conf/log4j.properties
>> emperor/javaprojects/spark/conf/metrics.properties.template
>> /home/emperor/javaprojects/spark/conf/slaves.template
>> /home/emperor/javaprojects/spark/conf/spark-defaults.conf.template /home/em
>> ts/spark/conf/spark-env.sh.template
>> /home/emperor/javaprojects/spark/dist/conf
>> + cp /home/emperor/javaprojects/spark/README.md
>> /home/emperor/javaprojects/spark/dist
>> + cp -r /home/emperor/javaprojects/spark/bin
>> /home/emperor/javaprojects/spark/dist
>> + cp -r /home/emperor/javaprojects/spark/python
>> /home/emperor/javaprojects/spark/dist
>> + cp -r /home/emperor/javaprojects/spark/sbin
>> /home/emperor/javaprojects/spark/dist
>> + cp -r /home/emperor/javaprojects/spark/ec2
>> /home/emperor/javaprojects/spark/dist
>> + '[' -d /home/emperor/javaprojects/spark/R/lib/SparkR ']'
>> + '[' false == true ']'
>> + '[' true == true ']'
>> + TARDIR_NAME='spark-[WARNING] See
>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest'
>> + TARDIR='/home/emperor/javaprojects/spark/spark-[WARNING] See
>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest'
>> + rm -rf '/home/emperor/javaprojects/spark/spark-[WARNING] See
>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest'
>> + cp -r /home/emperor/javaprojects/spark/dist
>> '/home/emperor/javaprojects/spark/spark-[WARNING] See
>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest'
>> cp: cannot create directory
>> `/home/emperor/javaprojects/spark/spark-[WARNING] See
>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest':
>> No such file or directory
>>
>>
>> On Tue, Oct 27, 2015 at 2:14 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>>> Can you try the same command shown in the pull request ?
>>>
>>> Thanks
>>>
>>> On Oct 27, 2015, at 12:40 AM, Kayode Odeyemi <dreyemi@gmail.com> wrote:
>>>
>>> Thank you.
>>>
>>> But I'm getting same warnings and it's still preventing the archive from
>>> being generated.
>>>
>>> I've ran this on both OSX Lion and Ubuntu 12. Same error. No .gz file
>>>
>>> On Mon, Oct 26, 2015 at 9:10 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>>
>>>> Looks like '-Pyarn' was missing in your command.
>>>>
>>>> On Mon, Oct 26, 2015 at 12:06 PM, Kayode Odeyemi <dreyemi@gmail.com>
>>>> wrote:
>>>>
>>>>> I used this command which is synonymous to what you have:
>>>>>
>>>>> ./make-distribution.sh --name spark-latest --tgz --mvn mvn
>>>>> -Dhadoop.version=2.6.0 -Phadoop-2.6 -Phive -Phive-thriftserver -DskipTests
>>>>> clean package -U
>>>>>
>>>>> But I still see WARNINGS like this in the output and no .gz file
>>>>> created:
>>>>>
>>>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/.part-r-00005.gz.parquet.crc:
>>>>> No such file or directory
>>>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet:
>>>>> No such file or directory
>>>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9:
>>>>> No such file or directory
>>>>> cp:
>>>>> /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9:
>>>>> unable to copy extended attributes to
>>>>> /usr/local/spark-latest/spark-[WARNING] See
>>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9:
>>>>> No such file or directory
>>>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1:
>>>>> No such file or directory
>>>>> cp:
>>>>> /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1:
>>>>> unable to copy extended attributes to
>>>>> /usr/local/spark-latest/spark-[WARNING] See
>>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1:
>>>>> No such file or directory
>>>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/.part-r-00007.gz.parquet.crc:
>>>>> No such file or directory
>>>>>
>>>>> On Mon, Oct 26, 2015 at 8:58 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>>>>
>>>>>> If you use the command shown in:
>>>>>> https://github.com/apache/spark/pull/9281
>>>>>>
>>>>>> You should have got the following:
>>>>>>
>>>>>>
>>>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/part-r-00008.gz.parquet
>>>>>>
>>>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/part-r-00007.gz.parquet
>>>>>>
>>>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00004.gz.parquet
>>>>>>
>>>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00002.gz.parquet
>>>>>>
>>>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet
>>>>>>
>>>>>> On Mon, Oct 26, 2015 at 11:47 AM, Kayode Odeyemi <dreyemi@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I see a lot of stuffs like this after the a successful maven
build:
>>>>>>>
>>>>>>> cp: /usr/local/spark-latest/spark-[WARNING] See
>>>>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/
>>>>>>> part-r-00008.gz.parquet: No such file or directory
>>>>>>>
>>>>>>> Seems it fails when it tries to package the build as an archive.
>>>>>>>
>>>>>>> I'm using the latest code on github master.
>>>>>>>
>>>>>>> Any ideas please?
>>>>>>>
>>>>>>> On Mon, Oct 26, 2015 at 6:20 PM, Yana Kadiyska <
>>>>>>> yana.kadiyska@gmail.com> wrote:
>>>>>>>
>>>>>>>> In 1.4 ./make_distribution produces a .tgz file in the root
>>>>>>>> directory (same directory that make_distribution is in)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 26, 2015 at 8:46 AM, Kayode Odeyemi <dreyemi@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> The ./make_distribution task completed. However, I can't
seem to
>>>>>>>>> locate the
>>>>>>>>> .tar.gz file.
>>>>>>>>>
>>>>>>>>> Where does Spark save this? or should I just work with
the dist
>>>>>>>>> directory?
>>>>>>>>>
>>>>>>>>> On Fri, Oct 23, 2015 at 4:23 PM, Kayode Odeyemi <dreyemi@gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> I saw this when I tested manually (without ./make-distribution)
>>>>>>>>>>
>>>>>>>>>> Detected Maven Version: 3.2.2 is not in the allowed
range 3.3.3.
>>>>>>>>>>
>>>>>>>>>> So I simply upgraded maven to 3.3.3.
>>>>>>>>>>
>>>>>>>>>> Resolved. Thanks
>>>>>>>>>>
>>>>>>>>>> On Fri, Oct 23, 2015 at 3:17 PM, Sean Owen <sowen@cloudera.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> This doesn't show the actual error output from
Maven. I have a
>>>>>>>>>>> strong
>>>>>>>>>>> guess that you haven't set MAVEN_OPTS to increase
the memory
>>>>>>>>>>> Maven can
>>>>>>>>>>> use.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Oct 23, 2015 at 6:14 AM, Kayode Odeyemi
<
>>>>>>>>>>> dreyemi@gmail.com> wrote:
>>>>>>>>>>> > Hi,
>>>>>>>>>>> >
>>>>>>>>>>> > I can't seem to get a successful maven build.
Please see
>>>>>>>>>>> command output
>>>>>>>>>>> > below:
>>>>>>>>>>> >
>>>>>>>>>>> > bash-3.2$ ./make-distribution.sh --name
spark-latest --tgz
>>>>>>>>>>> --mvn mvn
>>>>>>>>>>> > -Dhadoop.version=2.7.0 -Phadoop-2.7 -Phive
-Phive-thriftserver
>>>>>>>>>>> -DskipTests
>>>>>>>>>>> > clean package
>>>>>>>>>>> > +++ dirname ./make-distribution.sh
>>>>>>>>>>> > ++ cd .
>>>>>>>>>>> > ++ pwd
>>>>>>>>>>> > + SPARK_HOME=/usr/local/spark-latest
>>>>>>>>>>> > + DISTDIR=/usr/local/spark-latest/dist
>>>>>>>>>>> > + SPARK_TACHYON=false
>>>>>>>>>>> > + TACHYON_VERSION=0.7.1
>>>>>>>>>>> > + TACHYON_TGZ=tachyon-0.7.1-bin.tar.gz
>>>>>>>>>>> > +
>>>>>>>>>>> > TACHYON_URL=
>>>>>>>>>>> https://github.com/amplab/tachyon/releases/download/v0.7.1/tachyon-0.7.1-bin.tar.gz
>>>>>>>>>>> > + MAKE_TGZ=false
>>>>>>>>>>> > + NAME=none
>>>>>>>>>>> > + MVN=/usr/local/spark-latest/build/mvn
>>>>>>>>>>> > + ((  12  ))
>>>>>>>>>>> > + case $1 in
>>>>>>>>>>> > + NAME=spark-latest
>>>>>>>>>>> > + shift
>>>>>>>>>>> > + shift
>>>>>>>>>>> > + ((  10  ))
>>>>>>>>>>> > + case $1 in
>>>>>>>>>>> > + MAKE_TGZ=true
>>>>>>>>>>> > + shift
>>>>>>>>>>> > + ((  9  ))
>>>>>>>>>>> > + case $1 in
>>>>>>>>>>> > + MVN=mvn
>>>>>>>>>>> > + shift
>>>>>>>>>>> > + shift
>>>>>>>>>>> > + ((  7  ))
>>>>>>>>>>> > + case $1 in
>>>>>>>>>>> > + break
>>>>>>>>>>> > + '[' -z
>>>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home
']'
>>>>>>>>>>> > + '[' -z
>>>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home
']'
>>>>>>>>>>> > ++ command -v git
>>>>>>>>>>> > + '[' /usr/bin/git ']'
>>>>>>>>>>> > ++ git rev-parse --short HEAD
>>>>>>>>>>> > + GITREV=487d409
>>>>>>>>>>> > + '[' '!' -z 487d409 ']'
>>>>>>>>>>> > + GITREVSTRING=' (git revision 487d409)'
>>>>>>>>>>> > + unset GITREV
>>>>>>>>>>> > ++ command -v mvn
>>>>>>>>>>> > + '[' '!' /usr/bin/mvn ']'
>>>>>>>>>>> > ++ mvn help:evaluate -Dexpression=project.version
>>>>>>>>>>> -Dhadoop.version=2.7.0
>>>>>>>>>>> > -Phadoop-2.7 -Phive -Phive-thriftserver
-DskipTests clean
>>>>>>>>>>> package
>>>>>>>>>>> > ++ grep -v INFO
>>>>>>>>>>> > ++ tail -n 1
>>>>>>>>>>> > + VERSION='[ERROR] [Help 1]
>>>>>>>>>>> >
>>>>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
>>>>>>>>>>> '
>>>>>>>>>>> >
>>>>>>>>>>> > Same output error with JDK 7
>>>>>>>>>>> >
>>>>>>>>>>> > Appreciate your help.
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message