ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Senthil <senthi...@gmail.com>
Subject Re: HDP 2.2 + Spark 1.4
Date Thu, 10 Sep 2015 21:20:59 GMT
All,

I noticed Hortonworks has published Spark 1.4.1 Preview packages.
http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/

It worked for me with HDP 2.2

Thanks

Senthil

On Wed, Sep 9, 2015 at 12:07 AM, Senthil <senthil69@gmail.com> wrote:

> Hi Christian
>
> Thank you so much. the build meets my requirements. However I have trouble
> in running the spark job after installation. The jobs are failing with the
> following error.  I get the same error when I run SparkPI app or
> sparkshell. Also I could not see the image illustration to modify yarn.site
> (hdp.stack and hdp.version) . Could you tell us how to set hdp-stack. Any
> help will be greatly appreciated.
>
>
> Here is the error on console as well as job log
>
> Application application_1441682258490_0008 failed 2 times due to AM
> Container for appattempt_1441682258490_0008_000002 exited with exitCode: 1
> For more detailed output, check application tracking page:
> http://ip-10-0-3-206.us-west-2.compute.internal:8088/proxy/application_1441682258490_0008/Then,
> click on links to logs of each attempt.
> Diagnostics: Exception from container-launch.
> Container id: container_1441682258490_0008_02_000001
> Exit code: 1
> Exception message:
> /hadoop/yarn/local/usercache/root/appcache/application_1441682258490_0008/container_1441682258490_0008_02_000001/launch_container.sh:
> line 27:
> $PWD:$PWD/__hadoop_conf__:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:
> bad substitution
> Stack trace: ExitCodeException exitCode=1:
> /hadoop/yarn/local/usercache/root/appcache/application_1441682258490_0008/container_1441682258490_0008_02_000001/launch_container.sh:
> line 27:
> $PWD:$PWD/__hadoop_conf__:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:
> bad substitution
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1
> Failing this attempt. Failing the application.
>
>
>
> Regards,
>
> Senthil
>
> On Mon, Sep 7, 2015 at 10:42 PM, Christian Tzolov <ctzolov@pivotal.io>
> wrote:
>
>> Hi Senthil,
>>
>> I've build Spark 1.4 RPMs using BigTop. Here are the instructions:
>> http://blog.tzolov.net/2015/07/how-to-install-spark-140-on-pivotalhd.html?view=sidebar
>>
>> it has been tested with HDP2.2, PHD3.0. Make sure to set the right
>> stack.name and stack.versions!
>>
>> Cheers,
>> Christian
>>
>> On 7 September 2015 at 11:47, Senthil <senthil69@gmail.com> wrote:
>>
>>>
>>> Has anyone built HDP compatible RPMs (CentOS6) for Spark 1.4?
>>>
>>> Using Ambari, I am trying to automate Hadoop cluster setup with Hadoop
>>> 2.6, Hive 0.14 and Spark 1.4. Ambari supports HDP 2.2 and able to install
>>> Hadoop 2.6 and Hive. I like to customize the Ambari stack to include Spark
>>> 1.4 instead of 1.2 (bundled) .
>>>
>>> Regards,
>>>
>>> - Senthil
>>>
>>
>>
>>
>> --
>> Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution
>> Architect, EMEA Practice Team | Pivotal <http://pivotal.io/>
>> ctzolov@pivotal.io|+31610285517
>>
>
>
>
> --
> - Senthil
>



-- 
- Senthil

Mime
View raw message