spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deenar Toraskar <deenar.toras...@gmail.com>
Subject Re: Spark 1.5 on CDH 5.4.0
Date Fri, 23 Oct 2015 16:31:03 GMT
I got this working. For others trying this It turns out in Spark 1.3/CDH5.4

spark.yarn.jar=local:/opt/cloudera/parcels/....

I had changed this to reflect the 1.5.1 version of spark assembly jar

spark.yarn.jar=/opt/spark-1.5.1-bin/...

and this didn't work, I had to drop the "local:" prefix

spark.yarn.jar=/opt/spark-1.5.1-bin/...

Regards
Deenar

On 23 October 2015 at 17:30, Deenar Toraskar <
deenar.toraskar@thinkreactive.co.uk> wrote:

> I got this working. For others trying this It turns out in Spark 1.3/CDH5.4
>
> spark.yarn.jar=local:/opt/cloudera/parcels/....
>
> I had changed this to reflect the 1.5.1 version of spark assembly jar
>
> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>
> and this didn't work, I had to drop the "local:" prefix
>
> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>
> Regards
> Deenar
>
>
>
>
> *Think Reactive Ltd*
> deenar.toraskar@thinkreactive.co.uk
> 07714140812
>
>
>
> On 23 October 2015 at 13:34, Deenar Toraskar <deenar.toraskar@gmail.com>
> wrote:
>
>> Sandy
>>
>> The assembly jar does contain org.apache.spark.deploy.yarn.ExecutorLauncher.
>> I am trying to find out how i can increase the logging level, so I know the
>> exact classpath used by Yarn ContainerLaunch.
>>
>> Deenar
>>
>> On 23 October 2015 at 03:30, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
>>
>>> Hi Deenar,
>>>
>>> The version of Spark you have may not be compiled with YARN support.  If
>>> you inspect the contents of the assembly jar, does
>>> org.apache.spark.deploy.yarn.ExecutorLauncher exist?  If not, you'll
>>> need to find a version that does have the YARN classes.  You can also build
>>> your own using the -Pyarn flag.
>>>
>>> -Sandy
>>>
>>> On Thu, Oct 22, 2015 at 9:04 AM, Deenar Toraskar <
>>> deenar.toraskar@gmail.com> wrote:
>>>
>>>> Hi I have got the prebuilt version of Spark 1.5 for Hadoop 2.6 (
>>>> http://www.apache.org/dyn/closer.lua/spark/spark-1.5.1/spark-1.5.1-bin-hadoop2.6.tgz)
>>>> working with CDH 5.4.0 in local mode on a cluster with Kerberos. It works
>>>> well including connecting to the Hive metastore. I am facing an issue
>>>> running spark jobs in yarn-client/yarn-cluster mode. The executors fail to
>>>> start as java cannot find ExecutorLauncher. Error: Could not find or
>>>> load main class org.apache.spark.deploy.yarn.ExecutorLauncher client
>>>> token: N/Adiagnostics: Application application_1443531450011_13437
>>>> failed 2 times due to AM Container for
>>>> appattempt_1443531450011_13437_000002 exited with exitCode: 1Stack
>>>> trace: ExitCodeException exitCode=1:at
>>>> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)at
>>>> org.apache.hadoop.util.Shell.run(Shell.java:455)at
>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)at
>>>> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:293)at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)at
>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at
>>>> java.lang.Thread.run(Thread.java:745) Any ideas as to what might be
>>>> going wrong. Also how can I turn on more detailed logging to see what
>>>> command line is being run by Yarn to launch containers? RegardsDeenar
>>>>
>>>
>>>
>>
>

Mime
View raw message