samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navina Ramesh <nram...@linkedin.com.INVALID>
Subject Re: Deploying samza job to a remote yarn resource manager
Date Tue, 23 Jun 2015 21:35:29 GMT
Hi George,
JobRunner is meant to run on the localhost only. You have write a wrapper
script (perhaps in python or bash) to ssh to the RM and then run the
JobRunner. RM machine should already have HADOOP_YARN_HOME and
HADOOP_YARN_CONF_DIR configured, if the RM is running. So, probably not
necessary to include in the classpath.

Cheers,
Navina

On 6/23/15, 8:44 AM, "George Li" <gli@ca.ibm.com> wrote:

>Hi Navina and Milinda,
>
>Thanks for your reply. I get it working by setting HADOOP_YARN_HOME and
>HADOOP_CONF_DIR
>
>export HADOOP_YARN_HOME=$HOME/hadoop-2.6.0/
>export HADOOP_CONF_DIR=$HOME/hadoop-2.6.0/etc/hadoop
>
>,i.e., similar to what run-class.sh does
>
>
>I am also trying to figure out how to deploy samza job by code. By
>porting 
>over org.apache.samza.job.JobRunner, I am able to deploy a job as long as
>yarn resource manager is on the same host, but how do I pass the remote
>resource manager address? Maybe I should include  HADOOP_YARN_HOME and
>HADOOP_CONF_DIR as classpath?
>
>George
>
>
>
>
>From:   Navina Ramesh <nramesh@linkedin.com.INVALID>
>To:     "dev@samza.apache.org" <dev@samza.apache.org>,
>Date:   22/06/2015 05:57 PM
>Subject:        Re: Deploying samza job to a remote yarn resource manager
>
>
>
>Hi George,
>
>Looks like there is a formatting issue on the samza website. The
>environment variable name should be HADOOP_YARN_HOME.
>In your case, it should point to hadoop home: $HOME/hadoop-2.6.0/
>
>Please let me know if it works.
>
>Cheers!
>Navina
>
>On 6/22/15, 2:11 PM, "George Li" <gli@ca.ibm.com> wrote:
>
>>Hi,
>>
>>According to
>>
>>https://samza.apache.org/learn/documentation/0.9/jobs/yarn-jobs.html
>>
>>, to deploy samza jobs to a remote yarn resource manager, I need to set
>>the HADOOPYARNHOME environment variable. What should be the value of this
>>variable if my yarn-site.xml is at
>>$HOME/hadoop-2.6.0/etc/hadoop/yarn-site.xml?
>>
>>Thanks,
>>
>>George
>
>


Mime
View raw message