hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chandeep Singh ...@chandeep.com>
Subject Re: Steps to Run Spark Scala job from Oozie on EC2 Hadoop clsuter
Date Mon, 07 Mar 2016 16:02:07 GMT
As a work around you could put your spark-submit statement in a shell script and then use Oozie’s
SSH action to execute that script.

> On Mar 7, 2016, at 3:58 PM, Neelesh Salian <nsalian@cloudera.com> wrote:
> 
> Hi Divya,
> 
> This link should have the details that you need to begin using the Spark Action on Oozie:
> https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html <https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html>
> 
> Thanks.
> 
> On Mon, Mar 7, 2016 at 7:52 AM, Benjamin Kim <bbuild11@gmail.com <mailto:bbuild11@gmail.com>>
wrote:
> To comment…
> 
> At my company, we have not gotten it to work in any other mode than local. If we try
any of the yarn modes, it fails with a “file does not exist” error when trying to locate
the executable jar. I mentioned this to the Hue users group, which we used for this, and they
replied that the Spark Action is very basic implementation and that they will be writing their
own for production use.
> 
> That’s all I know...
> 
>> On Mar 7, 2016, at 1:18 AM, Deepak Sharma <deepakmca05@gmail.com <mailto:deepakmca05@gmail.com>>
wrote:
>> 
>> There is Spark action defined for oozie workflows.
>> Though I am not sure if it supports only Java SPARK jobs or Scala jobs as well.
>> https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html <https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html>
>> Thanks
>> Deepak
>> 
>> On Mon, Mar 7, 2016 at 2:44 PM, Divya Gehlot <divya.htconex@gmail.com <mailto:divya.htconex@gmail.com>>
wrote:
>> Hi,
>> 
>> Could somebody help me by providing the steps /redirect me  to blog/documentation
on how to run Spark job written in scala through Oozie.
>> 
>> Would really appreciate the help.
>> 
>> 
>> 
>> Thanks,
>> Divya 
>> 
>> 
>> 
>> -- 
>> Thanks
>> Deepak
>> www.bigdatabig.com <http://www.bigdatabig.com/>
>> www.keosha.net <http://www.keosha.net/>
> 
> 
> 
> -- 
> Neelesh Srinivas Salian
> Customer Operations Engineer
> 
> 
> 


Mime
View raw message