hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dileep kumar <dileep...@gmail.com>
Subject Re: Steps to Run Spark Scala job from Oozie on EC2 Hadoop clsuter
Date Tue, 08 Mar 2016 06:52:14 GMT
Hi Divya,

Please find below code to invoke spark from oozie.

Oozie file:
+++++++++++++

<workflow-app xmlns="uri:oozie:workflow:0.2" name="DE_2_workflow_US">
        <start to="spark-scala"/>
        <action name="mysparkscala">
                <shell xmlns="uri:oozie:shell-action:0.1">
                        <job-tracker>maprfs:///</job-tracker>
                        <name-node>maprfs:///</name-node>
                        <configuration>
                                <property>
                                        <name>mapred.job.queue.name</name>

                                        <value>dileep</value>
                                </property>
                        </configuration>
                        <exec>sparkshell.sh</exec>

<file>/ggg/gms/gmsrffr/dev/dileep/sparkshell.sh</file>
                  <file>/axp/gms/gmsrffr/dev/dileep/sparkshell.scala</file>
                </shell>
                <ok to="end" />
                <error to="fail" />
        </action>
        <kill name="fail">
                 <message>Java failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
                      </kill>
        <end name="end" />
</workflow-app>


#############
SparkShell.sh
##############

/opt/mapr/spark/spark-1.2.1/bin/spark-shell --name perf108pret1
--num-executors 1 --executor-cores 1 --executor-memory 1G --driver-memory
2G -i sparkshell.scala
exit 0

On Mon, Mar 7, 2016 at 9:32 PM, Chandeep Singh <cs@chandeep.com> wrote:

> As a work around you could put your spark-submit statement in a shell
> script and then use Oozie’s SSH action to execute that script.
>
> On Mar 7, 2016, at 3:58 PM, Neelesh Salian <nsalian@cloudera.com> wrote:
>
> Hi Divya,
>
> This link should have the details that you need to begin using the Spark
> Action on Oozie:
> https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html
>
> Thanks.
>
> On Mon, Mar 7, 2016 at 7:52 AM, Benjamin Kim <bbuild11@gmail.com> wrote:
>
>> To comment…
>>
>> At my company, we have not gotten it to work in any other mode than
>> local. If we try any of the yarn modes, it fails with a “file does not
>> exist” error when trying to locate the executable jar. I mentioned this to
>> the Hue users group, which we used for this, and they replied that the
>> Spark Action is very basic implementation and that they will be writing
>> their own for production use.
>>
>> That’s all I know...
>>
>> On Mar 7, 2016, at 1:18 AM, Deepak Sharma <deepakmca05@gmail.com> wrote:
>>
>> There is Spark action defined for oozie workflows.
>> Though I am not sure if it supports only Java SPARK jobs or Scala jobs as
>> well.
>> https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html
>> Thanks
>> Deepak
>>
>> On Mon, Mar 7, 2016 at 2:44 PM, Divya Gehlot <divya.htconex@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Could somebody help me by providing the steps /redirect me  to
>>> blog/documentation on how to run Spark job written in scala through Oozie.
>>>
>>> Would really appreciate the help.
>>>
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>>
>>
>>
>
>
> --
> Neelesh Srinivas Salian
> Customer Operations Engineer
>
>
>
>
>


-- 
Regards
Dileep Kumar
+91 9742443302

Mime
View raw message