beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sathish Jayaraman <sath...@quartic.ai>
Subject Re: Spark job hangs up at Evaluating ParMultiDo(ParseInput)
Date Tue, 01 Aug 2017 10:46:36 GMT
Hi JB,

Not when I submit is using the option —master local. But when I submit with with spark master’s
port, The job executes and hangs at step 'Registering block manager 192.168.0.2:58956 with
366.3 MB RAM, BlockManagerId(0, 192.168.0.2, 58956, None)’. Attached screenshots of the
HistoryUI & Dashboard. Also raised a question in Stackoverflow<https://stackoverflow.com/questions/45435238/apache-beam-job-hangs-up-when-submitted-via-spark-submit>
too.

Command:
~/spark/bin/spark-submit --class org.apache.beam.examples.WordCount --master spark://Sathish-MacBook-Pro.local:7077
target/word-count-beam-0.1.jar --runner=SparkRunner --inputFile=pom.xml --output=counts


[cid:cb6c7891-d1a4-40b5-ab98-0d4c870a02ef@CANPRD01.PROD.OUTLOOK.COM]
[cid:868b65ac-15a1-4f91-935b-6bf5e7d4ec43@CANPRD01.PROD.OUTLOOK.COM]

Regards,
Sathish. J

On 01-Aug-2017, at 4:02 PM, Jean-Baptiste Onofré <jb@nanthrax.net<mailto:jb@nanthrax.net>>
wrote:

Hi Sathish,

Do you see the tasks submitted on the history server ?

Regards
JB

On 08/01/2017 11:51 AM, Sathish Jayaraman wrote:
Hi,
I am trying to execute Beam example in local spark setup. When I try to submit the sample
WordCount jar via spark-submit, the job just hangs at 'INFO SparkRunner$Evaluator: Evaluating
ParMultiDo(ExtractWords)’. But it runs fine when executed directly. Below is the command
I used to submit the job in spark local,
$ ~/spark/bin/spark-submit --class "org.apache.beam.examples.WordCount"   --master local[4]
target/word-count-beam-0.1.jar —inputFile=./pom.xml --output=csvout --runner=SparkRunner
Have attached log file for reference. Can anyone please help me find out whats going on?
Regards,
Sathish. J

--
Jean-Baptiste Onofré
jbonofre@apache.org<mailto:jbonofre@apache.org>
http://blog.nanthrax.net
Talend - http://www.talend.com

Mime
View raw message