flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sourabh Mehta <sourabhmehta2...@gmail.com>
Subject Unable to run flink job in dataproc cluster with jobmanager provided
Date Wed, 17 Jun 2020 15:05:23 GMT
Hi Team,

I'm  exploring flink for one of my use case, I'm facing some issues while
running a flink job in cluster mode. Below are the steps I followed to
setup and run job in cluster mode :
1. Setup flink on google cloud dataproc using
https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/flink

2. After setting up the cluster I could see the flink session started and
could see the UI for the same.

3 Submitted job from dataproc master node using below command

sudo HADOOP_CONF_DIR=/etc/hadoop/conf /usr/lib/flink/bin/flink run -m
yarn-cluster -yid application_1592311654771_0001 -class
com.sm.flink.FlinkDriver /usr/lib/flink/lib/flink-1.0.10-sm-SNAPSHOT.jar
hdfs://cluster-flink-poc-m:8020/user/flink/rocksdb/

After running the job I see the job started successfully but created a mini
local cluster and ran in local mode. I don't see any jobs submitted to
JobManger and I also see 0 task managers on UI.

Can someone please help me understand here?, do let me know what input is
required to investigate the same.

Mime
View raw message