flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Hoang <a...@parcelperform.com>
Subject Akka tcp address on AWS EMR between job/task managers
Date Mon, 24 Jun 2019 05:04:03 GMT
Hi guys,

I want to use some client lib that integrated with akka system
Flink aslo already created akka system job manager and those task manager can use akka by
its akka.tcp addess
My architecture stucture currently is:

EMR (flink 1.8)
1 core node
1 master node
Flink job on Yarn

I have on task manager and one job manager:

On my task manager I got this log:

container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:52,171 INFO  akka.remote.Remoting
                                         - Remoting started; listening on addresses :[akka.tcp://flink@ip-10-0-5-203.ap-southeast-1.compute.internal:46267]
container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:52,183 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils
        - Actor system started at akka.tcp://flink@ip-10-0-5-203.ap-southeast-1.compute.internal:46267

container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:52,284 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner
      - Actor system started at akka.tcp://flink-metrics@ip-10-0-5-203.ap-southeast-1.compute.internal:36753
container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:52,731 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService
             - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor
at akka://flink/user/taskmanager_0 .
container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:52,755 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor
           - Connecting to ResourceManager akka.tcp://flink@ip-10-0-5-203.ap-southeast-1.compute.internal:33547/user/resourcemanager(00000000000000000000000000000000).
container_1560142756526_0001_01_000002/taskmanager.log:2019-06-10 06:32:53,153 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor
           - Successful registration at resource manager akka.tcp://flink@ip-10-0-5-203.ap-southeast-1.compute.internal:33547/user/resourcemanager
under registration id 31415730214d58ecaf8396835ccce613.

On my job manager log:
container_1560142756526_0001_01_000001/jobmanager.out:2019-06-10 06:32:43,473 INFO  org.apache.flink.yarn.YarnResourceManager
                    - Registering job manager 00000000000000000000000000000000@akka.tcp://flink@ip-10-
0-5-203.ap-southeast-1.compute.internal:33547/user/jobmanager_0 for job 3111b27f8e4a873ad4301bbe2f7e7069.
container_1560142756526_0001_01_000001/jobmanager.out:2019-06-10 06:32:43,484 INFO  org.apache.flink.yarn.YarnResourceManager
                    - Registered job manager 00000000000000000000000000000000@akka.tcp://flink@ip-10-0
-5-203.ap-southeast-1.compute.internal:33547/user/jobmanager_0 for job 3111b27f8e4a873ad4301bbe2f7e7069.

I want to access to this actor system for my client lib (like akka http, redisscala…). Those
need to declare akka.tcp address too.
  implicit val akkaSystem = akka.actor.ActorSystem(“akka.tcp”)

How do I find those address dynamically so that I can use them in AWS emr?

Thanks


Mime
View raw message