flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: issue running flink in docker
Date Thu, 11 May 2017 08:26:00 GMT
Glad to hear it!

Overlay networks (as most container infras use) are tricky and we need to
add some code to make diagnostics of issues easier in those cases...

Stephan


On Wed, May 10, 2017 at 9:30 PM, David Brelloch <brelloch@gmail.com> wrote:

> Stephan,
>
> Thanks for pointing us in the right direction on the different addresses.
> That was the issue.
>
> David
>
> On Wed, May 10, 2017 at 3:03 PM, Stephan Ewen <sewen@apache.org> wrote:
>
>> Hi!
>>
>> Can it be that some hostname / IP address mapping / etc gets thrown off
>> somewhere in the process?
>>
>> This exception looks like the following happens:
>>
>>   - JobManager gets a message from a TaskManager that a partition is
>> ready, notifies other TaskManagers
>>   - TaskManager gets the update message, connects to the address of the
>> indicated TaskManager
>>   - That taskmanager does not have that partition
>>
>> Is it possible that JobManager / TaskManager see different names /
>> addresses?
>>
>> Also, is that Flink 1.2, DataSet job?
>>
>> Stephan
>>
>>
>>
>> On Wed, May 10, 2017 at 7:05 PM, David Brelloch <brelloch@gmail.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> We are attempting to run flink 1.2 in a distributed dockerized
>>> environment and are running into issues when running jobs in parallel.
>>>
>>> The exception we are getting fairly quickly after start up is:
>>>
>>> org.apache.flink.runtime.io.network.partition.PartitionNotFoundException: Partition
d3d8404aa26bedafd77e88bdfd88375b@84037703da6706cd1017f53fd8b818cd not found.
>>> 	at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.failPartitionRequest(RemoteInputChannel.java:204)
>>> 	at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.retriggerSubpartitionRequest(RemoteInputChannel.java:129)
>>> 	at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.retriggerPartitionRequest(SingleInputGate.java:331)
>>> 	at org.apache.flink.runtime.taskmanager.Task.onPartitionStateUpdate(Task.java:1244)
>>> 	at org.apache.flink.runtime.taskmanager.Task$2.apply(Task.java:1082)
>>> 	at org.apache.flink.runtime.taskmanager.Task$2.apply(Task.java:1077)
>>> 	at org.apache.flink.runtime.concurrent.impl.FlinkFuture$5.onComplete(FlinkFuture.java:259)
>>> 	at akka.dispatch.OnComplete.internal(Future.scala:248)
>>> 	at akka.dispatch.OnComplete.internal(Future.scala:245)
>>> 	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
>>> 	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
>>> 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>>> 	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
>>> 	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
>>> 	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
>>> 	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
>>> 	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>>> 	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
>>> 	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>>> 	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>>> 	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>
>>> This only occurs when running in parallel but I don't have a lot to go
>>> on from the exception. We have configured the following ports:
>>> jobmanager.rpc.port: 6123
>>> taskmanager.rpc.port: 6122
>>> taskmanager.data.port: 6121
>>>
>>> And have mapped the docker ports 6121 and 6122 on the task managers as
>>> well as 6123 on the job manager.
>>>
>>> Does anyone have any suggestions for other places to look or settings to
>>> try?
>>>
>>> Thanks,
>>> David
>>>
>>>
>>>
>>>
>>
>

Mime
View raw message