Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CANC1h_vz3CsTreb73eubZ==K19AQbSQZ3A2AG56jW8NY1k=6YA@mail.gmail.com>
References: 
 <CA+3U0ZkDnDNKhSjDRG=+5NgqEKzo-2qEkw_BvEwez77PB8TMAQ@mail.gmail.com>
	<CAGco--a3PmYnKOB4wkdTKi4d8YW3GQ-t-SxuL=Vwa8e0DW36eA@mail.gmail.com>
	<CA+3U0Z=6A90uAJS2fD4HTouZpQZ3uHjbkVE4QkFQ-ckbm+Qy0A@mail.gmail.com>
	<CANC1h_vz3CsTreb73eubZ==K19AQbSQZ3A2AG56jW8NY1k=6YA@mail.gmail.com>
Date: Thu, 29 Oct 2015 13:44:26 -0400
Message-ID: 
 <CA+3U0ZmO3tCTPJ254JTsWrREMbbDLT74hEA=ctEuY4KQdO4JPA@mail.gmail.com>
Subject: Re: Diagnosing TaskManager disappearance
From: Greg Hogan <code@greghogan.com>
To: dev@flink.apache.org
Content-Type: multipart/alternative; boundary=001a11349b2cb4f925052341dc44

--001a11349b2cb4f925052341dc44
Content-Type: text/plain; charset=UTF-8

I removed the use of numactl but left in starting two TaskManagers and am
still seeing TaskManagers crash.
>From the JobManager log:

17:36:06,412 WARN
akka.remote.ReliableDeliverySupervisor                        - Association
with remote system [akka.tcp://flink@10.0.88.140:45742] has failed, address
is now gated for [5000] ms. Reason is: [Disassociated].
17:36:06,567 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph        - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (370/2322)
(cac9927a8568c2ad79439262a91478af) switched from RUNNING to FAILED
17:36:06,572 INFO
org.apache.flink.runtime.jobmanager.JobManager                - Status of
job 14d946015fd7b35eb801ea6fee5af9e4 (Flink Java Job at Thu Oct 29 17:34:48
UTC 2015) changed to FAILING.
java.lang.Exception: The data preparation for task 'CHAIN GroupReduce
(Compute scores) -> FlatMap (checksum())' , caused an error: Error
obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated
due to an exception: Connection unexpectedly closed by remote task manager
'ip-10-0-88-140/10.0.88.140:58558'. This might indicate that the remote
task manager was lost.
        at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:465)
        at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error obtaining the sorted input:
Thread 'SortMerger Reading Thread' terminated due to an exception:
Connection unexpectedly closed by remote task manager 'ip-10-0-88-140/
10.0.88.140:58558'. This might indicate that the remote task manager was
lost.
        at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:619)
        at
org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1089)
        at
org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94)
        at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:459)
        ... 3 more
Caused by: java.io.IOException: Thread 'SortMerger Reading Thread'
terminated due to an exception: Connection unexpectedly closed by remote
task manager 'ip-10-0-88-140/10.0.88.140:58558'. This might indicate that
the remote task manager was lost.
        at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:800)
Caused by:
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
Connection unexpectedly closed by remote task manager 'ip-10-0-88-140/
10.0.88.140:58558'. This might indicate that the remote task manager was
lost.
        at
org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.channelInactive(PartitionRequestClientHandler.java:119)
        at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
        at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
        at
io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
        at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
        at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
        at
io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:306)
        at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
        at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
        at
io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:828)
        at
io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:621)
        at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
        at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
        at java.lang.Thread.run(Thread.java:745)
17:36:06,587 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph        - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (367/2322)
(d63c681a18b8164bc24936df1ecb159b) switched from RUNNING to FAILED


On Thu, Oct 29, 2015 at 1:00 PM, Stephan Ewen <sewen@apache.org> wrote:

> Hi Greg!
>
> Interesting... When you say the TaskManagers are dropping, are the
> TaskManager processes crashing, or are they loosing connection to the
> JobManager?
>
> Greetings,
> Stephan
>
>
> On Thu, Oct 29, 2015 at 9:56 AM, Greg Hogan <code@greghogan.com> wrote:
>
> > I recently discovered that AWS uses NUMA for its largest nodes. An
> example
> > c4.8xlarge:
> >
> > $ numactl --hardware
> > available: 2 nodes (0-1)
> > node 0 cpus: 0 1 2 3 4 5 6 7 8 18 19 20 21 22 23 24 25 26
> > node 0 size: 29813 MB
> > node 0 free: 24537 MB
> > node 1 cpus: 9 10 11 12 13 14 15 16 17 27 28 29 30 31 32 33 34 35
> > node 1 size: 30574 MB
> > node 1 free: 22757 MB
> > node distances:
> > node   0   1
> >   0:  10  20
> >   1:  20  10
> >
> > I discovered yesterday that Flink performed ~20-30% faster on large
> > datasets by running two NUMA-constrained TaskManagers per node. The
> > JobManager node ran a single TaskManager. Resources were divided in half
> > relative to running a single TaskManager.
> >
> > The changes from the tail of /bin/taskmanager.sh:
> >
> > -"${FLINK_BIN_DIR}"/flink-daemon.sh $STARTSTOP taskmanager "${args[@]}"
> > +numactl --membind=0 --cpunodebind=0 "${FLINK_BIN_DIR}"/flink-daemon.sh
> > $STARTSTOP taskmanager "${args[@]}"
> > +numactl --membind=1 --cpunodebind=1 "${FLINK_BIN_DIR}"/flink-daemon.sh
> > $STARTSTOP taskmanager "${args[@]}"
> >
> > After reverting this change the system is again stable. I had not
> > experienced issues using numactl when running 16 nodes.
> >
> > Greg
> >
>

--001a11349b2cb4f925052341dc44--