flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From static-max <flasha...@googlemail.com>
Subject "Slow ReadProcessor" warnings when using BucketSink
Date Tue, 11 Oct 2016 08:51:47 GMT
Hi,

I have a low throughput job (approx. 1000 messager per Minute), that
consumes from Kafka und writes directly to HDFS. After an hour or so, I get
the following warnings in the Task Manager log:

2016-10-10 01:59:44,635 WARN  org.apache.hadoop.hdfs.DFSClient
                 - Slow ReadProcessor read fields took 30001ms
(threshold=30000ms); ack: seqno: 66 reply: SUCCESS reply: SUCCESS reply:
SUCCESS downstreamAckTimeNanos: 1599276 flag: 0 flag: 0 flag: 0, targets:
[DatanodeInfoWithStorage[Node1, Node2, Node3]]
2016-10-10 02:04:44,635 WARN  org.apache.hadoop.hdfs.DFSClient
                 - Slow ReadProcessor read fields took 30002ms
(threshold=30000ms); ack: seqno: 13 reply: SUCCESS reply: SUCCESS reply:
SUCCESS downstreamAckTimeNanos: 2394027 flag: 0 flag: 0 flag: 0, targets:
[DatanodeInfoWithStorage[Node1, Node2, Node3]]
2016-10-10 02:05:14,635 WARN  org.apache.hadoop.hdfs.DFSClient
                 - Slow ReadProcessor read fields took 30001ms
(threshold=30000ms); ack: seqno: 17 reply: SUCCESS reply: SUCCESS reply:
SUCCESS downstreamAckTimeNanos: 2547467 flag: 0 flag: 0 flag: 0, targets:
[DatanodeInfoWithStorage[Node1, Node2, Node3]]

I have not found any erros or warning at the datanodes or the namenode.
Every other application using HDFS performs fine. I have very little load
and network latency is fine also. I also checked GC, disk I/O.

The files written are very small (only a few MB), so writing the blocks
should be fast.

The threshold is crossed only 1 or 2 ms, this makes me wonder.

Does anyone have an Idea where to look next or how to fix these warnings?

Mime
View raw message