flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anat Rozenzon <a...@viber.com>
Subject Flume is replaying log for hours now
Date Thu, 08 Aug 2013 05:52:51 GMT
Hi,

I'm trying to restart Flume. My setup is:

Acro source => File channel 1 => HDFS sink
                   => File channel 2 => Another HDFS sink
                   => File channel 3 => File sink

But it seem to be doing replayLog for hours now, after seeing this
yesterday, I even tried setting use-fast-replay=true, but it didn't help.

Each file channel capacity is 100000000, is this too high for Flume? I
started on lower number but then it complained that the channel is getting
filled so I made it higher.

My log is repeatedly writing such lines:
08 Aug 2013 01:36:22,856 INFO  [lifecycleSupervisor-1-1]
(org.apache.flume.channel.file.ReplayHandler.replayLog:293)  - Read 3240000
records
08 Aug 2013 01:36:41,324 INFO  [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.ReplayHandler.replayLog:293)  - Read 3350000
records
08 Aug 2013 01:38:35,794 INFO  [lifecycleSupervisor-1-1]
(org.apache.flume.channel.file.ReplayHandler.replayLog:293)  - Read 3250000
records
08 Aug 2013 01:40:48,759 INFO  [lifecycleSupervisor-1-1]
(org.apache.flume.channel.file.ReplayHandler.replayLog:293)  - Read 3260000
records
08 Aug 2013 01:41:01,684 INFO  [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.ReplayHandler.replayLog:293)  - Read 4090000
records
08 Aug 2013 01:41:36,691 INFO  [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.ReplayHandler.replayLog:293)  - Read 4100000
records
08 Aug 2013 01:42:27,528 INFO  [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.ReplayHandler.replayLog:293)  - Read 4110000
records
08 Aug 2013 01:42:57,725 INFO  [lifecycleSupervisor-1-1]
(org.apache.flume.channel.file.ReplayHandler.replayLog:293)  - Read 3270000
records


In attaching jstack output, I wasn't sure what the threads are doing but in
any case many of them seem to be waiting..

Any idea what I can do to make the server start?

Thanks
Anat

Mime
View raw message