flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Smaine Kahlouch <smaine.kahlo...@smartjog.com>
Subject Spillable memory perf tests : source stops sending
Date Thu, 05 Mar 2015 10:53:58 GMT
Hello,

I'm currently testing the spillable memory channel.
The idea is to have a fast channel coupled with a file channel in order 
to spool logs when our hadoop cluster is unreachable.

To give you the whole architecture :

source-server -> tcp/syslog -> flume-spool (memory + spooling) -> avro 
-> flume-destination (hadoop-cluster)

My test scenario is :
- send 100 000 logs from the source server
- stop the flume-destination during this sending
- wait a few seconds in order to spool on the flume-spool server
- start the flume-destination
- check if all the logs have been received on the flume-destination server.

The problem is that when i stop the destination server, the source 
server stops sending *completely*.
Is this the normal behaviour ? Should it spool on the flume server 
instead of stopping the source ?

On my flume logs i can see the following exception :
05 Mar 2015 10:22:13,828 INFO 
[SinkRunner-PollingRunner-DefaultSinkProcessor] 
(org.apache.flume.sink.AbstractRpcSink.createConnection:217)  - Rpc sink 
sin1: Building RpcClient with hostname: 10.115.77.6, port: 55555
05 Mar 2015 10:22:13,828 INFO 
[SinkRunner-PollingRunner-DefaultSinkProcessor] 
(org.apache.flume.sink.AvroSink.initializeRpcClient:126)  - Attempting 
to create Avro Rpc client.
05 Mar 2015 10:22:13,862 WARN 
[SinkRunner-PollingRunner-DefaultSinkProcessor] 
(org.apache.flume.api.NettyAvroRpcClient.configure:634)  - Using default 
maxIOWorkers
05 Mar 2015 10:22:13,869 ERROR 
[SinkRunner-PollingRunner-DefaultSinkProcessor] 
(org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver 
event. Exception follows.
java.lang.NullPointerException
     at 
org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.doRollback(SpillableMemoryChannel.java:590)
     at 
org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
     at 
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:394)
     at 
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
     at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
     at java.lang.Thread.run(Thread.java:745)

Please find enclosed the flume configurations (spooling server and 
destination server).

Regards,

-- 
Smaine Kahlouch - Engineer, Research & Engineering
Arkena | T: +33 1 5868 6196
27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
arkena.com


Mime
View raw message