flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roy Camp <a2b2...@gmail.com>
Subject DFO Configuration Help
Date Mon, 09 Apr 2012 21:09:21 GMT
This has been driving me crazy for days so thank you ahead of time -- I
appreciate your time!


The setup:
Version*:* 0.9.4-cdh3u3
Multiple agents collect data (host1.corp.com) and send it to a collector (
fc.corp.com)
The collector will store the record in Cassandra using a custom plugin.
The collector will also fan out and send the event to a thrift socket (a
python service which does counts on events)
Agents to Cassandra should be E2E and the fan out to the thrift socket
should be DFO.

The issue:
Killing the python service and bring it back online (with and without
events being sent while it was offline) causes events to go missing.
On occasion when the service is back online, new events will not even get
through.
I have been unable to establish a set of repeatable steps -- it all seems
rather inconsistent.
I understand an event can get lost when discovering the connection is now
dead -- that is fine.


What should my config string by to achieve the desired functionality?


Current Config:
host1.corp.com: thriftSource(12345) | agentE2ESink("fc.corp.com", 35856);
fc.corp.com : collectorSource(35856) | [cassandraBasin("testKeySpace", "
127.0.0.1:9160"), <insistentOpen stubbornAppend
thriftSink("127.0.0.1",30313) ? diskFailover insistentAppend stubbornAppend
insistentOpen thriftSink("127.0.0.1",30313)>];

I have also tried these alternatives:
fc.corp.com : collectorSource(35856) | [cassandraBasin("rltest", "
127.0.0.1:9160"), < lazyOpen stubbornAppend thriftSink("127.0.0.1",30313) ?
diskFailover insistentOpen lazyOpen stubbornAppend
thriftSink("127.0.0.1",30313) >];
fc.corp.com : collectorSource(35856) | [cassandraBasin("rltest", "
127.0.0.1:9160"), < insistentOpen stubbornAppend
thriftSink("127.0.0.1",30313) ? diskFailover insistentOpen stubbornAppend
thriftSink("127.0.0.1",30313) >];
fc.corp.com : collectorSource(35856) | [cassandraBasin("testKeySpace", "
127.0.0.1:9160"), diskFailover insistentAppend stubbornAppend insistentOpen
thriftSink("127.0.0.1",30313)];

Mime
View raw message