flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kaushik Shreekanth <kkau...@outlook.com>
Subject Flume log4j appender not converting data to avro
Date Wed, 08 Apr 2015 22:16:46 GMT
I'm using Flume log4j Appender to send generic records from my application to a flume avro
source. However at the sink, the data is seen as plain text. I'm not sure why it was not converted
to avro format. 
This is my log4j2 flume appender config:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Console name="ConsoleAppender" target="SYSTEM_OUT">
<PatternLayout pattern="%m%n" />
<Flume name="FlumeAppender" compress="false" type="Avro">
<Agent host="localhost" port="41414"/>
<Root level="debug">
<AppenderRef ref="ConsoleAppender" />
<AppenderRef ref="FlumeAppender" />

Dependencies in my application:


This is my flume.conf:
# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory
# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to Connect it to channel ch1.
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind =
agent1.sources.avro-source1.port = 41414

# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.file-roll-sink.type = file_roll
agent1.sinks.file-roll-sink.sink.directory = /Users/ds/flume_events
agent1.sinks.file-roll-sink.sink.rollInterval = 0

# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = file-roll-sink

#chain the different components together
agent1.sinks.file-roll-sink.channel = ch1

agent1.sources.avro-source1.channels = ch1

My java code that logs events:
public static void main(String[] args) throws IOException {
Logger logger = LogManager.getLogger();
URL url = FlumeTest.class.getClassLoader().getResource("user.avsc");
Schema schema = new Schema.Parser().parse(new File(url.getFile()));
for (i = 0; i <= 10; i++) {
GenericRecord user1 = new GenericData.Record(schema);
user1.put("name", "abc" + String.valueOf(i));
user1.put("id", i);


My sink output is seen as:
{"name": "abc0", "id": 0, "favorite_color": null}
{"name": "abc1", "id": 1, "favorite_color": null}
{"name": "abc2", "id": 2, "favorite_color": null}
{"name": "abc3", "id": 3, "favorite_color": null}
{"name": "abc4", "id": 4, "favorite_color": null}
{"name": "abc5", "id": 5, "favorite_color": null}
"name": "abc6", "id": 6, "favorite_color": null}
{"name": "abc7", "id": 7, "favorite_color": null}
{"name": "abc8", "id": 8, "favorite_color": null}
{"name": "abc9", "id": 9, "favorite_color": null}
{"name": "abc10", "id": 10, "favorite_color": null}

As mentioned, the final output is in plain text and not Avro, can someone please let me know
what I'm doing wrong?

View raw message