flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yogi nerella <ynerella...@gmail.com>
Subject flume to collect log files from various sources
Date Wed, 23 Jan 2013 22:36:52 GMT

I am a new bee for flume, would like to get some feedback on if what I am
doing works correctly.

My application runs on multiple hosts, I want to collect all log files to a
central location.

1.  In My application I will ship all relevant log4jappender jar files, and
configure the agent's host, port information.

2. I will run a simple agent, and configure a source (avro), channel
(memory), and sink (file_roll).

File_roll will write the events recieved via AVRO source and write them to
the corresponding file_roll sink.

3. Do I need one agent for each host, or is there a way I can configure to
write to multiple files, dynamically based on the real source the message
is getting recieved.

4. If I modify the configuration file, will the agent re-read the
information.   (Ex:  I want to add a new host, and collect the log files to
a new directory).

5.  In my simple test, messages sent from my app are getting lost, and I
see the following information in the console.

14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
14:27:29,604  WARN NettyAvroRpcClient:426 - Invalid value for batchSize: 0;
Using default value.
14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
{tcpNoDelay=true, connectTimeoutMillis=20000}
14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, / => /] BOUND: /
log4j:ERROR Flume append() failed.
14:27:29,940  INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST 2013
14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from

Appreciate your help

View raw message