flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brent Halsey <mrbr...@gmail.com>
Subject Re: Preserving syslog information
Date Tue, 03 Jul 2012 17:44:00 GMT
It's possible that you've run into FLUME-1277 "Error parsing Syslog
rfc 3164 messages with null values".  Basically, the date is skipped,
and null values (hyphens) are interpreted as a null date.  Potential
fixes are to use FLUME-1277's patch, make sure you don't have hyphens
in your syslog message, or change the date format to rfc 5424 style.

The flume syslog parser doesn't extract syslog tags (program name),
either.  We've just started patching SyslogUtils.java to pull this
out.

 -brent

On Mon, Jul 2, 2012 at 9:54 PM, Brian Hart <bbhart@bbhart.com> wrote:
>
> I'm working on a project where DNS & DHCP log data need to be aggregated
> from 180+ servers spread around the WAN down to one (maybe two) centralized
> servers.  From the central server(s), I'll need to scp them to another
> company periodically throughout the day.  It's not critical for each message
> to reach the central servers, but it'd be really nice if they did.
>
> I have some architecture questions, but my blocker right now is that my
> syslog messages are only coming across to the central server as "<sending
> user>: <log text>" (eg. "hart_b: This is test 1") and I'm losing the other
> syslog info like date, hostname, and facility.
>
> I searching the mailing list and wiki, but I can't figure out how to do this
> in 1.1.0-incubating.  Syslog on my test DHCP server points to the IP for
> 'remote1', and you can see the rest in my conf file (below).  I think I'm
> supposed to use the syslog serializer, but I'm not clear on how to do that.
>
> # CENTRAL NODE
> central.channels.ch1.type = memory
>
> central.sources.avro-source1.channels = ch1
> central.sources.avro-source1.type = avro
> central.sources.avro-source1.bind = 0.0.0.0
> central.sources.avro-source1.port = 41414
>
> central.sinks.fileroll_sink1.channel = ch1
> central.sinks.fileroll_sink1.type = file_roll
> central.sinks.fileroll_sink1.sink.directory = /opt/logs_from_flume/
> central.sinks.fileroll_sink1.sink.rollInterval = 30
>
> central.channels = ch1
> central.sources = avro-source1
> central.sinks = fileroll_sink1
>
> # REMOTE NODE 1 - North America
> remote1.channels.ch1.type = memory
>
> remote1.sources.syslog-source1.channels = ch1
> remote1.sources.syslog-source1.type = syslogudp
> remote1.sources.syslog-source1.host = 0.0.0.0
> remote1.sources.syslog-source1.port = 514
>
> remote1.sinks.avro-sink1.channel = ch1
> remote1.sinks.avro-sink1.type = avro
> remote1.sinks.avro-sink1.hostname = 192.168.1.60
> remote1.sinks.avro-sink1.port = 41414
> remote1.sinks.avro-sink1.batch-size = 100
>
> remote1.channels = ch1
> remote1.sources = syslog-source1
> remote1.sinks = avro-sink1
>
> -=-=-
> Apologies for asking what might be a basic question, but how can I preserve
> the syslog info so that it makes it into the rolling files on Central?
>
> Thanks,
> Brian
>
>

Mime
View raw message