flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Neal <cwn...@gmail.com>
Subject Re: data saved in hdfs by flume ng is garbled
Date Wed, 12 Sep 2012 14:27:08 GMT
Hi,

The log4J appender doesn't pay any attention to the PatternLayout, at least
up through 1.3.0-SNAPSHOT.  That's why you only see %m and nothing else.

As for the garbled text, your property of this:

agent.sinks.hdfsSink.hdfs.file.Type

should be this:

agent.sinks.hdfsSink.hdfs.fileType

Since you had file.Type, Flume was using the default of SequenceFile.

Hope that helps!
Chris



On Wed, Sep 12, 2012 at 1:58 AM, Hao Jian <haojian@neusoft.com> wrote:

> **
>  Hi,
> I'm using flume ng(1.2.0) to collect logs from log4j and save logs to hdfs
> . There are two problems:
> (1) Flume only collect %m in log4j, but not %d, %p, %t ...
> (2) The log saved in hdfs is garbled, not plain text.
>  My log4j configuration is as fowllows:
>  <appender name="Console" class="org.apache.log4j.ConsoleAppender">
> <layout class="org.apache.log4j.PatternLayout">
> <param name="ConversionPattern"
> value="%d >> %-5p >> %t >> %l >> %m%n"/>
> </layout>
> </appender>
> <appender name="flume"
> class="org.apache.flume.clients.log4jappender.Log4jAppender">
> <param name="Hostname" value="10.4.46.125" />
> <param name="Port" value="44444" />
> <layout class="org.apache.log4j.PatternLayout">
> <param name="ConversionPattern"
> value="%d >> %-5p >> %t >> %l >> %m%n"/>
> </layout>
> </appender>
>  <root>
> <level value="info"/>
> <appender-ref ref="Console"/>
> <appender-ref ref="flume"/>
> </root>
>  The console print is:
>  2012-09-12 11:43:37,391 >> INFO >> main >> Main.main(Main.java:22)
>>
> this is test1
> 2012-09-12 11:43:37,454 >> INFO >> main >> Main.main(Main.java:22)
>> this
> is test2
> 2012-09-12 11:43:37,460 >> INFO >> main >> Main.main(Main.java:22)
>> this
> is test3
> 2012-09-12 11:43:37,465 >> INFO >> main >> Main.main(Main.java:22)
>> this
> is test4
> 2012-09-12 11:43:37,470 >> INFO >> main >> Main.main(Main.java:22)
>> this
> is test5
> 2012-09-12 11:43:37,475 >> INFO >> main >> Main.main(Main.java:22)
>> this
> is test6
> 2012-09-12 11:43:37,480 >> INFO >> main >> Main.main(Main.java:22)
>> this
> is test7
> 2012-09-12 11:43:37,485 >> INFO >> main >> Main.main(Main.java:22)
>> this
> is test8
> 2012-09-12 11:43:37,492 >> INFO >> main >> Main.main(Main.java:22)
>> this
> is test9
> 2012-09-12 11:43:37,497 >> INFO >> main >> Main.main(Main.java:22)
>> this
> is test10
>  The flume configuration is:
>  agent.channels = jdbcChannel memChannel
> agent.sinks = hdfsSink fileSink
>  agent.channels.jdbcChannel.type = jdbc
> agent.channels.memChannel.type = memory
>  agent.sources.avroSrc.type = avro
>  agent.sources.avroSrc.bind = 0.0.0.0
> agent.sources.avroSrc.port = 40000
> agent.sources.avroSrc.channels = jdbcChannel memChannel
>  agent.sinks.hdfsSink.channel = jdbcChannel
> agent.sinks.hdfsSink.type = hdfs
>  agent.sinks.hdfsSink.hdfs.path = hdfs://10.4.44.134/flume/events/
> agent.sinks.hdfsSink.hdfs.rollCount = 0
> agent.sinks.hdfsSink.hdfs.writeFormat = Writable
> agent.sinks.hdfsSink.hdfs.file.Type = DataStream
>  agent.sinks.fileSink.channel = memChannel
> agent.sinks.fileSink.type = FILE_ROLL
> agent.sinks.fileSink.sink.directory = /var/log/flume/
> agent.sinks.fileSink.sink.rollInterval = 60
>  The log saved in /var/log/flume is:
>  this is test1
> this is test2
> this is test3
> this is test4
> this is test5
> this is test6
> this is test7
> this is test8
> this is test9
> this is test10
>  The log save in hdfs is:
>  SEQ
> !org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable�������
> �
> �D� ���# �cQ��� ��� �� 9���K���
> this is test1��� ��� �� 9���m���
> this is test2��� ��� �� 9������
> this is test3��� ��� �� 9������
> this is test4��� ��� �� 9������
> this is test5��� ��� �� 9�������
> this is test6��� ��� �� 9�������
> this is test7��� ��� �� 9�������
> this is test8��� ��� �� 9�������
> this is test9��� ��� �� 9������� this is test10
>  or:
>
> SEQ!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd
>
> \ufffdD\ufffd\ufffd\ufffd\ufffd#\ufffdcQ\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffdK\ufffd\ufffd\ufffd
> this is
> test1\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffdm\ufffd\ufffd\ufffd
> this is
> test2\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd
> this is
> test3\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd
> this is
> test4\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd
> this is
> test5\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd
> this is
> test6\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd
> this is
> test7\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd
> this is
> test8\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd
> this is
> test9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffdthis
> is test10
>  I've tried to set "agent.sinks.hdfsSink.hdfs.writeFormat" and
> "agent.sinks.hdfsSink.hdfs.file.Type" to other value, but the data in hdfs
> is still garbled.
>
> 2012-09-12
> ------------------------------
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>

Mime
View raw message