flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: 回复: About file channel yum error
Date Tue, 16 Oct 2012 15:10:24 GMT
When the agent sees an OOM, can you take a thread dump (kill -3 <pid>) and
post it on pastebin? Please be sure to include the heap summary section at
the bottom of the dump.

Brock

On Mon, Oct 15, 2012 at 10:08 PM, Yanzhi.liu <404307305@qq.com> wrote:

>   Hello Brock:
>     My flume's configuration is
> agent_foo.sources = tailsource-1
> agent_foo.channels = fileChannel
> agent_foo.sinks = hdfsSink-1
> agent_foo.sources.tailsource-1.type =
> com.chinacache.cpis.sources.CpisAvroSource
> agent_foo.sources.tailsource-1.channels=fileChannel
> agent_foo.sources.tailsource-1.bind=221.130.18.90
> agent_foo.sources.tailsource-1.port=4545
> #agent_foo.sources.tailsource-1.threads=10
> agent_foo.channels.fileChannel.type = file
> agent_foo.channels.fileChannel.checkpointDir=/home/hadoop/checkpoint
> agent_foo.channels.fileChannel.dataDirs=/data/cache2/data
> agent_foo.channels.fileChannel.maxFileSize=524288000
> #agent_foo.channels.fileChannel.capacity = 10000
> agent_foo.sinks.hdfsSink-1.type = hdfs
> agent_foo.sinks.hdfsSink-1.channel = fileChannel
> agent_foo.sinks.hdfsSink-1.hdfs.path =
> hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{deviceId}/%{hour}
> #agent_foo.sinks.hdfsSink-1.hdfs.path =
> hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{hour}/%{deviceId}
> agent_foo.sinks.hdfsSink-1.hdfs.filePrefix = cpisfc-assess.log
> agent_foo.sinks.hdfsSink-1.hdfs.rollInterval=600
> agent_foo.sinks.hdfsSink-1.hdfs.rollCount=0
> agent_foo.sinks.hdfsSink-1.hdfs.rollSize = 506870912
> agent_foo.sinks.hdfsSink-1.hdfs.fileType=CompressedStream
> agent_foo.sinks.hdfsSink-1.hdfs.codeC=gzip
> agent_foo.sinks.hdfsSink-1.hdfs.writeFormat=Text
> agent_foo.sinks.hdfsSink-1.hdfs.batchSize=1
> agent_foo.sinks.hdfsSink-1.serializer=avro_event
>     And the java configuration was JAVA_OPTS="-Xms100m -Xmx2048m
> -Dcom.sun.management.jmxremote";
> Thank you very much!
> My Name :
> Yanzhi Liu
>  **
>
>
> ------------------ 原始邮件 ------------------
>  *发件人:* "Brock Noland"<brock@cloudera.com>;
> *发送时间:* 2012年10月16日(星期二) 中午11:01
> *收件人:* "user"<user@flume.apache.org>; **
>  *主题:* Re: About file channel yum error
>
> What is your channel capacity and configured agent heap size?
>
> On Monday, October 15, 2012 at 9:19 PM, Yanzhi.liu wrote:
>
>    Hello everyone:
>     I am using the flume 1.3.0.But the flume clusters was working after
> two days and the flume was stoped. And the flume.log was only error about
> the:
> 14 Oct 2012 15:29:17,018 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.HDFSEventSink.process:448)  - process failed
> java.lang.OutOfMemoryError
>         at java.io.RandomAccessFile.readBytes(Native Method)
>         at java.io.RandomAccessFile.read(RandomAccessFile.java:338)
>         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
>         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
>         at
> org.apache.flume.channel.file.LogFile.readDelimitedBuffer(LogFile.java:486)
>         at
> org.apache.flume.channel.file.LogFileV3$RandomReader.doGet(LogFileV3.java:258)
>         at
> org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:298)
>         at org.apache.flume.channel.file.Log.get(Log.java:409)
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:447)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:387)
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:662)
> So I want to how to avoid the error.
> Thanks very much!
> My Name:
> Yanzhi Liu
>
>
> **
>



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Mime
View raw message