flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HEEJIN SOHN <heejin.s...@samsung.com>
Subject Re: Re: Re: java.lang.OutOfMemoryError: Direct buffer memory
Date Tue, 03 Mar 2015 00:23:44 GMT
<HTML><HEAD><TITLE>Samsung Enterprise Portal mySingle</TITLE>
<META content=IE=5 http-equiv=X-UA-Compatible>
<META content="text/html; charset=utf-8" http-equiv=Content-Type>
<STYLE id=mysingle_style>P {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN-TOP: 5px
}
TD {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN-TOP: 5px
}
LI {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN-TOP: 5px
}
BODY {
	FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN: 10px; LINE-HEIGHT: 1.4
}
</STYLE>

<META name=GENERATOR content=ActiveSquare></HEAD>
<BODY>
<META name=GENERATOR content=ActiveSquare>
<P>Cloudera manager provides a resident memory usage graph which just increases monotonically.
It seems that deallocation occurred from time to time, but the amount is negligible considering
the allocation rate.</P>
<P>Direct buffer size was set up to 10g and this tendency was independent of buffer
size.</P>
<P>Also, the thrift client&nbsp;which sends flume events stopped last night and
continued to work after 10 hours. During that time, the memory usage of the flume agent has
never changed (Garbage collection has never happened).</P>
<P>Anyway, it seems that the problem is caused by memory deallocation(or garbage collection)
which never or barely&nbsp;occurred during the operation of file channel (or channel processor).</P>
<P>I doubted HW problem. However, this test was performed on other server and the result
was the same. </P>
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>Thanks for your comments.</P>
<P>Heejin Sohn</P>
<P>&nbsp;</P>
<P>------- <B>Original Message</B> -------</P>
<P><B>Sender</B> : Ashish&lt;paliwalashish@gmail.com&gt;</P>
<P><B>Date</B> : 2015-03-02 13:53 (GMT+09:00)</P>
<P><B>Title</B> : Re: Re: java.lang.OutOfMemoryError: Direct buffer memory</P>
<P>&nbsp;</P>
<DIV dir=ltr>Can you monitor how much off-heap/direct buffers are allocated? Try to
up the&nbsp;<SPAN style="FONT-SIZE: 12px">-XX:MaxDirectMemorySize=2g to a higher
value and see if the Agents holds good. If the allocation rate is higher than deallocation
rate, you would still end up with OOME.&nbsp;</SPAN> 
<DIV><SPAN style="FONT-SIZE: 12px"><BR></SPAN></DIV>
<DIV><SPAN style="FONT-SIZE: 12px">Let's try few setting and monitor how things
go.<BR></SPAN>
<DIV><SPAN style="FONT-SIZE: 12px"><BR></SPAN></DIV>
<DIV><SPAN style="FONT-SIZE: 12px"><BR></SPAN>
<DIV><BR></DIV>
<DIV><BR></DIV></DIV></DIV></DIV>
<DIV class=gmail_extra><BR>
<DIV class=gmail_quote>On Mon, Mar 2, 2015 at 9:41 AM, HEEJIN SOHN <SPAN dir=ltr>&lt;<A
href="mailto:heejin.sohn@samsung.com" target=_blank>heejin.sohn@samsung.com</A>&gt;</SPAN>
wrote:<BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT:
#ccc 1px solid">
<DIV>
<P>Wan Yi //</P>
<P>Tried as you've suggested (maxFileSize = 1M), but only met the same exception.</P>
<P>&nbsp;</P>
<P>Hari Shreedharan//</P>
<P>All flume options are set using Cloudera manager. (CDH-5.2.0-1)</P>
<P>Also, possible event size is under 100MB.</P>
<P>&nbsp;</P>
<P>Here is the log pattern.</P>
<P>8:15:26.367 PM &nbsp;INFO &nbsp;org.apache.flume.sink.hdfs.BucketWriter Creating
/user/25068525641.jpg.27.1425035726356.tmp<BR>8:15:36.407 PM &nbsp;INFO &nbsp;org.apache.flume.channel.file.Log
Roll start /data/2/flumechannel/data<BR>8:15:36.407 PM &nbsp;INFO &nbsp;org.apache.flume.channel.file.LogFile
Opened /data/2/flumechannel/data/log-4<BR>8:15:36.447 PM &nbsp;INFO &nbsp;org.apache.flume.channel.file.LogFile
Closing /data/2/flumechannel/data/log-3<BR>8:15:36.447 PM &nbsp;INFO &nbsp;org.apache.flume.channel.file.Log
Roll end</P>
<DIV>
<DIV class=h5><BR>8:15:36.450 PM &nbsp;ERROR &nbsp;org.apache.flume.channel.ChannelProcessor
Error while writing to required channel: FileChannel c1 { dataDirs: [/data/2/flumechannel/data]
}<BR>java.lang.OutOfMemoryError: Direct buffer memory<BR>&nbsp;at java.nio.Bits.reserveMemory(Bits.java:658)<BR>&nbsp;at
java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:123)<BR>&nbsp;at
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)<BR>&nbsp;at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)<BR>&nbsp;at
sun.nio.ch.IOUtil.write(IOUtil.java:58)<BR>&nbsp;at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)<BR>&nbsp;at
org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:313)<BR>&nbsp;at
org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:267)<BR>&nbsp;at org.apache.flume.channel.file.Log.put(Log.java:633)<BR>&nbsp;at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)<BR>&nbsp;at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)<BR>&nbsp;at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)<BR>&nbsp;at
org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)<BR>&nbsp;at
org.apache.flume.source.ThriftSource$ThriftSourceHandler.append(ThriftSource.java:253)<BR>&nbsp;at
org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:251)<BR>&nbsp;at
org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:236)<BR>&nbsp;at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)<BR>&nbsp;at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)<BR>&nbsp;at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)<BR>&nbsp;at
org.apache.thrift.server.Invocation.run(Invocation.java:18)<BR>&nbsp;at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<BR>&nbsp;at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<BR>&nbsp;at
java.lang.Thread.run(Thread.java:745)</DIV></DIV>
<P></P>
<P><BR>8:15:36.451 PM &nbsp;ERROR &nbsp;org.apache.thrift.ProcessFunction
Internal error processing append java.lang.OutOfMemoryError: Direct buffer memory<SPAN><BR>&nbsp;at
java.nio.Bits.reserveMemory(Bits.java:658)<BR>&nbsp;at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:123)<BR>&nbsp;at
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)<BR>&nbsp;at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)<BR>&nbsp;at
sun.nio.ch.IOUtil.write(IOUtil.java:58)<BR>&nbsp;at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)<BR>&nbsp;at
org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:313)<BR>&nbsp;at
org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:267)<BR>&nbsp;at org.apache.flume.channel.file.Log.put(Log.java:633)<BR>&nbsp;at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)<BR>&nbsp;at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)<BR>&nbsp;at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)<BR>&nbsp;at
org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)<BR>&nbsp;at
org.apache.flume.source.ThriftSource$ThriftSourceHandler.append(ThriftSource.java:253)<BR>&nbsp;at
org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:251)<BR>&nbsp;at
org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:236)<BR>&nbsp;at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)<BR>&nbsp;at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)<BR>&nbsp;at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)<BR>&nbsp;at
org.apache.thrift.server.Invocation.run(Invocation.java:18)<BR>&nbsp;at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<BR>&nbsp;at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<BR>&nbsp;at
java.lang.Thread.run(Thread.java:745)<BR></SPAN></P>
<P>&nbsp;</P>
<P>Thank you very much.</P>
<P>&nbsp;</P>
<P>Heejin Sohn.</P>
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>------- <B>Original Message</B> -------</P>
<P><B>Sender</B> : Hari Shreedharan&lt;<A href="mailto:hshreedharan@cloudera.com"
target=_blank>hshreedharan@cloudera.com</A>&gt;</P>
<P><B>Date</B> : 2015-03-02 12:22 (GMT+09:00)</P>
<P><B>Title</B> : Re: java.lang.OutOfMemoryError: Direct buffer memory</P>
<DIV>
<DIV class=h5>
<P>&nbsp;</P>How are you passing the JVM options in. Is the event being written
more than 2g?&nbsp;<SPAN></SPAN><BR><BR>On Sunday, March 1, 2015,
Wan Yi(武汉_技术部_搜索与精准化_万毅) &lt;<A href="mailto:wanyi@yhd.com"
target=_blank>wanyi@yhd.com</A>&gt; wrote:<BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT:
#ccc 1px solid"><U></U><U></U>
<DIV lang=ZH-CN style="MARGIN: 7.5pt" link="blue" vlink="purple">
<DIV>
<P class=MsoNormal><SPAN lang=EN-US>tier1.channels.c1.maxFileSize = 500000000</SPAN><SPAN
lang=EN-US><U></U><U></U></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P class=MsoNormal><SPAN lang=EN-US style="FONT-SIZE: 10.5pt; FONT-FAMILY: 'Calibri','sans-serif';
COLOR: #1f497d">try to change the maxFileSize to a small value, may be 1M or 10M<U></U><U></U></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US style="FONT-SIZE: 10.5pt; FONT-FAMILY: 'Calibri','sans-serif';
COLOR: #1f497d"><U></U><U></U></SPAN>&nbsp;</P>
<DIV>
<P class=MsoNormal style="TEXT-ALIGN: justify; MARGIN: 0cm 0cm 0pt; TEXT-JUSTIFY: inter-ideograph"><B><SPAN
lang=EN-US style="FONT-SIZE: 10.5pt; FONT-FAMILY: 'Verdana','sans-serif'; COLOR: #1f497d"><U></U><U></U></SPAN></B>&nbsp;</P></DIV>
<P class=MsoNormal><SPAN lang=EN-US style="FONT-SIZE: 10.5pt; FONT-FAMILY: 'Calibri','sans-serif';
COLOR: #1f497d">Wayne Wan<U></U><U></U></SPAN></P>
<DIV>
<DIV style="BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium
none; PADDING-BOTTOM: 0cm; PADDING-TOP: 3pt; PADDING-LEFT: 0cm; BORDER-LEFT: medium none;
PADDING-RIGHT: 0cm">
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><B><SPAN style="FONT-SIZE: 10pt;
FONT-FAMILY: 宋体">发件人<SPAN lang=EN-US>:</SPAN></SPAN></B><SPAN
lang=EN-US style="FONT-SIZE: 10pt; FONT-FAMILY: 宋体"> </SPAN><SPAN style="FONT-SIZE:
10pt; FONT-FAMILY: 'Batang','serif'">손희진</SPAN><SPAN lang=EN-US style="FONT-SIZE:
10pt; FONT-FAMILY: 宋体"> [mailto:<A>heejin.sohn@samsung.com</A>] <BR></SPAN><B><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 宋体">发送时间<SPAN lang=EN-US>:</SPAN></SPAN></B><SPAN
lang=EN-US style="FONT-SIZE: 10pt; FONT-FAMILY: 宋体"> 2015</SPAN><SPAN style="FONT-SIZE:
10pt; FONT-FAMILY: 宋体">年<SPAN lang=EN-US>3</SPAN>月<SPAN lang=EN-US>2</SPAN>日<SPAN
lang=EN-US> 8:28<BR></SPAN><B>收件人<SPAN lang=EN-US>:</SPAN></B><SPAN
lang=EN-US> <A>user@flume.apache.org</A><BR></SPAN><B>主题<SPAN
lang=EN-US>:</SPAN></B><SPAN lang=EN-US> java.lang.OutOfMemoryError:
Direct buffer memory<U></U><U></U></SPAN></SPAN></P></DIV></DIV>
<P class=MsoNormal><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>Hi there,<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>I'm testing flume with thrift source, file channel, and HDFS
sink. <U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>Also there is a flume client which sends events using thrift,
and the size of each event is up to 30 Mbytes.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>It works fine for a short period, but after a few minutes,
following error occurs at org.apache.flume.channel.ChannelProcessor.<U></U><U></U></SPAN></P>
<TABLE cellPadding=0 border=0>
<TBODY>
<TR>
<TD style="PADDING-BOTTOM: 0.75pt; PADDING-TOP: 0.75pt; PADDING-LEFT: 0.75pt; PADDING-RIGHT:
0.75pt">
<P class=MsoNormal style="MARGIN-BOTTOM: 3.75pt; MARGIN-LEFT: 0cm; MARGIN-RIGHT: 0cm"><SPAN
lang=EN-US style="FONT-SIZE: 9pt; FONT-FAMILY: GulimChe">8:15:36.450 PM<U></U><U></U></SPAN></P></TD>
<TD style="PADDING-BOTTOM: 0.75pt; PADDING-TOP: 0.75pt; PADDING-LEFT: 0.75pt; PADDING-RIGHT:
0.75pt">
<P class=MsoNormal style="MARGIN-BOTTOM: 3.75pt; MARGIN-LEFT: 0cm; MARGIN-RIGHT: 0cm"><SPAN
lang=EN-US style="FONT-SIZE: 9pt; FONT-FAMILY: GulimChe">ERROR<U></U><U></U></SPAN></P></TD>
<TD style="PADDING-BOTTOM: 0.75pt; PADDING-TOP: 0.75pt; PADDING-LEFT: 0.75pt; PADDING-RIGHT:
0.75pt">
<P class=MsoNormal style="MARGIN-BOTTOM: 3.75pt; MARGIN-LEFT: 0cm; MARGIN-RIGHT: 0cm"><SPAN
lang=EN-US style="FONT-SIZE: 9pt; FONT-FAMILY: GulimChe">org.apache.flume.channel.ChannelProcessor<U></U><U></U></SPAN></P></TD>
<TD style="PADDING-BOTTOM: 0.75pt; PADDING-TOP: 0.75pt; PADDING-LEFT: 0.75pt; PADDING-RIGHT:
0.75pt"><PRE><SPAN lang=EN-US>Error while writing to required channel: FileChannel
c1 { dataDirs: [/data/2/flumechannel/data] }<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>java.lang.OutOfMemoryError: Direct buffer memory<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.nio.Bits.reserveMemory(Bits.java:658)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:123)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.IOUtil.write(IOUtil.java:58)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:313)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:267)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.file.Log.put(Log.java:633)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.source.ThriftSource$ThriftSourceHandler.append(ThriftSource.java:253)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:251)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:236)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.thrift.server.Invocation.run(Invocation.java:18)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.lang.Thread.run(Thread.java:745)<U></U><U></U></SPAN></PRE></TD></TR></TBODY></TABLE>
<P><SPAN lang=EN-US>I increased direct memory size upto 2G byte, but it didn't
work.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>Here's my flume configuration.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>#source<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>tier1.sources.s1.type&nbsp;&nbsp;&nbsp;&nbsp;
= thrift<BR>tier1.sources.s1.bind&nbsp;&nbsp;&nbsp;&nbsp; = 0.0.0.0<BR>tier1.sources.s1.port&nbsp;&nbsp;&nbsp;&nbsp;
= 30010<BR>tier1.sources.s1.channels = c0 c1 memdefault<BR>tier1.sources.s1.selector.type
= multiplexing<BR>tier1.sources.s1.selector.header = category<BR>tier1.sources.s1.selector.mapping.Log4j
=&nbsp;c0 <BR>tier1.sources.s1.selector.mapping.Data = c1<BR>tier1.sources.s1.selector.default
= memDefault<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>#channel<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>tier1.channels.c1.type&nbsp;&nbsp; = memory<BR>tier1.channels.c1.checkpointDir=/data/2/flumechannel/checkpoint<BR>tier1.channels.c1.dataDirs=/data/2/flumechannel/data<BR>tier1.channels.c1.transactionCapacity
= 1<BR>tier1.channels.c1.maxFileSize = 500000000<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>#sink<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>tier1.sinks.k1.type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
= hdfs<BR>tier1.sinks.k1.channel&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
c1<BR>tier1.sinks.k1.hdfs.path = /user/soul<BR>tier1.sinks.k1.hdfs.round = false<BR>tier1.sinks.k1.hdfs.fileType
= DataStream<BR>tier1.sinks.k1.hdfs.rollCount = 1<BR>tier1.sinks.k1.hdfs.batchSize
= 1<BR>tier1.sinks.k1.hdfs.retryInterval = 10<BR>tier1.sinks.k1.hdfs.proxyUser
= soul<BR>tier1.sinks.k1.hdfs.maxOpenFiles = 10<BR>tier1.sinks.k1.hdfs.idleTimeOut
= 900<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>and java config. option: -Xmx2g -XX:MaxDirectMemorySize=2g
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC <U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>When I use memory channel instead of file channel, it works
great.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>I can't understand this phenomena.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>The only clue that I have is that the exception&nbsp;always
occurs after "org.apache.flume.channel.file.Log: Roll end"<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>Hope your comments.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>Thank you.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN lang=EN-US><U></U><U></U></SPAN></P></DIV></DIV></BLOCKQUOTE>
<P>&nbsp;</P></DIV></DIV>
<P>&nbsp;</P></DIV><IMG border=0 src="http://ext.samsung.net/mailcheck/SeenTimeChecker?do=61dd5ae2e408a7162486384e88cb346bf1952ec14b898c73279babd5c747fd394603607fb4f10f2b93a068f28056b538f4bcdeced46ed5ee08cece8541bc14eacf878f9a26ce15a0"
width=0 height=0></BLOCKQUOTE></DIV><BR><BR clear=all>
<DIV><BR></DIV>-- <BR>
<DIV class=gmail_signature>thanks<BR>ashish<BR><BR>Blog: <A href="http://www.ashishpaliwal.com/blog">http://www.ashishpaliwal.com/blog</A><BR>My
Photo Galleries: <A href="http://www.pbase.com/ashishpaliwal">http://www.pbase.com/ashishpaliwal</A></DIV></DIV>
<P>&nbsp;</P><!--SP:heejin.sohn--><!--heejin.sohn:EP-->
<P>&nbsp;</P></BODY></HTML><img src='http://ext.samsung.net/mailcheck/SeenTimeChecker?do=61dd5ae2e408a7166c5e03b278b8424d9195c6080acae71b279babd5c747fd394603607fb4f10f2b93a068f28056b538f4bcdeced46ed5ee08cece8541bc14eacf878f9a26ce15a0'
border=0 width=0 height=0 style='display:none'>
Mime
View raw message