Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5400EDCB0 for ; Fri, 25 Jan 2013 08:38:38 +0000 (UTC) Received: (qmail 80756 invoked by uid 500); 25 Jan 2013 08:38:36 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 79000 invoked by uid 500); 25 Jan 2013 08:38:34 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 78930 invoked by uid 99); 25 Jan 2013 08:38:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jan 2013 08:38:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ablozhou@gmail.com designates 209.85.217.173 as permitted sender) Received: from [209.85.217.173] (HELO mail-lb0-f173.google.com) (209.85.217.173) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jan 2013 08:38:25 +0000 Received: by mail-lb0-f173.google.com with SMTP id gf7so278222lbb.32 for ; Fri, 25 Jan 2013 00:38:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=NOGmtJy/SPwUiWTLPywMNtrY+7kUl0fyNYnAVZYR3s0=; b=zHU34iVXxjMzu9D2yK/13xN9shvw451uFZiY0ccjQ4SKOGbVgNP12QmT2GKss1UBe0 6Hm80NyaGmrrkwIYhgBpy/CwgyQGqEusZ+8UPQ4aLhHfvNqu8tL4R3+UCHeh6BvyS9Ke b0mWviky1f+dY3iAec73F311pvU/kOIjsOZH75rN4YhsIMjqyloq6mPEbLhLBemvxET2 P1SrooQVj474PFg8aLGcOF5roatlCruZJDV7IkblaHcBb4YFJXIDQE2uI6y8h4Y/1LiY zK2pBiRsDMV/hQb+P/4aP/BqonzVhs2GwrSYiKm9Xli3Xa93EsUh50q/InyMfdLukUG0 YiNQ== X-Received: by 10.152.114.42 with SMTP id jd10mr4407841lab.31.1359102912688; Fri, 25 Jan 2013 00:35:12 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.30.231 with HTTP; Fri, 25 Jan 2013 00:34:52 -0800 (PST) In-Reply-To: References: From: =?UTF-8?B?5ZGo5qKm5oOz?= Date: Fri, 25 Jan 2013 16:34:52 +0800 Message-ID: Subject: Re: flume 0.9.4 windows version out of memory To: user@flume.apache.org Content-Type: multipart/alternative; boundary=f46d040839d152f4e804d418ce2f X-Virus-Checked: Checked by ClamAV on apache.org --f46d040839d152f4e804d418ce2f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I found that if I change windows agent configure fromtail("c:\\test.txt") agentSink("hadoop48",35853) ;to tail("c:\\test.txt")agentDFOSink("hadoop48",35853) ; then the agent changes status from OPENING to ACTIVE. my windows agent is :192.168.20.81, while linux collector is : 192.168.10.4= 8 another linux agent is 192.168.10.46. 10.46 have no problem of agentSink. Thanks! Andy. 2013/1/25 =E5=91=A8=E6=A2=A6=E6=83=B3 > when i start the flume node service,the agent is OPENING. > Hello, > > I need to gather logs from windows 7, but the flume master and collector > is running on linux. I download 0.9.4 exe file from flume official web. > > When I start the node from windows, the status is OPENING. > > zhouhh.TKOffice.localzhouhh.TKOffice.local zhouhh.TKOffice.localOPENINGFr= i > Jan 25 11:45:01 CST 2013 4 > > And the log reports OutOfMemoryError. > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stdout.2013-01-25.log > > 2013-01-25 15:31:03 Commons Daemon procrun stdout initializedListening fo= r > transport dt_socket at address: 8888 > java.lang.OutOfMemoryError: Java heap space > Dumping heap to java_pid8100.hprof ... > Heap dump file created [13270531 bytes in 0.115 secs] > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stderr.2013-01-25.log > > 2013-01-25 15:31:03 Commons Daemon procrun stderr initialized > 13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4 > 13/01/25 15:31:03 INFO agent.FlumeNode: rev > 8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a > 13/01/25 15:31:03 INFO agent.FlumeNode: Compiled on Mon Jun 20 14:46:53 > PDT 2011 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.runtime.name= =3DJava(TM) > SE Runtime Environment > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.boot.library.path=3DC:\PROGRA~1\Java\jre7\bin > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vm.version=3D23.3-b01 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vm.vendor=3DOracle Corporation > 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vendor.url= =3D > http://java.oracle.com/ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property path.separator=3D= ; > 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vm.name=3DJa= va > HotSpot(TM) Client VM > 13/01/25 15:31:03 INFO agent.FlumeNode: System property file.encoding.pkg= =3D > sun.io > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.country=3DCN > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.script=3D > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.os.patch.level=3DService Pack 1 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vm.specification.name=3DJava Virtual Machine Specification > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > user.dir=3DC:\Windows\system32 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.runtime.version=3D1.7.0_07-b11 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property flume=3D > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.awt.graphicsenv=3Dsun.awt.Win32GraphicsEnvironment > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.endorsed.dirs=3DC:\PROGRA~1\Java\jre7\lib\endorsed > 13/01/25 15:31:03 INFO agent.FlumeNode: System property os.arch=3Dx86 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.io.tmpdir=3DC:\Windows\TEMP\ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property line.separator=3D > > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vm.specification.vendor=3DOracle Corporation > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.variant=3D > 13/01/25 15:31:03 INFO agent.FlumeNode: System property os.name=3DWindows= 7 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > flume.home=3Dd:\PROGRA~1\Cloudera\FLUME0~1.4\bin\.. > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.jnu.encoding=3DGBK > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.library.path=3Dd:\PROGRA~1\Cloudera\FLUME0~1.4\bin;C:\Windows\Sun\Ja= va\bin;C:\Windows\system32;C:\Windows;C:\Program > Files\Common > Files\NetSarang;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\W= indows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\= ;C:\Program > Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL > Server\100\DTS\Binn\;C:\Program Files\Common Files\Thunder > Network\KanKan\Codecs;d:\php;d:\php\ext;D:\Program Files\MySQL\MySQL Serv= er > 5.5\bin;D:\mysql\bin;D:\wxWidgets-2.9.1\include;D:\wxWidgets-2.9.1\includ= e\msvc;W:\Server\include;W:\Base\include;W:\Client\include;d:\Program > Files\Lua\5.1;d:\Program Files\Lua\5.1\clibs;D:\Program > Files\TortoiseSVN\bin;d:\Program Files\Git\cmd;d:\Program > Files\Git\bin;D:\Program Files\TortoiseGit\bin;D:\docbook\bin;;. > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.specification.name=3DJava Platform API Specification > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.class.version=3D51.0 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.management.compiler=3DHotSpot Client Compiler > 13/01/25 15:31:03 INFO agent.FlumeNode: System property os.version=3D6.1 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.home=3DC:\ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > user.timezone=3DAsia/Shanghai > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.awt.printerjob=3Dsun.awt.windows.WPrinterJob > 13/01/25 15:31:03 INFO agent.FlumeNode: System property file.encoding=3DG= BK > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.specification.version=3D1.7 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.class.path=3Dd:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\build\classes;d:= \PROGRA~1\Cloudera\FLUME0~1.4\lib\activation-1.1.jar;d:\PROGRA~1\Cloudera\F= LUME0~1.4\lib\ahocorasick-2.x.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\ant-1= .6.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-2.7.7.jar;d:\PROGRA~1\Cl= oudera\FLUME0~1.4\lib\antlr-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\ant= lr-runtime-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\asm-3.1.jar;d:\PROGR= A~1\Cloudera\FLUME0~1.4\lib\automaton-1.0.0.jar;d:\PROGRA~1\Cloudera\FLUME0= ~1.4\lib\avro-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-compiler-1= .5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-ipc-1.5.1.jar;d:\PROGRA~1= \Cloudera\FLUME0~1.4\lib\commons-cli-1.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.= 4\lib\commons-codec-1.3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-col= lections-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-el-1.0.jar;d:\= PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-httpclient-3.1.jar;d:\PROGRA~1\Clo= udera\FLUME0~1.4\lib\commons-lang-2.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\l= ib\commons-logging-1.0.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-ne= t-1.4.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\core-3.1.1.jar;d:\PROGRA~1\= Cloudera\FLUME0~1.4\lib\flume-core-0.9.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.= 4\lib\guava-r07.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\hadoop-core-0.20.2-= CDH3B4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\hsqldb-1.8.0.7.jar;d:\PROGRA= ~1\Cloudera\FLUME0~1.4\lib\irclib-1.0.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4= \lib\jackson-core-asl-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson= -jaxrs-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-mapper-asl-1.5= .5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-xc-1.5.5.jar;d:\PROGRA~1= \Cloudera\FLUME0~1.4\lib\jasper-compiler-5.5.23.jar;d:\PROGRA~1\Cloudera\FL= UME0~1.4\lib\jasper-runtime-5.5.23.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\= jaxb-api-2.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jaxb-impl-2.1.12.jar;d= :\PROGRA~1\Cloudera\FLUME0~1.4\lib\jersey-core-1.4.jar;d:\PROGRA~1\Cloudera= \FLUME0~1.4\lib\jersey-json-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jer= sey-server-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jets3t-0.6.1.jar;d:\= PROGRA~1\Cloudera\FLUME0~1.4\lib\jettison-1.1.jar;d:\PROGRA~1\Cloudera\FLUM= E0~1.4\lib\jetty-6.1.26.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jetty-util-= 6.1.26.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jline-0.9.94.jar;d:\PROGRA~1= \Cloudera\FLUME0~1.4\lib\jms-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\js= p-api-2.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jsp-api-2.1.jar;d:\PROGRA= ~1\Cloudera\FLUME0~1.4\lib\libthrift-0.6.0-cdh.jar;d:\PROGRA~1\Cloudera\FLU= ME0~1.4\lib\log4j-1.2.15.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\mail-1.4.j= ar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\netty-3.2.4.Final.jar;d:\PROGRA~1\Cl= oudera\FLUME0~1.4\lib\oro-2.0.8.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\par= anamer-2.3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\servlet-api-2.5-20081211= .jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\servlet-api-2.5.jar;d:\PROGRA~1\Cl= oudera\FLUME0~1.4\lib\slf4j-api-1.5.11.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\= lib\slf4j-log4j12-1.5.11.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\snappy-jav= a-1.0.1-rc3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stax-api-1.0-2.jar;d:\P= ROGRA~1\Cloudera\FLUME0~1.4\lib\stax-api-1.0.1.jar;d:\PROGRA~1\Cloudera\FLU= ME0~1.4\lib\stringtemplate-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\velo= city-1.6.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\xmlenc-0.52.jar;d:\PROGR= A~1\Cloudera\FLUME0~1.4\lib\zookeeper-3.3.1-10.jar;d:\PROGRA~1\Cloudera\FLU= ME0~1.4\bin\..\conf > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.name=3DZHOUH= H$ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > flume.root.logger=3DINFO,console > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vm.specification.version=3D1.7 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.home=3DC:\PROGRA~1\Java\jre7 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.arch.data.model=3D32 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.language=3Dz= h > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.specification.vendor=3DOracle Corporation > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > awt.toolkit=3Dsun.awt.windows.WToolkit > 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vm.info=3Dmi= xed > mode > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.version=3D1.7.0_07 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.ext.dirs=3DC:\PROGRA~1\Java\jre7\lib\ext;C:\Windows\Sun\Java\lib\ext > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.boot.class.path=3DC:\PROGRA~1\Java\jre7\lib\alt-rt.jar;C:\PROGRA~1\Ja= va\jre7\lib\resources.jar;C:\PROGRA~1\Java\jre7\lib\rt.jar;C:\PROGRA~1\Java= \jre7\lib\sunrsasign.jar;C:\PROGRA~1\Java\jre7\lib\jsse.jar;C:\PROGRA~1\Jav= a\jre7\lib\jce.jar;C:\PROGRA~1\Java\jre7\lib\charsets.jar;C:\PROGRA~1\Java\= jre7\lib\jfr.jar;C:\PROGRA~1\Java\jre7\classes > 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vendor=3DOra= cle > Corporation > 13/01/25 15:31:03 INFO agent.FlumeNode: System property file.separator=3D= \ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vendor.url.bug=3Dhttp://bugreport.sun.com/bugreport/ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.io.unicode.encoding=3DUnicodeLittle > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.cpu.endian=3Dlittle > 13/01/25 15:31:03 INFO agent.FlumeNode: System property sun.desktop=3Dwin= dows > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.cpu.isalist=3Dpentium_pro+mmx pentium_pro pentium+mmx pentium i486 i3= 86 > i86 > 13/01/25 15:31:03 INFO agent.FlumeNode: Starting flume agent on: > zhouhh.TKOffice.local > 13/01/25 15:31:03 INFO agent.FlumeNode: Working directory is: > C:\Windows\system32\. > 13/01/25 15:31:03 INFO conf.FlumeConfiguration: Loading configurations > from d:/PROGRA~1/Cloudera/FLUME0~1.4/conf > 13/01/25 15:31:03 WARN text.FormatFactory: Unable to load output format > plugin class - Class not found > 13/01/25 15:31:03 INFO mortbay.log: Logging to > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > org.mortbay.log.Slf4jLog > 13/01/25 15:31:03 INFO util.InternalHttpServer: Starting internal HTTP > server > 13/01/25 15:31:03 INFO mortbay.log: jetty-6.1.26 > 13/01/25 15:31:03 INFO mortbay.log: Extract > d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\webapps\flumeagent.war to > C:\Windows\TEMP\Jetty_0_0_0_0_35862_flumeagent.war__flumeagent__4cgrz3\we= bapp > 13/01/25 15:31:03 INFO mortbay.log: Started > SelectChannelConnector@0.0.0.0:35862 > 13/01/25 15:31:03 INFO util.InternalHttpServer: Server started > 13/01/25 15:31:03 INFO agent.MultiMasterRPC: No active master RPC > connection > 13/01/25 15:31:04 INFO agent.ThriftMasterRPC: Connected to master at > hadoop48:35872 > 13/01/25 15:31:04 INFO agent.LogicalNodeManager: creating new logical nod= e > zhouhh.TKOffice.local > 13/01/25 15:31:04 INFO agent.FlumeNode: Hadoop Security enabled: false > 13/01/25 15:31:04 INFO util.InternalHttpServer: Starting internal HTTP > server > 13/01/25 15:31:04 INFO mortbay.log: jetty-6.1.26 > 13/01/25 15:31:04 INFO mortbay.log: Extract > d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\webapps\flumeagent.war to > C:\Windows\TEMP\Jetty_0_0_0_0_35862_flumeagent.war__flumeagent__4cgrz3\we= bapp > 13/01/25 15:31:04 INFO mortbay.log: Started > SelectChannelConnector@0.0.0.0:35862 > 13/01/25 15:31:04 INFO util.InternalHttpServer: Server started > 13/01/25 15:31:09 INFO agent.LogicalNode: Node config successfully set to > com.cloudera.flume.conf.FlumeConfigData@231214 > 13/01/25 15:31:09 INFO durability.NaiveFileWALManager: NaiveFileWALManage= r > is now open > 13/01/25 15:31:09 INFO durability.NaiveFileWALManager: Attempting to > recover c:\tmp\flume-ZHOUHH$\agent\zhouhh.TKOffice.local\sending / > 20130124-103034656+0800.2144885518902.00000028 > 13/01/25 15:31:09 INFO debug.MemorySinkSource: Opening mem source sink > that has with 0 elements > 13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Starting checksum > group called zhouhh.TKOffice.local2151099224995 > 13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: initial checksum is > 13c6a648fe8 > 13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Finishing checksum > group called 'zhouhh.TKOffice.local2151099224995' > 13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Checksum succeeded > 13c6a648fe8 > 13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: moved from partial to > complete zhouhh.TKOffice.local2151099224995 > Exception in thread "logicalNode zhouhh.TKOffice.local-25" > java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:= 59) > at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101) > at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1945) > at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1845) > at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1891) > at > com.cloudera.flume.handlers.hdfs.SeqfileEventSource.next(SeqfileEventSour= ce.java:62) > at com.cloudera.flume.core.EventUtil.dumpAll(EventUtil.java:47) > at > com.cloudera.flume.agent.durability.NaiveFileWALManager.checkAndStripAckF= raming(NaiveFileWALManager.java:378) > at > com.cloudera.flume.agent.durability.NaiveFileWALManager.recoverLog(NaiveF= ileWALManager.java:288) > at > com.cloudera.flume.agent.durability.NaiveFileWALManager.recover(NaiveFile= WALManager.java:411) > at > com.cloudera.flume.agent.durability.NaiveFileWALDeco.open(NaiveFileWALDec= o.java:240) > at com.cloudera.flume.agent.AgentSink.open(AgentSink.java:150) > at > com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDri= ver.java:88) > > > I have 4GB memory, and the source file to tail only 1KB, What can I do to > remove this error? > > Thanks! > Andy > > --f46d040839d152f4e804d418ce2f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I found that if I change windows agent configure from
tail("c:\\test.= txt") agentSink(&q= uot;hadoop48",35853) ;
to
tail("c:\\test.txt")=
agentDFOSink("hadoop48",35853) ;

then the agent changes status from OPENING to ACTIVE. my windows agent is :192.168.20.81, while linux collector is : 192.168.10.4= 8
another linux agent is=C2=A0192.168.10.46. 10.46 have no proble= m of agentSink.

Thanks!
Andy.
2013/1/25 =E5=91=A8=E6=A2=A6=E6=83=B3 <ablozh= ou@gmail.com>
when i start the flume node service,the agent is OPENING.
He= llo,

I need to gather logs from windows 7, but the= flume master and collector is running on linux. I download 0.9.4 exe file = from flume official web.

When I start the node from windows, the status is OPENI= NG.

zhouhh.TKOff= ice.local zhouhh.TKOffice.localOPENINGFri Jan 25 11:45:01 CST 2013 4

And the log reports Ou= tOfMemoryError.

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D
D:\Program Files\Cloudera\Flume 0.9.4\log\flumeno= de-stdout.2013-01-25.log

2013-01-25 15:31:03 Commons Daemon procrun stdout initi= alizedListening for transport dt_socket at address: 8888
java.lan= g.OutOfMemoryError: Java heap space
Dumping heap to java_pid8100.= hprof ...
Heap dump file created [13270531 bytes in 0.115 secs]

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
D:\Program= Files\Cloudera\Flume 0.9.4\log\flumenode-stderr.2013-01-25.log
<= div>
2013-01-25 15:31:03 Commons Daemon procrun stderr initia= lized
13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4
13/01/25= 15:31:03 INFO agent.FlumeNode: =C2=A0rev 8d0aa4be4ee50bb664cb0d1624e5634f4= 6d7b62a
13/01/25 15:31:03 INFO agent.FlumeNode: Compiled =C2=A0on= =C2=A0Mon Jun 20 14:46:53 PDT 2011
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.runtime.name=3DJava(TM) S= E Runtime Environment
13/01/25 15:31:03 INFO agent.FlumeNode: Sys= tem property sun.boot.library.path=3DC:\PROGRA~1\Java\jre7\bin
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vm.versio= n=3D23.3-b01
13/01/25 15:31:03 INFO agent.FlumeNode: System prope= rty java.vm.vendor=3DOracle Corporation
13/01/25 15:31:03 INFO ag= ent.FlumeNode: System property java.vendor.url=3Dhttp://java.oracle.com/
13/01/25 15:31:03 INFO agent.FlumeNode: System property path.separator= =3D;
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vm.name=3DJava HotSp= ot(TM) Client VM
13/01/25 15:31:03 INFO agent.FlumeNode: System property file.encoding.= pkg=3Dsun.io
13/0= 1/25 15:31:03 INFO agent.FlumeNode: System property user.country=3DCN
=
13/01/25 15:31:03 INFO agent.FlumeNode: System property user.script=3D
13/01/25 15:31:03 INFO agent.FlumeNode: System property sun.os.patch.l= evel=3DService Pack 1
13/01/25 15:31:03 INFO agent.FlumeNode: Sys= tem property java.vm.specification.name=3DJava Virtual Machine Specification
13/01/25 15:31:03 INFO agent.FlumeNode: System property user.dir=3DC:\= Windows\system32
13/01/25 15:31:03 INFO agent.FlumeNode: System p= roperty java.runtime.version=3D1.7.0_07-b11
13/01/25 15:31:03 INF= O agent.FlumeNode: System property flume=3D
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.awt.graph= icsenv=3Dsun.awt.Win32GraphicsEnvironment
13/01/25 15:31:03 INFO = agent.FlumeNode: System property java.endorsed.dirs=3DC:\PROGRA~1\Java\jre7= \lib\endorsed
13/01/25 15:31:03 INFO agent.FlumeNode: System property os.arch=3Dx86<= /div>
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.io.t= mpdir=3DC:\Windows\TEMP\
13/01/25 15:31:03 INFO agent.FlumeNode: = System property line.separator=3D

13/01/25 15:31:03 INFO agent.FlumeNode: System property= java.vm.specification.vendor=3DOracle Corporation
13/01/25 15:31= :03 INFO agent.FlumeNode: System property user.variant=3D
13/01/2= 5 15:31:03 INFO agent.FlumeNode: System property os.name=3DWindows 7
13/01/25 15:31:03 INFO agent.FlumeNode: System property flume.home=3Dd= :\PROGRA~1\Cloudera\FLUME0~1.4\bin\..
13/01/25 15:31:03 INFO agen= t.FlumeNode: System property sun.jnu.encoding=3DGBK
13/01/25 15:3= 1:03 INFO agent.FlumeNode: System property java.library.path=3Dd:\PROGRA~1\= Cloudera\FLUME0~1.4\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Wind= ows;C:\Program Files\Common Files\NetSarang;C:\Python27\;C:\Python27\Script= s;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System= 32\WindowsPowerShell\v1.0\;C:\Program Files\Microsoft SQL Server\100\Tools\= Binn\;C:\Program Files\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files\= Common Files\Thunder Network\KanKan\Codecs;d:\php;d:\php\ext;D:\Program Fil= es\MySQL\MySQL Server 5.5\bin;D:\mysql\bin;D:\wxWidgets-2.9.1\include;D:\wx= Widgets-2.9.1\include\msvc;W:\Server\include;W:\Base\include;W:\Client\incl= ude;d:\Program Files\Lua\5.1;d:\Program Files\Lua\5.1\clibs;D:\Program File= s\TortoiseSVN\bin;d:\Program Files\Git\cmd;d:\Program Files\Git\bin;D:\Prog= ram Files\TortoiseGit\bin;D:\docbook\bin;;.
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.specification.name= =3DJava Platform API Specification
13/01/25 15:31:03 INFO agent.F= lumeNode: System property java.class.version=3D51.0
13/01/25 15:31:03 INFO agent.FlumeNode: System property sun.management= .compiler=3DHotSpot Client Compiler
13/01/25 15:31:03 INFO agent.= FlumeNode: System property os.version=3D6.1
13/01/25 15:31:03 INF= O agent.FlumeNode: System property user.home=3DC:\
13/01/25 15:31:03 INFO agent.FlumeNode: System property user.timezone= =3DAsia/Shanghai
13/01/25 15:31:03 INFO agent.FlumeNode: System p= roperty java.awt.printerjob=3Dsun.awt.windows.WPrinterJob
13/01/2= 5 15:31:03 INFO agent.FlumeNode: System property file.encoding=3DGBK
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.specifica= tion.version=3D1.7
13/01/25 15:31:03 INFO agent.FlumeNode: System= property java.class.path=3Dd:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\build\cl= asses;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\activation-1.1.jar;d:\PROGRA~1\Cl= oudera\FLUME0~1.4\lib\ahocorasick-2.x.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\l= ib\ant-1.6.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-2.7.7.jar;d:\PRO= GRA~1\Cloudera\FLUME0~1.4\lib\antlr-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4= \lib\antlr-runtime-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\asm-3.1.jar;= d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\automaton-1.0.0.jar;d:\PROGRA~1\Clouder= a\FLUME0~1.4\lib\avro-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-co= mpiler-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-ipc-1.5.1.jar;d:\= PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-cli-1.2.jar;d:\PROGRA~1\Cloudera\F= LUME0~1.4\lib\commons-codec-1.3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\com= mons-collections-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-el-1.0= .jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-httpclient-3.1.jar;d:\PROG= RA~1\Cloudera\FLUME0~1.4\lib\commons-lang-2.5.jar;d:\PROGRA~1\Cloudera\FLUM= E0~1.4\lib\commons-logging-1.0.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\co= mmons-net-1.4.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\core-3.1.1.jar;d:\P= ROGRA~1\Cloudera\FLUME0~1.4\lib\flume-core-0.9.4.jar;d:\PROGRA~1\Cloudera\F= LUME0~1.4\lib\guava-r07.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\hadoop-core= -0.20.2-CDH3B4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\hsqldb-1.8.0.7.jar;d= :\PROGRA~1\Cloudera\FLUME0~1.4\lib\irclib-1.0.0.jar;d:\PROGRA~1\Cloudera\FL= UME0~1.4\lib\jackson-core-asl-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib= \jackson-jaxrs-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-mapper= -asl-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-xc-1.5.5.jar;d:\= PROGRA~1\Cloudera\FLUME0~1.4\lib\jasper-compiler-5.5.23.jar;d:\PROGRA~1\Clo= udera\FLUME0~1.4\lib\jasper-runtime-5.5.23.jar;d:\PROGRA~1\Cloudera\FLUME0~= 1.4\lib\jaxb-api-2.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jaxb-impl-2.1.= 12.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jersey-core-1.4.jar;d:\PROGRA~1\= Cloudera\FLUME0~1.4\lib\jersey-json-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4= \lib\jersey-server-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jets3t-0.6.1= .jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jettison-1.1.jar;d:\PROGRA~1\Cloud= era\FLUME0~1.4\lib\jetty-6.1.26.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jet= ty-util-6.1.26.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jline-0.9.94.jar;d:\= PROGRA~1\Cloudera\FLUME0~1.4\lib\jms-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.= 4\lib\jsp-api-2.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jsp-api-2.1.jar;d= :\PROGRA~1\Cloudera\FLUME0~1.4\lib\libthrift-0.6.0-cdh.jar;d:\PROGRA~1\Clou= dera\FLUME0~1.4\lib\log4j-1.2.15.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\ma= il-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\netty-3.2.4.Final.jar;d:\PRO= GRA~1\Cloudera\FLUME0~1.4\lib\oro-2.0.8.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4= \lib\paranamer-2.3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\servlet-api-2.5-= 20081211.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\servlet-api-2.5.jar;d:\PRO= GRA~1\Cloudera\FLUME0~1.4\lib\slf4j-api-1.5.11.jar;d:\PROGRA~1\Cloudera\FLU= ME0~1.4\lib\slf4j-log4j12-1.5.11.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\sn= appy-java-1.0.1-rc3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stax-api-1.0-2.= jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stax-api-1.0.1.jar;d:\PROGRA~1\Clou= dera\FLUME0~1.4\lib\stringtemplate-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\= lib\velocity-1.6.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\xmlenc-0.52.jar;= d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\zookeeper-3.3.1-10.jar;d:\PROGRA~1\Clou= dera\FLUME0~1.4\bin\..\conf
13/01/25 15:31:03 INFO agent.FlumeNode: System property user.name=3DZHOUHH$
13/01/25= 15:31:03 INFO agent.FlumeNode: System property flume.root.logger=3DINFO,co= nsole
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vm.specif= ication.version=3D1.7
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.home=3DC:= \PROGRA~1\Java\jre7
13/01/25 15:31:03 INFO agent.FlumeNode: Syste= m property sun.arch.data.model=3D32
13/01/25 15:31:03 INFO agent.= FlumeNode: System property user.language=3Dzh
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.specifica= tion.vendor=3DOracle Corporation
13/01/25 15:31:03 INFO agent.Flu= meNode: System property awt.toolkit=3Dsun.awt.windows.WToolkit
13= /01/25 15:31:03 INFO agent.FlumeNode: System property java.vm.info=3Dmixed mode
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.version= =3D1.7.0_07
13/01/25 15:31:03 INFO agent.FlumeNode: System proper= ty java.ext.dirs=3DC:\PROGRA~1\Java\jre7\lib\ext;C:\Windows\Sun\Java\lib\ex= t
13/01/25 15:31:03 INFO agent.FlumeNode: System property sun.boot.class= .path=3DC:\PROGRA~1\Java\jre7\lib\alt-rt.jar;C:\PROGRA~1\Java\jre7\lib\reso= urces.jar;C:\PROGRA~1\Java\jre7\lib\rt.jar;C:\PROGRA~1\Java\jre7\lib\sunrsa= sign.jar;C:\PROGRA~1\Java\jre7\lib\jsse.jar;C:\PROGRA~1\Java\jre7\lib\jce.j= ar;C:\PROGRA~1\Java\jre7\lib\charsets.jar;C:\PROGRA~1\Java\jre7\lib\jfr.jar= ;C:\PROGRA~1\Java\jre7\classes
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vendor=3D= Oracle Corporation
13/01/25 15:31:03 INFO agent.FlumeNode: System= property file.separator=3D\
13/01/25 15:31:03 INFO agent.FlumeNo= de: System property java.vendor.url.bug=3Dhttp://bugreport.sun.com/bugreport/
13/01/25 15:31:03 INFO agent.FlumeNode: System property sun.io.unicode= .encoding=3DUnicodeLittle
13/01/25 15:31:03 INFO agent.FlumeNode:= System property sun.cpu.endian=3Dlittle
13/01/25 15:31:03 INFO a= gent.FlumeNode: System property sun.desktop=3Dwindows
13/01/25 15:31:03 INFO agent.FlumeNode: System property sun.cpu.isalis= t=3Dpentium_pro+mmx pentium_pro pentium+mmx pentium i486 i386 i86
13/01/25 15:31:03 INFO agent.FlumeNode: Starting flume agent on: zhouhh.TK= Office.local
13/01/25 15:31:03 INFO agent.FlumeNode: =C2=A0Working directory is: C:= \Windows\system32\.
13/01/25 15:31:03 INFO conf.FlumeConfiguratio= n: Loading configurations from d:/PROGRA~1/Cloudera/FLUME0~1.4/conf
13/01/25 15:31:03 WARN text.FormatFactory: Unable to load output format plu= gin class =C2=A0- Class not found
13/01/25 15:31:03 INFO mortbay.= log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.= mortbay.log.Slf4jLog
13/01/25 15:31:03 INFO util.InternalHttpServer: Starting internal HTTP= server
13/01/25 15:31:03 INFO mortbay.log: jetty-6.1.26
13/01/25 15:31:03 INFO mortbay.log: Extract d:\PROGRA~1\Cloudera\FLUME0~1= .4\bin\..\webapps\flumeagent.war to C:\Windows\TEMP\Jetty_0_0_0_0_35862_flu= meagent.war__flumeagent__4cgrz3\webapp
13/01/25 15:31:03 INFO mortbay.log: Started SelectChannelConnector@0.0.= 0.0:35862
13/01/25 15:31:03 INFO util.InternalHttpServer: Ser= ver started
13/01/25 15:31:03 INFO agent.MultiMasterRPC: No active master RPC conn= ection
13/01/25 15:31:04 INFO agent.ThriftMasterRPC: Connected to= master at hadoop48:35872
13/01/25 15:31:04 INFO agent.LogicalNod= eManager: creating new logical node zhouhh.TKOffice.local
13/01/25 15:31:04 INFO agent.FlumeNode: Hadoop Security enabled: false=
13/01/25 15:31:04 INFO util.InternalHttpServer: Starting interna= l HTTP server
13/01/25 15:31:04 INFO mortbay.log: jetty-6.1.26
13/01/25 15:31:04 INFO mortbay.log: Extract d:\PROGRA~1\Cloudera\FLUME= 0~1.4\bin\..\webapps\flumeagent.war to C:\Windows\TEMP\Jetty_0_0_0_0_35862_= flumeagent.war__flumeagent__4cgrz3\webapp
13/01/25 15:31:04 INFO = mortbay.log: Started SelectChannelConnector@0.0.0.0:35862
13/01/25 15:31:04 INFO util.InternalHttpServer: Server started
13/01/25 15:31:09 INFO agent.LogicalNode: Node config successfully set t= o com.cloudera.flume.conf.FlumeConfigData@231214
13/01/25 15:31:0= 9 INFO durability.NaiveFileWALManager: NaiveFileWALManager is now open
13/01/25 15:31:09 INFO durability.NaiveFileWALManager: Attempting to r= ecover c:\tmp\flume-ZHOUHH$\agent\zhouhh.TKOffice.local\sending / 20130124-= 103034656+0800.2144885518902.00000028
13/01/25 15:31:09 INFO debu= g.MemorySinkSource: Opening mem source sink that has with 0 elements
13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Starting checksum = group called zhouhh.TKOffice.local2151099224995
13/01/25 15:31:09= INFO endtoend.AckChecksumChecker: initial checksum is 13c6a648fe8
13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Finishing checksum= group called 'zhouhh.TKOffice.local2151099224995'
13/01/= 25 15:31:09 INFO endtoend.AckChecksumChecker: Checksum succeeded 13c6a648fe= 8
13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: moved from partial= to complete zhouhh.TKOffice.local2151099224995
Exception in thre= ad "logicalNode zhouhh.TKOffice.local-25" java.lang.OutOfMemoryEr= ror: Java heap space
at org.apache.hadoop.io.D= ataOutputBuffer$Buffer.write(DataOutputBuffer.java:59)
at org.apache.hadoop.io.DataOutputBuffer= .write(DataOutputBuffer.java:101)
at org.apache.hadoop.io.S= equenceFile$Reader.next(SequenceFile.java:1945)
at org.apache.hadoop.io.SequenceFile$Reader.nex= t(SequenceFile.java:1845)
at org.apache.hadoop.io.S= equenceFile$Reader.next(SequenceFile.java:1891)
at com.cloudera.flume.handlers.hdfs.SeqfileEven= tSource.next(SeqfileEventSource.java:62)
at com.cloudera.flume.cor= e.EventUtil.dumpAll(EventUtil.java:47)
at com.cloudera.flume.agent.durability.NaiveFileWALManag= er.checkAndStripAckFraming(NaiveFileWALManager.java:378)
at com.cloudera.flume.age= nt.durability.NaiveFileWALManager.recoverLog(NaiveFileWALManager.java:288)<= /div>
at com.cloudera.flum= e.agent.durability.NaiveFileWALManager.recover(NaiveFileWALManager.java:411= )
at com.cloudera.flume.age= nt.durability.NaiveFileWALDeco.open(NaiveFileWALDeco.java:240)
at com.cloudera.flume.agent.Agen= tSink.open(AgentSink.java:150)
at com.cloudera.flume.cor= e.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
=


I have 4GB memory, and the source file t= o tail only 1KB, What can I do to remove this error?

Thanks!
Andy


--f46d040839d152f4e804d418ce2f--