Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 220EAD4FC for ; Tue, 30 Oct 2012 23:06:40 +0000 (UTC) Received: (qmail 12334 invoked by uid 500); 30 Oct 2012 23:06:39 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 12290 invoked by uid 500); 30 Oct 2012 23:06:39 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 12281 invoked by uid 99); 30 Oct 2012 23:06:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Oct 2012 23:06:39 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of pchavez@verticalsearchworks.com designates 66.150.157.191 as permitted sender) Received: from [66.150.157.191] (HELO mail.verticalsearchworks.com) (66.150.157.191) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Oct 2012 23:06:31 +0000 Received: from CNVR-EXCH01.convera.com ([10.21.0.21]) by cnvr-exch01 ([10.21.0.21]) with mapi; Tue, 30 Oct 2012 16:06:09 -0700 From: Paul Chavez To: "user@flume.apache.org" Date: Tue, 30 Oct 2012 16:06:08 -0700 Subject: RE: Help determining exit code from flume avro-client Thread-Topic: Help determining exit code from flume avro-client Thread-Index: Ac22zxQoZYn1w2AZTI25yy642yY6BQAI+zFQ Message-ID: References: <2EDC751B6B27466BB81DB27C3870D780@cloudera.com> In-Reply-To: <2EDC751B6B27466BB81DB27C3870D780@cloudera.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CFA73F1CD063634CB7C435EBBC36BA4B0C7988BBE6cnvrexch01_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CFA73F1CD063634CB7C435EBBC36BA4B0C7988BBE6cnvrexch01_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thank you, I was unsure if that was desired behavior or not. I updated the = class as a POC to add exit codes for each of those catch blocks and it work= s great. Thanks, Paul ________________________________ From: Hari Shreedharan [mailto:hshreedharan@cloudera.com] Sent: Tuesday, October 30, 2012 11:47 AM To: user@flume.apache.org Subject: Re: Help determining exit code from flume avro-client Hi Paul. Thanks for reporting this. It does look like Flume does not return non-zero= error code even if there was an exception. I filed https://issues.apache.o= rg/jira/browse/FLUME-1670 to address this. Thanks, Hari -- Hari Shreedharan On Tuesday, October 30, 2012 at 11:25 AM, Paul Chavez wrote: I am working on getting the flume avro-client functionality working on Wind= ows, and am currently stuck on how to determine if the file was sent succes= sfully. I used the blog post at http://mapredit.blogspot.com/2012/07/run-flume-13x-= on-windows.html to guide me and have built the 1.4.0 snapshot successfully.= I can manually send files along to flume using the ''org.apache.flume.clie= nt.avro.AvroCLIClient" class. The full command I am using is invoked via batch file with the following si= ngle line: "C:\Program Files (x86)\Java\jdk1.6.0_37\bin\java.exe" -Xmx20m -Dlog4j.conf= iguration=3Dfile:///C:\apache-flume-1.4.0-SNAPSHOT\conf\log4j.properties -c= p "C:\apache-flume-1.4.0-SNAPSHOT\lib\*" org.apache.flume.client.avro.AvroC= LIClient -H 10.20.2.8 -p 6240 -F%1 My intention is to invoke from Powershell like so: .\flume.bat C:\IISLogs\W3SVC100\u_ex121029.log This is working well and I am able to successfully send IIS and our app log= s to flume and then onto HDFS without issues. Now I am trying to automate t= he process, and need to know if the operation failed for some reason. If I = simply turn off the flume source I can trigger an error which is dumped to = console like so (using flume.root.logger=3DDEBUG,console in the log4j file)= : 2012-10-30 11:14:22,185 (main) [ERROR - org.apache.flume.client.avro.AvroCL= IClient.main(AvroCLIClient.java:76)] Unable t o open connection to Flume. Exception follows. org.apache.flume.FlumeException: NettyAvroRpcClient { host: x.x.x.x, port: = 6240 }: RPC connection error at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClie= nt.java:117) at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClie= nt.java:93) at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcCl= ient.java:507) at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClie= ntFactory.java:169) at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.jav= a:168) at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.ja= va:68) Caused by: java.io.IOException: Error connecting to /x.x.x.x:6240 at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver= .java:261) at org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.jav= a:203) at org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.jav= a:152) at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClie= nt.java:106) ... 5 more Caused by: java.net.ConnectException: Connection refused: no further inform= ation at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.jav= a:599) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$B= oss.connect(NioClientSocketPipelineSink.java: 401) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$B= oss.processSelectedKeys(NioClientSocketPipeli neSink.java:370) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$B= oss.run(NioClientSocketPipelineSink.java:292) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoo= lExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:908) at java.lang.Thread.run(Thread.java:662) 2012-10-30 11:14:22,185 (main) [DEBUG - org.apache.flume.client.avro.AvroCL= IClient.main(AvroCLIClient.java:81)] Exiting Despite the ERROR message being logged to the console, java.exe still exits= with code 0, indicating success. Is there a way to use the console appende= r to trigger java.exe to send a non-0 exit code if an error occurs so the r= est of my automation tooling can detect the error? I can parse the output w= ith Powershell and have it send a non-0 exit code itself, but looking for a= more 'native' way to do this. thanks, Paul Chavez --_000_CFA73F1CD063634CB7C435EBBC36BA4B0C7988BBE6cnvrexch01_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Thank you, I was unsure if that was desired behavior = or not. I=20 updated the class as a POC to add exit codes for each of those catch blocks= and=20 it works great.
 
Thanks,
Paul
 <= /DIV>
From: Hari Shreedharan=20 [mailto:hshreedharan@cloudera.com]
Sent: Tuesday, October 30, 20= 12=20 11:47 AM
To: user@flume.apache.org
Subject: Re: Help=20 determining exit code from flume avro-client

Hi Paul.

Thanks for reporting this. It does loo= k like=20 Flume does not return non-zero error code even if there was an exception. I= =20 filed https://issues.ap= ache.org/jira/browse/FLUME-1670 to=20 address this. 

Thanks,
Hari

-- 
Hari Shreedharan

On Tuesday, October 30, 2012 at 11:25 AM, Paul = Chavez=20 wrote:

I am working on getting the flume= =20 avro-client functionality working on Windows, and am currently stuck on h= ow to=20 determine if the file was sent successfully.
 
I used the blog post at http://mapredit.blogspot.com/2012/07/run-flume-1= 3x-on-windows.html to=20 guide me and have built the 1.4.0 snapshot successfully. I can manually s= end=20 files along to flume using the ''org.apache.flume.client.avro.AvroCLIClie= nt"=20 class.
 
The full command I am using is inv= oked via=20 batch file with the following single line:
"C:\Program Files=20 (x86)\Java\jdk1.6.0_37\bin\java.exe" -Xmx20m=20 -Dlog4j.configuration=3Dfile:///C:\apache-flume-1.4.0-SNAPSHOT\conf\log4j= .properties=20 -cp "C:\apache-flume-1.4.0-SNAPSHOT\lib\*"=20 org.apache.flume.client.avro.AvroCLIClient -H 10.20.2.8 -p 6240=20 -F%1
 
My intention is to invoke from Pow= ershell=20 like so:
.\flume.bat=20 C:\IISLogs\W3SVC100\u_ex121029.log
 
This is working well and I am able= to=20 successfully send IIS and our app logs to flume and then onto HDFS withou= t=20 issues. Now I am trying to automate the process, and need to know if the= =20 operation failed for some reason. If I simply turn off the flume source I= can=20 trigger an error which is dumped to console like so (using=20 flume.root.logger=3DDEBUG,console in the log4j file):
 
 
2012-10-30 11:14:22,185 (main) [ER= ROR -=20 org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] U= nable=20 t
o open connection to Flume. Exception=20 follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host:=20 x.x.x.x, port: 6240 }: RPC connection=20 error
        at=20 org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:1= 17)
       =20 at=20 org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:9= 3)
       =20 at=20 org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java= :507)
       =20 at=20 org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory= .java:169)
       =20 at=20 org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)       =20 at=20 org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)Caused=20 by: java.io.IOException: Error connecting to=20 /x.x.x.x:6240
        at=20 org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261= )
       =20 at=20 org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:2= 03)
       =20 at=20 org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:1= 52)
       =20 at=20 org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:1= 06)
       =20 ... 5 more
Caused by: java.net.ConnectException: Connection refused: n= o=20 further information
        at=20 sun.nio.ch.SocketChannelImpl.checkConnect(Native=20 Method)
        at=20 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)       =20 at=20 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.conne= ct(NioClientSocketPipelineSink.java:
401)
    &nb= sp;  =20 at=20 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.proce= ssSelectedKeys(NioClientSocketPipeli
neSink.java:370)
  &nb= sp;    =20 at=20 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(N= ioClientSocketPipelineSink.java:292)
 
     &nbs= p; =20 at=20 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor= .java:886)
       =20 at=20 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:908)
       =20 at java.lang.Thread.run(Thread.java:662)
2012-10-30 11:14:22,185 (main= )=20 [DEBUG -=20 org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)]=20 Exiting
 
Despite the ERROR message being lo= gged to=20 the console, java.exe still exits with code 0, indicating success. Is the= re a=20 way to use the console appender to trigger java.exe to send a non-0 exit = code=20 if an error occurs so the rest of my automation tooling can detect the er= ror?=20 I can parse the output with Powershell and have it send a non-0 exit code= =20 itself, but looking for a more 'native' way to do this.
 
thanks,
Paul=20 Chavez

--_000_CFA73F1CD063634CB7C435EBBC36BA4B0C7988BBE6cnvrexch01_--