Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A5D6F1003F for ; Tue, 31 Dec 2013 03:54:46 +0000 (UTC) Received: (qmail 26334 invoked by uid 500); 31 Dec 2013 03:54:42 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 26201 invoked by uid 500); 31 Dec 2013 03:54:41 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 26003 invoked by uid 99); 31 Dec 2013 03:54:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Dec 2013 03:54:37 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,UNPARSEABLE_RELAY X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [216.82.243.208] (HELO mail1.bemta8.messagelabs.com) (216.82.243.208) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Dec 2013 03:54:29 +0000 Received: from [216.82.241.100:25497] by server-16.bemta-8.messagelabs.com id E1/16-20886-FDF32C25; Tue, 31 Dec 2013 03:54:07 +0000 X-Env-Sender: Chhaya.Vishwakarma@lntinfotech.com X-Msg-Ref: server-11.tower-220.messagelabs.com!1388462044!5652617!1 X-Originating-IP: [202.54.41.44] X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 31390 invoked from network); 31 Dec 2013 03:54:06 -0000 Received: from mailsvr.ltitl.co.in (HELO PUNINMSHTCAS02.puneodc.lntinfotech.com) (202.54.41.44) by server-11.tower-220.messagelabs.com with AES128-SHA encrypted SMTP; 31 Dec 2013 03:54:06 -0000 Received: from puninmsmbx01.puneodc.lntinfotech.com ([169.254.2.74]) by PUNINMSHTCAS02.puneodc.lntinfotech.com ([172.20.8.114]) with mapi; Tue, 31 Dec 2013 09:24:03 +0530 From: Chhaya Vishwakarma To: "user@flume.apache.org" Date: Tue, 31 Dec 2013 09:24:02 +0530 Subject: RE: Event breaking in flume Thread-Topic: Event breaking in flume Thread-Index: Ac8Fah2GKY+YOC9xTkia7m8qpF37AwAcWSRg Message-ID: <969B118F8FF0104281DFF098C33475405474F09908@PUNINMSMBX01.puneodc.lntinfotech.com> References: <969B118F8FF0104281DFF098C33475405474F097B0@PUNINMSMBX01.puneodc.lntinfotech.com> <969B118F8FF0104281DFF098C33475405474F097E1@PUNINMSMBX01.puneodc.lntinfotech.com> <969B118F8FF0104281DFF098C33475405474F0980A@PUNINMSMBX01.puneodc.lntinfotech.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_969B118F8FF0104281DFF098C33475405474F09908PUNINMSMBX01p_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_969B118F8FF0104281DFF098C33475405474F09908PUNINMSMBX01p_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi brock Thanks. Using spooling directory with deserializer looks good however i don= 't have any idea of how to write custom deserializer. Can you give me little hint how should i go about writing my own deserializ= er it will be a great help. Regards, Chhaya Vishwakarma From: Brock Noland [mailto:brock@cloudera.com] Sent: Monday, December 30, 2013 7:48 PM To: user@flume.apache.org Subject: Re: Event breaking in flume Yes, it is possible to handle multi-line events and handling stack traces i= s very common place. However, using exec source is going to be limiting. The "correct" solution = is: 1) Use spooling directory source 2) Write a little deserializer to handle your format. Another solution is: 1) replace new lines with something like __NL__ by a perl script in your ex= ec source 2) Use morphlines to replace __NL__ with \n A third and less desirable solution would be: 1) Use the morphlines intercepter to merge multiple events to a single even= t. This will not work well for a varity or reasons but the most common bein= g that the exec source could hit it's "batch" size in the middle of of a st= ack trace in which case the stack trace will be in to different batches. Brock On Mon, Dec 30, 2013 at 5:05 AM, Joao Salcedo > wrote: Looks that it is possible based on regular expression pattern matching http://kitesdk.org/docs/current/kite-morphlines/morphlinesReferenceGuide.ht= ml#/readMultiLine On Mon, Dec 30, 2013 at 9:56 PM, Chhaya Vishwakarma > wrote: So is it not possible to handle multiline events in flume? From: Joao Salcedo [mailto:joao.salcedo@gmail.com] Sent: Monday, December 30, 2013 4:22 PM To: user@flume.apache.org Subject: Re: Event breaking in flume Maybe you can set up some morphlines and do some ETL in your event. I hope this help you. http://blog.cloudera.com/blog/2013/07/morphlines-the-easy-way-to-build-and-= integrate-etl-apps-for-apache-hadoop/ Cheers On Mon, Dec 30, 2013 at 9:34 PM, Ashish > wrote: I am not aware of any options out of the box. Maybe someone else can help. Alternate way is to write a custom source. On Mon, Dec 30, 2013 at 3:56 PM, Chhaya Vishwakarma > wrote: Hi Exec as source and tail command From: Ashish [mailto:paliwalashish@gmail.com] Sent: Monday, December 30, 2013 3:48 PM To: user@flume.apache.org Subject: Re: Event breaking in flume What is the Source you are using? On Mon, Dec 30, 2013 at 3:23 PM, Chhaya Vishwakarma > wrote: Hi, By default flume considers one line as one event, But I want to do breaking= on some other criteria how it can be achieved in flume? Is it possible to = do ? 10 Sep 2013 19:43:33,561 [WebContainer : 9] ERROR - An Error has occured fo= r com.marsh.framework.core.exception.MarshException: Record has been modifi= ed since last retrieved - Resubmit transaction 10 Sep 2013 19:43:33,561 [WebContainer : 9] ERROR - handleException():com.m= arsh.framework.core.exception.MarshException: Record has been modified sinc= e last retrieved - Resubmit transaction at com.marsh.csa.serviceagreement.ServiceAgreementImpl.updateAgreement= (ServiceAgreementImpl.java(Compiled Code)) at com.marsh.csa.serviceagreementmgmt.CSAManagerImpl.updateCSA(CSAMana= gerImpl.java(Compiled Code)) at com.marsh.csa.serviceagreementmgmt.ejb.EJSRemoteStatelessServiceagr= eementManager_3dcfd156.updateCSA(Unknown Source) at com.marsh.csa.serviceagreementmgmt.ejb._ServiceagreementManagerRemo= te_Stub.updateCSA(_ServiceagreementManagerRemote_Stub.java(Compiled Code)) at com.marsh.csa.proxy.CSAProxy.updateCSA(CSAProxy.java(Compiled Code)= ) at com.marsh.csa.serviceagreement.SaveCSAAction.performAction(SaveCSAA= ction.java(Compiled Code)) at com.marsh.csa.serviceagreement.CSAAbstractStrutsAction.execute(CSAA= bstractStrutsAction.java(Compiled Code)) at org.apache.struts.action.RequestProcessor.processActionPerform(Requ= estProcessor.java(Inlined Compiled Code)) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code= )) Caused by: com.marsh.framework.core.exception.MarshException: Record has be= en modified since last retrieved - Resubmit transaction at com.marsh.csa.serviceagreement.ServiceAgreementDAO.updateServiceAgr= eement(ServiceAgreementDAO.java(Compiled Code)) at com.marsh.csa.serviceagreement.ServiceAgreementDAO.update(ServiceAg= reementDAO.java(Compiled Code)) at com.marsh.csa.serviceagreement.SAUpdateImpl.updateServiceAgreement(= SAUpdateImpl.java(Compiled Code)) at com.marsh.csa.serviceagreement.SAUpdateImpl.update(SAUpdateImpl.jav= a(Compiled Code)) ... 26 more Caused by: com.marsh.framework.core.exception.MarshException: Record has be= en modified since last retrieved - Resubmit transaction at com.marsh.csa.serviceagreement.SaveCSAAction.performAction(SaveCSAA= ction.java(Compiled Code)) at com.marsh.csa.serviceagreement.CSAAbstractStrutsAction.execute(CSAA= bstractStrutsAction.java(Compiled Code)) at org.apache.struts.action.RequestProcessor.processActionPerform(Requ= estProcessor.java(Inlined Compiled Code)) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.= java(Compiled Code)) at org.apache.struts.action.ActionServlet.process(ActionServlet.java(I= nlined Compiled Code)) at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java(Co= mpiled Code)) at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Co= de)) at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Co= de)) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapp= er.java(Compiled Code)) this is a log file which I am writing to HBase. Whatever is highlighted das= yellow I want that as one event and gray as another event. Basically I want to break the events on Date? Is it possible to do ? Regards, Chhaya Vishwakarma ________________________________ The contents of this e-mail and any attachment(s) may contain confidential = or privileged information for the intended recipient(s). Unintended recipie= nts are prohibited from taking action on the basis of information in this e= -mail and using or disseminating the information, and must notify the sende= r and delete it from their system. L&T Infotech will not accept responsibil= ity or liability for the accuracy or completeness of, or the presence of an= y virus or disabling code in this e-mail" -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org --_000_969B118F8FF0104281DFF098C33475405474F09908PUNINMSMBX01p_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi brock<= o:p>

 

Thanks. Using spooling directory with deseria= lizer looks good however i don’t have any idea of how to write custom= deserializer.

Can you giv= e me little hint how should i go about writing my own deserializer it will = be a great help.

&nbs= p;

 

=

Regards,

Chhaya Vishwakarma

 

 

 

From: Brock Noland [mailto:brock@cloudera.com]
Sent: Monday, December 30, 2013 7:48 PM
To: user@flume.apache.org
= Subject: Re: Event breaking in flume

 

Yes, it is poss= ible to handle multi-line events and handling stack traces is very common p= lace. 

 

However, using exec source is going to be lim= iting. The "correct" solution is:

 

1) Use spooli= ng directory source

2) Write = a little deserializer to handle your format.

 

Anothe= r solution is:

 

1) replace new lines with something = like __NL__ by a perl script in your exec source

<= p class=3DMsoNormal>2) Use morphlines to replace __NL__ with \n<= /p>

 

A third and less desirable solution would be:

 

1) Use the morphlines intercepter to merge multiple events to a sin= gle event. This will not work well for a varity or reasons but the most com= mon being that the exec source could hit it's "batch" size in the= middle of of a stack trace in which case the stack trace will be in to dif= ferent batches.

 

Brock

 

 

On Mon, Dec 30, 2013 at 5:05 AM, Joao Salcedo <joao.salcedo@gmail.com>= wrote:

Looks that it is possible b= ased on regular expression pattern matching

 

=  

On Mon, Dec 30, 2013 at 9:56= PM, Chhaya Vishwakarma <Chhaya.Vishwakarma@lntinfotech.com> wrote:<= o:p>

So is it not possible to handle multi= line events in flume?

 =

From: Joao Salcedo [mailto:joao.salcedo@gmail.com]
Sent: Mon= day, December 30, 2013 4:22 PM


To: user@flume.apache.org
Subject: Re: Event breaking in f= lume

 

<= p class=3DMsoNormal style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:= auto'>Maybe you can set up some morphlines and do some ETL in your event.

 

I hope this= help you.

 

&= nbsp;

Cheers

 

On Mon, Dec 30, 2013 at 9:34 PM, Ashish = <paliwalash= ish@gmail.com> wrote:

I am not aware of a= ny options out of the box. Maybe someone else can help. 

Alternate way is to write a custom source.

 

On Mon, Dec 30, 201= 3 at 3:56 PM, Chhaya Vishwakarma <Chhaya.Vishwakarma@lntinfotech.com>= ; wrote:

Hi

Exec as source and tail command

<= span style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F= 497D'> 

 

From: Ashish [mailto:paliwalashish@gmail.com]
Sent: Monday, December= 30, 2013 3:48 PM
To: user@flume.apache.org
Subject: Re: Event brea= king in flume

 

What is the Source you are using?

 

On Mon, Dec 30, 2013 at 3:23 PM, Chhaya Vishwak= arma <Chhaya.Vishwakarma@lntinfotech.com> wrote:

=

Hi,

 

By default= flume considers one line as one event, But I want to do breaking on some o= ther criteria how it can be achieved in flume? Is it possible to do ?<= /o:p>

 

10 Sep 2013 19:43:33,561 [= WebContainer : 9] ERROR - An Error has occured for com.marsh.framework.core= .exception.MarshException: Record has been modified since last retrieved - = Resubmit transaction

 

10 Sep 2013 19:43:33,561 [WebContainer : 9] ERROR - handleException():com.= marsh.framework.core.exception.MarshException: Record has been modified sin= ce last retrieved - Resubmit transaction

&= nbsp;    at com.marsh.csa.serviceagreement.ServiceAgreementI= mpl.updateAgreement(ServiceAgreementImpl.java(Compiled Code))

     at com.marsh.csa.serviceagreem= entmgmt.CSAManagerImpl.updateCSA(CSAManagerImpl.java(Compiled Code))=

     at com.marsh.csa.servic= eagreementmgmt.ejb.EJSRemoteStatelessServiceagreementManager_3dcfd156.updat= eCSA(Unknown Source)

   &nb= sp; at com.marsh.csa.serviceagreementmgmt.ejb._ServiceagreementManagerRemot= e_Stub.updateCSA(_ServiceagreementManagerRemote_Stub.java(Compiled Code))

     at com.marsh.csa.p= roxy.CSAProxy.updateCSA(CSAProxy.java(Compiled Code))

=

     at com.marsh.csa.serviceagreement.Save= CSAAction.performAction(SaveCSAAction.java(Compiled Code))

     at com.marsh.csa.serviceagreement= .CSAAbstractStrutsAction.execute(CSAAbstractStrutsAction.java(Compiled Code= ))

     at org.apache.= struts.action.RequestProcessor.processActionPerform(RequestProcessor.java(I= nlined Compiled Code))

   &= nbsp; at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Cod= e))

Caused by: com.marsh.framework.core.ex= ception.MarshException: Record has been modified since last retrieved - Res= ubmit transaction

    = at com.marsh.csa.serviceagreement.ServiceAgreementDAO.updateServiceAgreeme= nt(ServiceAgreementDAO.java(Compiled Code))

     at com.marsh.csa.serviceagreement.ServiceAgree= mentDAO.update(ServiceAgreementDAO.java(Compiled Code))

     at com.marsh.csa.serviceagreement.SA= UpdateImpl.updateServiceAgreement(SAUpdateImpl.java(Compiled Code))<= o:p>

     at com.marsh.csa.service= agreement.SAUpdateImpl.update(SAUpdateImpl.java(Compiled Code))=

     ... 26 more

Caused by: com.marsh.framework.core.exception.MarshExcepti= on: Record has been modified since last retrieved - Resubmit transaction

     at com.marsh.csa.se= rviceagreement.SaveCSAAction.performAction(SaveCSAAction.java(Compiled Code= ))

     at com.marsh.c= sa.serviceagreement.CSAAbstractStrutsAction.execute(CSAAbstractStrutsAction= .java(Compiled Code))

   &n= bsp; at org.apache.struts.action.RequestProcessor.processActionPerform(Requ= estProcessor.java(Inlined Compiled Code))

=      at org.apache.struts.action.RequestProcessor.proce= ss(RequestProcessor.java(Compiled Code))

&= nbsp;    at org.apache.struts.action.ActionServlet.process(A= ctionServlet.java(Inlined Compiled Code))

=      at org.apache.struts.action.ActionServlet.doPost(A= ctionServlet.java(Compiled Code))

 &n= bsp;   at javax.servlet.http.HttpServlet.service(HttpServlet.java= (Compiled Code))

     = at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))<= /span>

     at com.ibm.ws.web= container.servlet.ServletWrapper.service(ServletWrapper.java(Compiled Code)= )

 

this is a log fil= e which I am writing to HBase. Whatever is highlighted das yellow I want th= at as one event and gray as another event.

Basically I= want to break the events on Date? Is it possible to do ?

 

 

 

 

 

Regard= s,

Chhaya Vishwakarma

 =

 

The conten= ts of this e-mail and any attachment(s) may contain confidential or privile= ged information for the intended recipient(s). Unintended recipients are pr= ohibited from taking action on the basis of information in this e-mail and = using or disseminating the information, and must notify the sender and dele= te it from their system. L&T Infotech will not accept responsibility or= liability for the accuracy or completeness of, or the presence of any viru= s or disabling code in this e-mail"

<= p class=3DMsoNormal style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:= auto'>

 

<= /div>

--
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog=
My Photo Galleries: http://www.pbase.com/ashishpaliwal

<= /div>



 

--
thanks
ashish

Bl= og: http://= www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpali= wal

 

 



=

 

--

Apache MRUnit - Unit t= esting MapReduce - h= ttp://mrunit.apache.org

= --_000_969B118F8FF0104281DFF098C33475405474F09908PUNINMSMBX01p_--