Return-Path: X-Original-To: apmail-incubator-chukwa-user-archive@www.apache.org Delivered-To: apmail-incubator-chukwa-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D68E168A5 for ; Mon, 13 Jun 2011 16:38:59 +0000 (UTC) Received: (qmail 29404 invoked by uid 500); 13 Jun 2011 16:38:59 -0000 Delivered-To: apmail-incubator-chukwa-user-archive@incubator.apache.org Received: (qmail 29387 invoked by uid 500); 13 Jun 2011 16:38:59 -0000 Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: chukwa-user@incubator.apache.org Delivered-To: mailing list chukwa-user@incubator.apache.org Received: (qmail 29380 invoked by uid 99); 13 Jun 2011 16:38:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:38:59 +0000 X-ASF-Spam-Status: No, hits=4.6 required=5.0 tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:38:52 +0000 Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35]) by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5DGc7qK056086; Mon, 13 Jun 2011 09:38:07 -0700 (PDT) Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Mon, 13 Jun 2011 09:38:08 -0700 From: Eric Yang To: "chukwa-user@incubator.apache.org" , "billgraham@gmail.com" Date: Mon, 13 Jun 2011 09:38:07 -0700 Subject: Re: How to use TsProcessor ? Thread-Topic: How to use TsProcessor ? Thread-Index: Acwp4odlgGg+X1xfRcy3mAF0ZLi5AAABb6Wu Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CA1B8AFF12E96eyangyahooinccom_" MIME-Version: 1.0 --_000_CA1B8AFF12E96eyangyahooinccom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hbase schema is defined by annotation in demux parsers. For TsProcessor, = it is a generic parser, it is not targeting a specific data type. HBaseWri= ter is currently not handling this generic parser well. The current implementation is writing data processed by TsProcessor to: TableName: TsProcessor ColumnFamily: log It may be possible to add handling of TsProcessor with this specification: Chunk DataType maps to TableName TsProcessor.time.regex.[some_data_type] maps to ColumnFamily If this is reasonable implementation, please file a jira. Thanks Regards, Eric On 6/13/11 8:56 AM, "Bill Graham" wrote: Apologies, but the documentation around processor configs is somewhat out d= ated (CHUKWA-538). For one, TsProcessor is not the default. DefaultProcessor is. You can chang= e this with the chukwa.demux.mapper.default.processor setting. https://issues.apache.org/jira/browse/CHUKWA-473 Also, ithe docs don't include some enhancements to TsProcessor to make it m= ore multi-purpose. It can handle any data type passed to it as long as it c= an parse out the date. See this JIRA for how to override the default or the= per-dataType date parsing logic: https://issues.apache.org/jira/browse/CHUKWA-472 Eric is best to field the HBase schema question. On Mon, Jun 13, 2011 at 1:57 AM, DKN wrote: I am just wondering if TsProcessor is a generic de-mux processor ? Is there= a documentation for what is TsProcessor and if there is a customizable way of using it for generic data types defined ? I wanted to extend a demux processor for log processing and came to know that TsProcessor is default i= n this wiki link : http://wiki.apache.org/hadoop/DemuxModification I also couldn't see this table definition for HBase (in the hbase.schema). Where can I find the schema that I can create the table definitions in HBas= e .. Thanks in advance. Cheers, DKN -- View this message in context: http://apache-chukwa.679492.n3.nabble.com/How= -to-use-TsProcessor-tp3058006p3058006.html Sent from the Chukwa - Users mailing list archive at Nabble.com. --_000_CA1B8AFF12E96eyangyahooinccom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Re: How to use TsProcessor ? Hbase schema is defined by annotation in demux  parsers.  F= or TsProcessor, it is a generic parser, it is not targeting a specific data= type.  HBaseWriter is currently not handling this generic parser well= .
The current implementation is writing data processed by TsProcessor to:

TableName: TsProcessor
ColumnFamily: log

It may be possible to add handling of TsProcessor with this specification:<= BR>
Chunk DataType maps to TableName
TsProcessor.time.regex.[some_data_type] maps to ColumnFamily

If this is reasonable implementation, please file a jira.  Thanks

Regards,
Eric

On 6/13/11 8:56 AM, "Bill Graham" <billgraham@gmail.com> wrote:

Apologies, but the documentation around pro= cessor configs is somewhat out dated (CHUKWA-538).

For one, TsProcessor is not the default. DefaultProcessor is. You can chang= e this with the chukwa.demux.mapper.default.processor setting.

https://issues= .apache.org/jira/browse/CHUKWA-473

Also, ithe docs don't include some enhancements to TsProcessor to make it m= ore multi-purpose. It can handle any data type passed to it as long as it c= an parse out the date. See this JIRA for how to override the default or the= per-dataType date parsing logic:

https://issues= .apache.org/jira/browse/CHUKWA-472


Eric is best to field the HBase schema question.


On Mon, Jun 13, 2011 at 1:57 AM, DKN <= devaprasad@in.ibm.com> wrote:
I am just wondering if TsProcessor is a gen= eric de-mux processor ? Is there a
documentation for what is TsProcessor and if there is a customizable way of=
using it for generic data types defined ? I wanted to extend a demux
processor for log processing and came to know that TsProcessor is default i= n
this wiki link : http://wiki.apache.org/hadoop/DemuxModification

I also couldn't see this table definition for HBase (in the hbase.schema).<= BR> Where can I find the schema that I can create the table definitions in HBas= e
..

Thanks in advance.

Cheers, DKN

--
View this message in context: http://apache-chukwa= .679492.n3.nabble.com/How-to-use-TsProcessor-tp3058006p3058006.html
Sent from the Chukwa - Users mailing list archive at Nabble.com.


--_000_CA1B8AFF12E96eyangyahooinccom_--