Mailing-List: contact chukwa-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-dev@incubator.apache.org
Message-ID: <1171194.67621292025540790.JavaMail.jira@thor>
Date: Fri, 10 Dec 2010 18:59:00 -0500 (EST)
From: "Bill Graham (JIRA)" <jira@apache.org>
To: chukwa-dev@incubator.apache.org
Subject: [jira] Commented: (CHUKWA-564) HBase output collector uses
 incorrect column family
In-Reply-To: <24998714.48261291936140740.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CHUKWA-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970345#action_12970345 ] 

Bill Graham commented on CHUKWA-564:
------------------------------------

I agree that there are limitations in using annotations on the processors. I think that where the data is written should be decoupled from the processors. A processor knows how to process data, but it shouldn't also state where the data should be written. Generic processors like TsProcessors could be used repeatedly for different data types, all of which should be written to different table/column-families. Coupling the two with annotations makes this difficult. You end up with empty subclasses used only to configure different data types to table/cfs via overridden annotations.

I suggest we externalize the table/cf mappings from the processors. Instead we could have something like an HBaseRouterFactory (or something perhaps named better) that the OutputCollector and the HBaseWriter interact with. HBaseRouterFactory has a method that takes in a dataType and probably also a ChukwaRecord and knows how to return the Table and ColumnFamily that the data should be written too. 

We could then configure that dataType 'foo' should use BarProcessor and write to table 'bat', column family 'biz'.

I don't know how we'd configure 'foo's payload to be written to multiple cfs though. What's the use case for why we'd want to write the same data to two locations?

There's still an unresolved separate problem of how to handle ORM-ish functionality as well, since reduxing the many parameters in the record body back to a single 'body' field can be sub-optimal.

> HBase output collector uses incorrect column family
> ---------------------------------------------------
>
>                 Key: CHUKWA-564
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-564
>             Project: Chukwa
>          Issue Type: Bug
>            Reporter: Bill Graham
>             Fix For: 0.5.0
>
>
> The HBase {{OutputCollector}} does this to obtain the column family from the data type:
> {noformat}
> cf = key.getReduceType().getBytes();
> {noformat}
> The column family should instead be taken by the {{@Table.columnFamily}} annotation on the processor.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.