chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shreyas subramanya (JIRA)" <>
Subject [jira] [Commented] (CHUKWA-700) Revisit Chukwa metrics schema design for HBase
Date Fri, 08 Nov 2013 02:16:18 GMT


shreyas subramanya commented on CHUKWA-700:

I think we need to provide support for multiple custom schema, since the performance would
depend on the actual scan scenario (as observed in Chukwa-667). Just like we can configure
the hbase table and column family in demux processor, we need to provide annotation for OutputCollector,
so that the demux processor can dictate the hbase schema.

> Revisit Chukwa metrics schema design for HBase
> ----------------------------------------------
>                 Key: CHUKWA-700
>                 URL:
>             Project: Chukwa
>          Issue Type: Bug
>          Components: Data Collection
>    Affects Versions: 0.6.0
>         Environment: MacOSX, Java
>            Reporter: Eric Yang
> Current Chukwa HBase schema looks like this:
> {code}
> <timestamp>-<primaryKey>   <columnFamily>:<cell>...
> {code}
> Monotonic increasing timestamp can not evenly distribute across region servers without
special handle and care periodically.
> It is time to revise the schema, and proposed schema looks like this:
> {code}
> <hhddmmyyyy>-<primaryId>  cf:<cell>...
> {code}
> Timestamp is stored with cell, row key helps to split data by hour, and a full hour of
metrics is stored on the same row.  PrimaryKey is replaced with hash id of the primary key.
 Metrics tables to aggregate metrics:
> chukwaMetrics -> chukwaMetricsMonthly -> chukwaMetricsYearly

This message was sent by Atlassian JIRA

View raw message