chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CHUKWA-734) Gora Storage System for Chuckwa Logs
Date Sat, 14 Mar 2015 21:37:38 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362042#comment-14362042
] 

Eric Yang commented on CHUKWA-734:
----------------------------------

Thanks Lewis,

A few suggestions, 

1. Need to include hbase-client for test case to pass:

                 <dependency>
                     <groupId>org.apache.hbase</groupId>
                    <artifactId>hbase-client</artifactId>
                    <version>${hbase.version}</version>
                </dependency>


2. gora.properties is better hosted in conf directory instead of src/main/resources.  This
allow user to configure it during deployment time instead of hardcode into jar file.

3. We may want to generate two gora.properties, one for test case, and one for release.  The
one with test case can run with in memory database to reduce test running time.  The production
one is preconfigured with hbase to make it easier for new comer to adopt this solution.

4. We probably want to have a developer guide for GoraWriter.  It is really powerful stuff
to enrich Chukwa's capability to write to different storage system.  Tutorial could help new
developers.

5. I encountered a issue when I configure gora.properties to write to HBase from chukwa agent.
 I get this error:

2015-03-14 14:12:59.451 java[11075:636025] Unable to load realm info from SCDynamicStore
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HTableDescriptor.addFamily(Lorg/apache/hadoop/hbase/HColumnDescriptor;)V
	at org.apache.gora.hbase.store.HBaseMapping$HBaseMappingBuilder.build(HBaseMapping.java:174)
	at org.apache.gora.hbase.store.HBaseStore.readMapping(HBaseStore.java:811)
	at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:116)
	at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:101)
	at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:160)
	at org.apache.gora.store.DataStoreFactory.getDataStore(DataStoreFactory.java:277)
	at org.apache.hadoop.chukwa.datacollection.writer.gora.GoraWriter.init(GoraWriter.java:67)
	at org.apache.hadoop.chukwa.datacollection.writer.gora.GoraWriter.<init>(GoraWriter.java:53)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at java.lang.Class.newInstance(Class.java:374)
	at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:100)
	at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.<init>(PipelineStageWriter.java:48)
	at org.apache.hadoop.chukwa.datacollection.connector.PipelineConnector.start(PipelineConnector.java:87)
	at org.apache.hadoop.chukwa.datacollection.agent.ChukwaAgent.main(ChukwaAgent.java:292)

This is what I added to gora.properties:

gora.datastore.default=org.apache.gora.hbase.store.HBaseStore
gora.datastore.autocreateschema=true

I am not sure if the last error was caused by the default Chukwa agent attempting to write
to system metrics into HBase using gora.  I am not sure if this would cause problem.  This
bring an interesting question on how we want to configure data type map to writers.

> Gora Storage System for Chuckwa Logs
> ------------------------------------
>
>                 Key: CHUKWA-734
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-734
>             Project: Chukwa
>          Issue Type: New Feature
>          Components: Data Collection
>    Affects Versions: 0.6.0
>            Reporter: Lewis John McGibbney
>             Fix For: 0.7.0
>
>         Attachments: CHUKWA-734.patch, CHUKWA-734v2.patch
>
>   Original Estimate: 5h
>  Remaining Estimate: 5h
>
> I would like to build a Gora-backed log-to-datastore module for Chuckwa. I am going to
work on this today.
> Gora is an in-memory data modeling and storage abstraction 
> http://gora.apache.org
> Gora powers the Apache Nutch 2.X software which generates a bunch of log data. Having
a Chuckwa monitoring tool for Nutch would be grand.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message