chukwa-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ey...@apache.org
Subject svn commit: r1221953 - in /incubator/chukwa/trunk: CHANGES.txt src/site/apt/collector.apt src/site/site.xml
Date Wed, 21 Dec 2011 23:50:49 GMT
Author: eyang
Date: Wed Dec 21 23:50:48 2011
New Revision: 1221953

URL: http://svn.apache.org/viewvc?rev=1221953&view=rev
Log:
CHUKWA-622. Updated HDFS instruction for collector configuration. (Eric Yang)

Modified:
    incubator/chukwa/trunk/CHANGES.txt
    incubator/chukwa/trunk/src/site/apt/collector.apt
    incubator/chukwa/trunk/src/site/site.xml

Modified: incubator/chukwa/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/CHANGES.txt?rev=1221953&r1=1221952&r2=1221953&view=diff
==============================================================================
--- incubator/chukwa/trunk/CHANGES.txt (original)
+++ incubator/chukwa/trunk/CHANGES.txt Wed Dec 21 23:50:48 2011
@@ -36,6 +36,8 @@ Trunk (unreleased changes)
 
   IMPROVEMENTS
 
+    CHUKWA-622. Updated HDFS instruction for collector configuration. (Eric Yang)
+
     CHUKWA-620. Updated code analysis and license check in build structure. (Eric Yang)
 
     CHUKWA-618. Added HICC programming guide. (Eric Yang)

Modified: incubator/chukwa/trunk/src/site/apt/collector.apt
URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/site/apt/collector.apt?rev=1221953&r1=1221952&r2=1221953&view=diff
==============================================================================
--- incubator/chukwa/trunk/src/site/apt/collector.apt (original)
+++ incubator/chukwa/trunk/src/site/apt/collector.apt Wed Dec 21 23:50:48 2011
@@ -18,9 +18,47 @@ Basic Operation
 
   Chukwa Collectors are responsible for accepting incoming data from Agents,
 and storing the data.  Most commonly, collectors simply write all received 
-to HDFS.  In this mode, the filesystem to write to is determined by the option
+to HBase or HDFS.  
+
+* HBase
+
+  For enabling streaming data to HBase, chukwa collector writer class can
+be configured in <chukwa-collector-conf.xml>.
+
+---
+<property>
+  <name>chukwaCollector.writerClass</name>
+  <value>org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter</value>
+</property>
+---
+
+  In this mode, HBase configuration is configured in <chukwa-env.sh>.
+HBASE_CONF_DIR should reference to HBae configuration directory to enable
+Chukwa Collector to load <hbase-site.xml> from class path.
+
+* HDFS
+
+  For enabling streaming data to HDFS, chukwa collector writer class can
+be configured in <chukwa-collector-conf.xml>.
+
+---
+<property>
+  <name>chukwaCollector.writerClass</name>
+  <value>org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter</value>
+</property>
+---
+
+  In this mode, the filesystem to write to is determined by the option
 <writer.hdfs.filesystem> in <chukwa-collector-conf.xml>.
 
+---
+<property>
+    <name>writer.hdfs.filesystem</name>
+    <value>hdfs://localhost:8020/</value>
+    <description>HDFS to dump to</description>
+</property>
+---
+
   This is the only option that you really need to specify to get a working 
 collector.
 
@@ -41,7 +79,7 @@ the default port number.
 bin/chukwa collector writer=pretend portno=8081
 ---
 
-Advanced options
+Advanced Options
 
   There are some advanced options, not necessarily documented in the
 collector conf file, that are helpful in using Chukwa in nonstandard ways.
@@ -77,7 +115,7 @@ HBaseWriter
 inside for convert unstructured data to semi-structured data, then load the
 key value pairs to HBase table.  HBaseWriter has the following configuration:
 
-  * <hbase.demux.package> Demux parser class package, HBaseWriter uses this 
+  * <<hbase.demux.package>> Demux parser class package, HBaseWriter uses this

     package name to validate HBase for annotated demux parser classes.
 
 ---
@@ -87,7 +125,7 @@ key value pairs to HBase table.  HBaseWr
 </property>
 ---
 
-  * <hbase.writer.verify.schema> Verify HBase Table schema with demux parser 
+  * <<hbase.writer.verify.schema>> Verify HBase Table schema with demux parser

     schema, log warning if there are mismatch between hbase schema and 
     demux parsers.
 
@@ -98,7 +136,7 @@ key value pairs to HBase table.  HBaseWr
 </property>
 ---
 
-  * <hbase.writer.halt.on.schema.mismatch> If this option is set to true, 
+  * <<hbase.writer.halt.on.schema.mismatch>> If this option is set to true, 
     and HBase table schema is mismatched with demux parser, collector will 
     shut down itself.
 
@@ -114,7 +152,7 @@ LocalWriter
   <LocalWriter> writes chunks of data to local disk then upload file to HDFS 
 as a whole file.  This writer is designed for high throughput environment.
 
-  * <chukwaCollector.localOutputDir> Location to buffer data before moving
+  * <<chukwaCollector.localOutputDir>> Location to buffer data before moving
     data to HDFS.
 
 ---
@@ -131,13 +169,37 @@ temp filename with <.chukwa> suffix.  Wh
 the filename is renamed with <.done> suffix.  SeqFileWriter has the following
 configuration in <chukwa-collector-conf.xml>.
 
-  * <writer.hdfs.filesystem> Location to name node address
+  * <<writer.hdfs.filesystem>> Location to name node address
 
-  * <chukwaCollector.outputDir> Location of collect data sink directory
+---
+<property>
+    <name>writer.hdfs.filesystem</name>
+    <value>hdfs://localhost:8020/</value>
+    <description>HDFS to dump to</description>
+</property>
+---
+
+  * <<chukwaCollector.outputDir>> Location of collect data sink directory
+
+---
+<property>
+    <name>chukwaCollector.outputDir</name>
+    <value>/chukwa/logs/</value>
+    <description>Chukwa data sink directory</description>
+</property>
+---
 
-  * <chukwaCollector.rotateInterval> File Rotation Interval
+  * <<chukwaCollector.rotateInterval>> File Rotation Interval
 
-  * <chukwaCollector.isFixedTimeRotatorScheme> A flag to indicate that the 
+---
+<property>
+    <name>chukwaCollector.rotateInterval</name>
+    <value>300000</value>
+    <description>Chukwa rotate interval (ms)</description>
+</property>
+---
+
+  * <<chukwaCollector.isFixedTimeRotatorScheme>> A flag to indicate that the

     collector should close at a fixed offset after every rotateInterval. 
     The default value is false which uses the default scheme where 
     collectors close after regular rotateIntervals.
@@ -148,9 +210,24 @@ configuration in <chukwa-collector-conf.
     isFixedTimeRotatorScheme is false, collectors will rotate approximately
     once every 5 minutes
 
-  * <chukwaCollector.fixedTimeIntervalOffset> Chukwa fixed time interval 
+---
+  <property>
+    <name>chukwaCollector.isFixedTimeRotatorScheme</name>
+    <value>false</value>
+  </property>
+---
+
+  * <<chukwaCollector.fixedTimeIntervalOffset>> Chukwa fixed time interval 
     offset value (ms)
 
+---
+<property>
+    <name>chukwaCollector.fixedTimeIntervalOffset</name>
+    <value>30000</value>
+    <description>Chukwa fixed time interval offset value (ms)</description>
+</property>
+---
+
 SocketTeeWriter
 
   The <SocketTeeWriter> allows external processes to watch

Modified: incubator/chukwa/trunk/src/site/site.xml
URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/site/site.xml?rev=1221953&r1=1221952&r2=1221953&view=diff
==============================================================================
--- incubator/chukwa/trunk/src/site/site.xml (original)
+++ incubator/chukwa/trunk/src/site/site.xml Wed Dec 21 23:50:48 2011
@@ -48,7 +48,7 @@
       <item name="Overview" href="index.html"/>
       <item name="Quick Start Guide" href="Quick_Start_Guide.html"/>
       <item name="HICC User Guide" href="hicc.html"/>
-      <item name="Configuration" href="admin.html">
+      <item name="Administration Guide" href="admin.html">
         <item name="Agent" href="agent.html"/>
         <item name="Collector" href="collector.html"/>
       </item>



Mime
View raw message