hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "Chukwa Adaptors List" by AriRabkin
Date Thu, 18 Jun 2009 21:42:18 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by AriRabkin:

The comment on the change is:
document adaptors more clearly.

- The following adaptors are built into Chukwa and can be added to the agent using the command
line provided after each. These lines can also be added to the conf/initial_adaptors file
to have agents run adaptors at startup time.
+ In Chukwa, data is collected by adaptors. Adaptors run inside the agent process, and can
be dynamically started and stopped. 
- In order to start an adaptor on an agent, you can telnet into the agent (i.e.  and give
the "ADD" command line to the agent.
+ (mention that commands can go in initial_adaptors or else at control port)
- '''!FileTailer'''
+ The general syntax of the add command is as follows:
- '''!FileTailingAdaptor'''
+ {{{add [name =] <adaptor_class_name> <datatype> <adaptor specific params>
<initial offset>.}}}
+ The `name` field lets you give this adaptor a meaningful name, in order to stop it later.
 If you do not specify a name, Chukwa will autogenerate one by hashing the class name and
parameters.  The adaptor class name specifies what adaptor to run.  The `datatype` specifies
how to process the collected data.  The initial offset should normally be zero; specifying
a nonzero initial offset lets you do advanced things like only processing a portion of a file.
- '''CharFileTailingAdaptorUTF8'''
+ The adaptors built into Chukwa are as follows:
- '''CharFileTailingAdaptorUT!F8NewLineEscaped'''  add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUT!F8NewLineEscaped
SysLog 0 /var/log/messages 0
+ '''!FileAdaptor''':  Push a whole file, as one Chunk, then exit.  Requires file name as
- '''!TerminatorThread'''
+ Example:   {{{add FileTailer }}}
+ '''!filetailer.FileTailingAdaptor''': The 
+   Pushes the file as flat bytes, ignoring the content. Useful for streaming binary data.
+ '''!filetailer.CharFileTailingAdaptorUTF8'''
+  The same, but breaking chunks only at carriage returns.  Useful for most ASCII log file
+ '''!filetailer.CharFileTailingAdaptorUT!F8NewLineEscaped'''  
+  The same, but breaking chunks only at non-escaped carriage returns.  Useful for pushing
Chukwa-formatted log files, where exception stack traces stay in a single chunk.

View raw message