chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerome Boulon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CHUKWA-102) Provide streaming adapter
Date Tue, 04 Aug 2009 20:09:14 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739128#action_12739128
] 

Jerome Boulon commented on CHUKWA-102:
--------------------------------------

The initial idea when Mac creates this Jira was to stream data from an application to the
local agent but my proposal is to send directly to the collector and scale horizontally if
possible then if you can no longer scale horizontally (too many files on HDFS for example)
then scale by creating a DAG of collector (similar to what scribe is doing).

So in my proposal there's no need for an Agent, the agent could be a simple as a cronjob that
read a file and stream it over to a remote collector.
I may put some logic inside the log4j Appender to fall back to disk in case of network issue
or to have the same kind of checkpoint directly at the log4j appender. But I'm still working
on the design.



> Provide streaming adapter
> -------------------------
>
>                 Key: CHUKWA-102
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-102
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: input tools
>            Reporter: Mac Yang
>
> Enable applications to send data directly to the agent without writing to disk first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message