chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <>
Subject [jira] Commented: (CHUKWA-102) Provide streaming adapter
Date Mon, 30 Nov 2009 01:41:20 GMT


Eric Yang commented on CHUKWA-102:

Resurrecting this jira.  The components to make this feature useful are:

Streaming adaptor - Listen to a port, and receive chunks, and send directly to collector,
spill to disk if it could not send the chunk.
Streaming appender - Chunk creator which can configure to send to an agent or an collector.
 There is no fail safe buffering in the appender.

Streaming appender could send to collector directly for small clusters to avoid agent setup.
 For large cluster, it should be configured to send to agent then collector.  These utilities
support dag configuration and avoid tcp incast problem.  Streaming appender should communicate
to streaming adaptor through local loopback when they are deployed on the same system.  This
provides best effort log delivery system with minimum effort for implementation.

> Provide streaming adapter
> -------------------------
>                 Key: CHUKWA-102
>                 URL:
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: input tools
>            Reporter: Mac Yang
> Enable applications to send data directly to the agent without writing to disk first.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message