chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shreyas subramanya (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CHUKWA-679) Create WebHDFSWriter
Date Wed, 16 Jan 2013 03:04:12 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554689#comment-13554689
] 

shreyas subramanya commented on CHUKWA-679:
-------------------------------------------

The idea is to append to text files (named based on chunk metadata) through webhdfs. The append
requires 2 post operations - the latter being a url to datanode. Since we expect small updates
to the file (like data from log4j appender), we want to cache the redirect url and keep posting
to the same datanode until the configured "rotate interval" (like 1 hour). Then we create
a new file, get a new datanode url to post to and continue.
                
> Create WebHDFSWriter
> --------------------
>
>                 Key: CHUKWA-679
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-679
>             Project: Chukwa
>          Issue Type: Sub-task
>          Components: Data Processors
>    Affects Versions: 0.5.0, 0.6.0
>         Environment: Java 6
>            Reporter: shreyas subramanya
>   Original Estimate: 10h
>  Remaining Estimate: 10h
>
> Create a pipelineable writer which uses WebHDFS REST api.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message