chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerome Boulon (JIRA)" <>
Subject [jira] Created: (CHUKWA-203) Track data loading from agent
Date Thu, 30 Apr 2009 16:30:30 GMT
Track data loading from agent

                 Key: CHUKWA-203
             Project: Hadoop Chukwa
          Issue Type: New Feature
          Components: data collection, Data Processors
            Reporter: Jerome Boulon
            Priority: Critical

Chukwa needs to track progress on all files for completeness reason. 
The  first step could be to send adaptor information to the backend for postprocess/storage.

This could be done at the same time of the writing checkpoint file by building a chunk and
post it to the queue.
In addition to that, we need to track all Add/Remove operations and the final offset for all
files, the easiest way to do this will be to generate this information at the beginning and
the end of each adaptor.

Based on that, we should be able to:
- track any file from the add to the remove, 
- validate that all data has been sent 
- track all files' rotation.
- record any permission issue (expiration policy)
- generate alerts


This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message