chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerome Boulon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CHUKWA-26) Rewrite processSinkFiles.sh in java to have a better error handling
Date Tue, 07 Apr 2009 16:32:13 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696605#action_12696605
] 

Jerome Boulon commented on CHUKWA-26:
-------------------------------------

Since we still don't have a wiki for Chukwa, I'll put more information here.
Corinne will work on documented this, like all the Chukwa  documentation.

All new daemon are responsible for taking data from the previous step and producing data for
the next one.
Each one running asynchronously from the others

- Collector-> DataSink  (input for DemuxManager)

- DemuxManager
   |-> Demux output (ChukwaRecord, input for PostProcessorManager)
   |-> move dataSink file to dataSinkArchive directory

- PostProcessorManager
  |-> consume demux output, load to database
  |-> move ChukwaRecord to /chukwa/repos/...

- ArchiveManager
  |-> every 2 hours compact dataSink files

- HourlyRolling
  |-> same as before except a fileName change, the filname now contains "HourlyDone" so
I can guarantee that the Hourly was done

- DailyRolling
  |-> same as before except that we are now waiting for hourlyRolling to be done before
processing a day


>>What does DemuxManager do?
DemuxManager is a daemon process. 
It takes care of scheduling Demux on DataSink files, limit the number of input file to demux,
force a reprocess of any dataSink files that were part of the previous demux if DemuxManager
has been killed and after 3 attempts to process the same list of DataSink files, DemuxManager
automatically move those faulty dataSink file to an Error directory

>>What does PostProcessorManager do?
Load all demuxOutput to DB

>>Do I need Nagios?
-No, if you're not adding your nagios information to chukwa-demux-conf.xml, DemuxManager will
not send anything to Nagios

>>or I think I have it but it's down
-Nothing, DemuxManager will try to send an NSCA command via a socket connection, this command
has no impact on DemuxManager.



> Rewrite processSinkFiles.sh in java to have a better error handling
> -------------------------------------------------------------------
>
>                 Key: CHUKWA-26
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-26
>             Project: Hadoop Chukwa
>          Issue Type: Improvement
>          Components: Data Processors
>            Reporter: Jerome Boulon
>            Assignee: Jerome Boulon
>            Priority: Blocker
>         Attachments: CHUKWA-26-2.patch, CHUKWA-26.patch, NagiosAppender-1.5.0.jar
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message