incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Re: What to do after database failures?
Date Fri, 23 Jul 2010 20:51:47 GMT
On 7/23/10 1:17 PM, "William Bajzek" <williambajzek@gmail.com> wrote:

> On Jul 23, 2010, at 9:58 AM, Eric Yang wrote:
>> MetricDataLoader can be modified to throw IOException to the executor class
>> MetricDataLoaderPool, which can throw exception to PostProcessor Manager.
>> PostProcessorManager moves the data to a temp directory.  The retry logic
>> can be added to PostProcessorManager by counting the number of retry with
>> the error out sequence file before sending it to InError directory.  It
>> should be the better route to manages error conditions.
> 
> I'll look into this, thanks. Other than that, is the only recourse to move the
> failed material back into the queue manually?

Yes.  This page contains some useful information on the data flow:

http://wiki.apache.org/hadoop/Chukwa_Processes_and_Data_Flow

In 3.3, it moves from demuxProcessing to postProcess directory.  If it
fails, move the data to demuxProcessing directory again and
PostProcessorManager will pick up the directory again and attempt to load
again.

Regards,
Eric


Mime
View raw message