chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerome Boulon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CHUKWA-155) Job History status arrive out of order causing the status to update incorrectly.
Date Wed, 22 Apr 2009 00:25:47 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701330#action_12701330
] 

Jerome Boulon commented on CHUKWA-155:
--------------------------------------

my 2 cents ...

Or we could have a more generic way:

The problem is that the mysql Job table does not contains columns for more than one state.
What we care about in this table is the final Job state. 
So why not create one additional key JOB_FINAL_STATE that will contain the final Job state,
aka, success, killed or failed.
the "JOB_FINAL_STATE" 's key could be created at the same time as the finish-time parsing
from JobHistory.
For example:
Job JOBID="job_200903310541_1200" FINISH_TIME="1238528943585" JOB_STATUS="SUCCESS" will give
JOB_FINAL_STATE="SUCCESS"


Then all others JOB_STATUS should remain unchanged, aka JOB_STATUS="WHAT_EVER_THE_VALUE_IS",
timestamp will be what ever is available at that time.
This will give us the transition-states table.


> Job History status arrive out of order causing the status to update incorrectly.
> --------------------------------------------------------------------------------
>
>                 Key: CHUKWA-155
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-155
>             Project: Hadoop Chukwa
>          Issue Type: Bug
>          Components: data collection, Data Processors
>         Environment: Redhat 5.1, Java 6
>            Reporter: Eric Yang
>            Assignee: Cheng
>            Priority: Critical
>         Attachments: chukwa-155.patch
>
>
> Job history contains lines like:
> Job JOBID="job_200903310541_1747" JOB_STATUS="RUNNING" .
> ...
> Job JOBID="job_200903310541_1747" FINISH_TIME="1238542231308" JOB_STATUS="SUCCESS" FINISHED_MAPS="1338"
FINISHED_REDUCES="760" FAILED_MAPS="78" FAILED_REDUCES="43" COUNTERS="..." .
> When pushing the data through collectors and demux, the data can arrive out of order.
 The database is updated with status "RUNNING" instead of "SUCCESS".  
> Chukwa Sequence ID can be used to sort out of order data before the data is pumped to
database.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message