hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5305) AggregatedLogDeletionService may delete recent stuff if its clock is out of sync
Date Wed, 05 Jun 2013 21:05:21 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13676349#comment-13676349

Steve Loughran commented on MAPREDUCE-5305:

I've not seen this happen in Hadoop, but have encountered it in other DFSs, where the FS server's
clock was 100% in sync with the others via NTP, but its time zone was in GMT, not PST. Result:
premature loss of data.

this is precisely why {{ant -diagnostics}} includes an FS offset test -bad time in the FS
can confuse a lot of dependency logic.

 Temp dir
Temp dir is /var/folders/57/xyts0qt105z1f1k0twk6rd8m0000gq/T/
Temp dir is writeable
Temp dir alignment with system clock is -191 ms

> AggregatedLogDeletionService may delete recent stuff if its clock is out of sync
> --------------------------------------------------------------------------------
>                 Key: MAPREDUCE-5305
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5305
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>    Affects Versions: 3.0.0, 2.1.0-beta
>            Reporter: Steve Loughran
>            Priority: Minor
> The {{AggregatedLogDeletionService}} compares file time with current time before deciding
whether to delete things. If the clock on the system is sufficiently wrong, it may overreact.
> If this is felt to be an issue, the fix would be for it to create a file on the DFS,
stat it, and work out out the offset.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message