hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5305) AggregatedLogDeletionService may delete recent stuff if its clock is out of sync
Date Wed, 05 Jun 2013 21:05:21 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13676349#comment-13676349
] 

Steve Loughran commented on MAPREDUCE-5305:
-------------------------------------------

I've not seen this happen in Hadoop, but have encountered it in other DFSs, where the FS server's
clock was 100% in sync with the others via NTP, but its time zone was in GMT, not PST. Result:
premature loss of data.

this is precisely why {{ant -diagnostics}} includes an FS offset test -bad time in the FS
can confuse a lot of dependency logic.
{code}

-------------------------------------------
 Temp dir
-------------------------------------------
Temp dir is /var/folders/57/xyts0qt105z1f1k0twk6rd8m0000gq/T/
Temp dir is writeable
Temp dir alignment with system clock is -191 ms

-------------------------------------------
{code}
                
> AggregatedLogDeletionService may delete recent stuff if its clock is out of sync
> --------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5305
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5305
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>    Affects Versions: 3.0.0, 2.1.0-beta
>            Reporter: Steve Loughran
>            Priority: Minor
>
> The {{AggregatedLogDeletionService}} compares file time with current time before deciding
whether to delete things. If the clock on the system is sufficiently wrong, it may overreact.
> If this is felt to be an issue, the fix would be for it to create a file on the DFS,
stat it, and work out out the offset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message