hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6680) JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)
Date Mon, 18 Apr 2016 22:34:25 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15246699#comment-15246699

Junping Du commented on MAPREDUCE-6680:

bq.  we can record latest scanning time on userDir as T2, and we scan the list of directory
in case: (T1 != T3) or (T1 = T3 but T1.toSeconds = T2.toSeconds), that can get rid of skip
problem and not involve many unnecessary scan - given the chance file update happen at the
same second with scan time is very low (scan interval is default to be 3 minutes).
Another possibility is T1 (X second Y millisecond) and T3 (X second Z millisecond) is cast
to second (X+1) in FS. To address this, we need to check (T1 != T3) or (T1 == T3 but T1.toSeconds
>= T2.toSeconds). Updated this in v3 patch.

> JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure
FileSystem, S3, etc.)
> ---------------------------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-6680
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>            Reporter: Junping Du
>            Assignee: Junping Du
>         Attachments: MAPREDUCE-6680-v2.patch, MAPREDUCE-6680-v3.patch, MAPREDUCE-6680.patch
> In our cluster based on a Cloud FileSystem, we notice JHS sometimes could skip directory
with .jhist file in scanning.
> The behavior is like:
> First round scan, doesn't found .jhist file:
> {noformat}
> 16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a directory with 6
files in it.
> 16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
> ...
> {noformat}
> Then, we see "Scan not needed of ..." for the same directory every 3 minutes until application
failed as timeout.
> From our analysis, we found the root cause is: most of Cloud File System (Azure FS, S3,
etc.) is truncating file/directory modification time to seconds instead of milliseconds -
which could due to limit of http protocol (from discussion at: https://forums.aws.amazon.com/thread.jspa?messageID=476615).

> So if the time sequence is happen to be: latest non .jhist file modification on directory
happens at T1, directory scanning happens at T2, .jhist file added to directory at T3. If
we have {{T1< T2 < T3}} and T1 is equal to T3 after truncating to seconds, this issue
could appear.

This message was sent by Atlassian JIRA

View raw message