Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Mon, 18 Apr 2016 22:34:25 +0000 (UTC)
From: "Junping Du (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12959652.1460988346000.261265.1461018865658@Atlassian.JIRA>
In-Reply-To: <JIRA.12959652.1460988346000@Atlassian.JIRA>
References: <JIRA.12959652.1460988346000@Atlassian.JIRA>
 <JIRA.12959652.1460988346197@arcas>
Subject: [jira] [Updated] (MAPREDUCE-6680) JHS UserLogDir scan algorithm
 sometime could skip directory with update in CloudFS (Azure FileSystem, S3,
 etc.)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/MAPREDUCE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Junping Du updated MAPREDUCE-6680:
----------------------------------
    Attachment: MAPREDUCE-6680-v3.patch

> JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6680
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>            Reporter: Junping Du
>            Assignee: Junping Du
>         Attachments: MAPREDUCE-6680-v2.patch, MAPREDUCE-6680-v3.patch, MAPREDUCE-6680.patch
>
>
> In our cluster based on a Cloud FileSystem, we notice JHS sometimes could skip directory with .jhist file in scanning.
> The behavior is like:
> First round scan, doesn't found .jhist file:
> {noformat}
> 16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a directory with 6 files in it.
> 16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
> ...
> {noformat}
> Then, we see "Scan not needed of ..." for the same directory every 3 minutes until application failed as timeout.
> From our analysis, we found the root cause is: most of Cloud File System (Azure FS, S3, etc.) is truncating file/directory modification time to seconds instead of milliseconds - which could due to limit of http protocol (from discussion at: https://forums.aws.amazon.com/thread.jspa?messageID=476615). 
> So if the time sequence is happen to be: latest non .jhist file modification on directory happens at T1, directory scanning happens at T2, .jhist file added to directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3 after truncating to seconds, this issue could appear.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)