Return-Path: X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 036C4118E5 for ; Tue, 19 Aug 2014 22:25:21 +0000 (UTC) Received: (qmail 34869 invoked by uid 500); 19 Aug 2014 22:25:20 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 34815 invoked by uid 500); 19 Aug 2014 22:25:20 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 34800 invoked by uid 99); 19 Aug 2014 22:25:20 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Aug 2014 22:25:20 +0000 Date: Tue, 19 Aug 2014 22:25:20 +0000 (UTC) From: "Siqi Li (JIRA)" To: mapreduce-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-4815: ------------------------------- Attachment: MAPREDUCE-4815.v4.patch > FileOutputCommitter.commitJob can be very slow for jobs with many output files > ------------------------------------------------------------------------------ > > Key: MAPREDUCE-4815 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 > Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 > Reporter: Jason Lowe > Assignee: Siqi Li > Attachments: MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch > > > If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.2#6252)