Return-Path: X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C525E18F84 for ; Fri, 4 Mar 2016 06:00:46 +0000 (UTC) Received: (qmail 6293 invoked by uid 500); 4 Mar 2016 06:00:46 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 6229 invoked by uid 500); 4 Mar 2016 06:00:46 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 6217 invoked by uid 99); 4 Mar 2016 06:00:46 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2016 06:00:46 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 3C5AC2C1F6B for ; Fri, 4 Mar 2016 06:00:46 +0000 (UTC) Date: Fri, 4 Mar 2016 06:00:46 +0000 (UTC) From: "Tsuyoshi Ozawa (JIRA)" To: mapreduce-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (MAPREDUCE-4522) DBOutputFormat Times out on large batch inserts MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-4522: -------------------------------------- Assignee: Shyam Gavulla (was: Nathan Jarus) > DBOutputFormat Times out on large batch inserts > ----------------------------------------------- > > Key: MAPREDUCE-4522 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4522 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller > Affects Versions: 0.20.205.0 > Reporter: Nathan Jarus > Assignee: Shyam Gavulla > Labels: newbie > > In DBRecordWriter#close(), progress is never updated. In large batch inserts, this can cause the reduce task to time out due to the amount of time it takes the SQL engine to process that insert. > Potential solutions I can see: > Don't batch inserts; do the insert when DBRecordWriter#write() is called (awful) > Spin up a thread in DBRecordWriter#close() and update progress in that. (gross) > I can provide code for either if you're interested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)