Return-Path: X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E132010423 for ; Mon, 9 Feb 2015 23:02:39 +0000 (UTC) Received: (qmail 40145 invoked by uid 500); 9 Feb 2015 23:02:39 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 40073 invoked by uid 500); 9 Feb 2015 23:02:39 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 40061 invoked by uid 99); 9 Feb 2015 23:02:39 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Feb 2015 23:02:39 +0000 Date: Mon, 9 Feb 2015 23:02:39 +0000 (UTC) From: "Hudson (JIRA)" To: mapreduce-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (MAPREDUCE-6237) Multiple mappers with DBInputFormat don't work because of reusing conections MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313096#comment-14313096 ] Hudson commented on MAPREDUCE-6237: ----------------------------------- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #100 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/100/]) MAPREDUCE-6237. Multiple mappers with DBInputFormat don't work because of reusing conections. Contributed by Kannan Rajah. (ozawa: rev 241336ca2b7cf97d7e0bd84dbe0542b72f304dc9) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/OracleDataDrivenDBInputFormat.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DataDrivenDBInputFormat.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java > Multiple mappers with DBInputFormat don't work because of reusing conections > ---------------------------------------------------------------------------- > > Key: MAPREDUCE-6237 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6237 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 > Affects Versions: 2.5.0, 2.6.0 > Reporter: Kannan Rajah > Assignee: Kannan Rajah > Fix For: 2.6.1 > > Attachments: mapreduce-6237.patch, mapreduce-6237.patch, mapreduce-6237.patch > > > DBInputFormat.createDBRecorder is reusing JDBC connections across instances of DBRecordReader. This is not a good idea. We should be creating separate connection. If performance is a concern, then we should be using connection pooling instead. > I looked at DBOutputFormat.getRecordReader. It actually creates a new Connection object for each DBRecordReader. So can we just change DBInputFormat to create new Connection every time? The connection reuse code was added as part of connection leak bug in MAPREDUCE-1443. Any reason for caching the connection? > We observed this issue in a customer setup where they were reading data from MySQL using Pig. As per customer, the query is returning two records which causes Pig to create two instances of DBRecordReader. These two instances are sharing the database connection instance. The first DBRecordReader runs to extract the first record from MySQL just fine, but then closes the shared connection instance. When the second DBRecordReader runs, it tries to execute a query to retrieve the second record on the closed shared connection instance, which fail. If we set > mapred.map.tasks to 1, the query will be successful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)