hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6237) DBRecordReader is not thread safe
Date Thu, 05 Feb 2015 19:26:34 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307813#comment-14307813

Hadoop QA commented on MAPREDUCE-6237:

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  against trunk revision d27439f.

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

      {color:red}-1 javac{color}.  The applied patch generated 1154 javac compiler warnings
(more than the trunk's current 1149 warnings).

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:red}-1 findbugs{color}.  The patch appears to introduce 13 new Findbugs (version
2.0.3) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5162//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5162//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5162//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5162//console

This message is automatically generated.

> DBRecordReader is not thread safe
> ---------------------------------
>                 Key: MAPREDUCE-6237
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6237
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 2.5.0
>            Reporter: Kannan Rajah
>            Assignee: Kannan Rajah
>         Attachments: mapreduce-6237.patch, mapreduce-6237.patch, mapreduce-6237.patch
> DBInputFormat.createDBRecorder is reusing JDBC connections across instances of DBRecordReader.
This is not a good idea. We should be creating separate connection. If performance is a concern,
then we should be using connection pooling instead.
> I looked at DBOutputFormat.getRecordReader. It actually creates a new Connection object
for each DBRecordReader. So can we just change DBInputFormat to create new Connection every
time? The connection reuse code was added as part of connection leak bug in MAPREDUCE-1443.
Any reason for caching the connection?
> We observed this issue in a customer setup where they were reading data from MySQL using
Pig. As per customer, the query is returning two records which causes Pig to create two instances
of DBRecordReader. These two instances are sharing the database connection instance. The first
DBRecordReader runs to extract the first record from MySQL just fine, but then closes the
shared connection instance. When the second DBRecordReader runs, it tries to execute a query
to retrieve the second record on the closed shared connection instance, which fail. If we
> mapred.map.tasks to 1, the query will be successful.

This message was sent by Atlassian JIRA

View raw message