hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-685) Sqoop will fail with OutOfMemory on large tables using mysql
Date Tue, 30 Jun 2009 21:20:47 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725812#action_12725812
] 

Aaron Kimball commented on MAPREDUCE-685:
-----------------------------------------

Added patch that fixes this issue. Also includes some other performance enhancements:

* MySQL now uses "LIMIT 1" when making SELECTs against tables for metadata-reading purposes.
* Transactions are no longer opened with TRANSACTION_SERIALIZABLE since it's unnecessary for
metadata reads

No new tests for this included since Hadoop testing doesn't mesh well with MySQL. I tested
locally by building a 1.7 GB table in mysql and reading into a local HDFS instance. This failed
before applying the patch, and succeeds afterwards.

> Sqoop will fail with OutOfMemory on large tables using mysql
> ------------------------------------------------------------
>
>                 Key: MAPREDUCE-685
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-685
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/sqoop
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: MAPREDUCE-685.patch
>
>
> The default MySQL JDBC client behavior is to buffer the entire ResultSet in the client
before allowing the user to use the ResultSet object. On large SELECTs, this can cause OutOfMemory
exceptions, even when the client intends to close the ResultSet after reading only a few rows.
The MySQL ConnManager should configure its connection to use row-at-a-time delivery of results
to the client.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message