hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-685) Sqoop will fail with OutOfMemory on large tables using mysql
Date Wed, 12 Aug 2009 16:09:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742426#action_12742426
] 

Aaron Kimball commented on MAPREDUCE-685:
-----------------------------------------

Because it's not actually the same fix ;) Postgresql wants you to do {{statement.setFetchSize(something_reasonable)}}
e.g., 40.

MySQL wants you to do {{statement.setFetchSize(INT_MIN)}}. The only cursor modes MySQL supports
are fully buffered (fetch size = 0) and fully row-wise cursors (fetch_size = INT_MIN).

That having been said, I have just finished a postgresql patch ready to post up here this
week :) Just waiting for some existing patches to get committed first so that it applies cleanly.

> Sqoop will fail with OutOfMemory on large tables using mysql
> ------------------------------------------------------------
>
>                 Key: MAPREDUCE-685
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-685
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/sqoop
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>             Fix For: 0.21.0
>
>         Attachments: MAPREDUCE-685.3.patch, MAPREDUCE-685.patch, MAPREDUCE-685.patch.2
>
>
> The default MySQL JDBC client behavior is to buffer the entire ResultSet in the client
before allowing the user to use the ResultSet object. On large SELECTs, this can cause OutOfMemory
exceptions, even when the client intends to close the ResultSet after reading only a few rows.
The MySQL ConnManager should configure its connection to use row-at-a-time delivery of results
to the client.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message