hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-685) Sqoop will fail with OutOfMemory on large tables using mysql
Date Wed, 01 Jul 2009 00:02:47 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725854#action_12725854
] 

Todd Lipcon commented on MAPREDUCE-685:
---------------------------------------

Couple notes:
- The SQL_BIG_RESULT hint I mentioned (offline) was meant for the query that actually returns
lots of rows. If you're doing LIMIT 1 you don't need it.
- Why check against null stmt in execute()? Isn't it assumed that passing null here would
throw an NPE?
- Also, why return null here instead of letting the SQLException fall through?


> Sqoop will fail with OutOfMemory on large tables using mysql
> ------------------------------------------------------------
>
>                 Key: MAPREDUCE-685
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-685
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/sqoop
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: MAPREDUCE-685.patch
>
>
> The default MySQL JDBC client behavior is to buffer the entire ResultSet in the client
before allowing the user to use the ResultSet object. On large SELECTs, this can cause OutOfMemory
exceptions, even when the client intends to close the ResultSet after reading only a few rows.
The MySQL ConnManager should configure its connection to use row-at-a-time delivery of results
to the client.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message