sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ruslan Dautkhanov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-3022) sqoop export for Oracle generates tremendous amounts of redo logs
Date Thu, 06 Oct 2016 23:44:21 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553578#comment-15553578

Ruslan Dautkhanov commented on SQOOP-3022:

You're right on APPEND_VALUES hint. Thank you for pointint to that [~maugli]

I just ran a few small tests with this hint on row by row inserts 
and managed to get "ORA-12838: cannot read/modify an object after modifying it in parallel"
on one of inserts. So there is some scalability problems with this hint. 

It may still work if you don't have long-living transactions (i.e. set -Dsqoop.export.statements.per.transaction=1
but tune up well -Dsqoop.export.records.per.statement ?).

On a side note, have you looked at Oracle's INSERT ALL command when you can specify multiple
rows in one INSERT statement?
See for example https://www.techonthenet.com/oracle/questions/insert_rows.php
"INSERT ALL" is supported since at least Oracle 10g - https://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_9014.htm
so it should not new feature, and shouldn't be a compatibility concern. INSERT ALL with APPEND_VALUES
hint may play better in terms of scalability.

> sqoop export for Oracle generates tremendous amounts of redo logs
> -----------------------------------------------------------------
>                 Key: SQOOP-3022
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3022
>             Project: Sqoop
>          Issue Type: Bug
>          Components: codegen, connectors, connectors/oracle
>    Affects Versions: 1.4.3, 1.4.4, 1.4.5, 1.4.6
>            Reporter: Ruslan Dautkhanov
>              Labels: export, oracle
> Sqoop export for Oracle generates tremendous amounts of redo logs (comparable to export
size or more).
> We have put target tables in nologgin mode, but Oracle will still generate redo logs
unless +APPEND Oracle insert hint is used.
> See https://oracle-base.com/articles/misc/append-hint for examples.
> Please add an option for sqoop to generate insert statements in Oracle with APPEND statement.
Our databases are swamped with redo/archived logs whenever we sqoop data to them. This is
easily avoidable. And from business prospective sqooping to staging tables in nologgin mode
is totally fine.
> Thank you.

This message was sent by Atlassian JIRA

View raw message