hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean McNamara (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-2889) LOAD DATA IF NOT EXISTS functionality
Date Wed, 21 Mar 2012 18:19:44 GMT

     [ https://issues.apache.org/jira/browse/HIVE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean McNamara updated HIVE-2889:
--------------------------------

    Description: 
*Background:*
The behavior of LOAD DATA LOCAL INPATH has changed.  It used to give you an error when trying
to copy in a log that already existed.  Now it re-names the file with copy_1 so the file always
goes into hdfs.


*Original discussion:*
http://mail-archives.apache.org/mod_mbox/hive-user/201203.mbox/%3CCB8D2849.14F69%25sean.mcnamara%40webtrends.com%3E


*Issue:*
There is no longer an atomic way to insert files into hive and guarantee that the file won't
go in twice.  Using OVERWRITE will cause other logs in the table/partition to be deleted.


*Example:*
{{usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_a.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"}}
{{/usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_b.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"}}
{{/usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_b.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"}}
{{/usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_b.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"}}

*Result:*
{{test_a.bz2
test_b.bz2
test_b_copy_1.bz2
test_b_copy_2.bz2}}

_test_b data was inserted 3 times, which is not the desired behavior in this instance._


*Proposal:*
Add _IF NOT EXISTS_ flag to indicate copy semantics.  If the the log file does not exist in
the table/partition, the log would go in normally.  If the log does exist in the table/partition
hive would return an error and return an exit code.


*Proposed HiveQL Example:*
{{LOAD DATA LOCAL IF NOT EXISTS INPATH 'test_a.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')}}

  was:
*Background:*
The behavior of LOAD DATA LOCAL INPATH has changed.  It used to give you an error when trying
to copy in a log that already existed.  Now it re-names the file with copy_1 so the file always
goes into hdfs.


*Original discussion:*
http://mail-archives.apache.org/mod_mbox/hive-user/201203.mbox/%3CCB8D2849.14F69%25sean.mcnamara%40webtrends.com%3E


*Issue:*
There is no longer an atomic way to insert files into hive and guarantee that the file won't
go in twice.  Using OVERWRITE will cause other logs in the table/partition to be deleted.


*Example:*
{{usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_a.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"
/usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_b.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"
/usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_b.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"
/usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_b.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"}}

*Result:*
{{test_a.bz2
test_b.bz2
test_b_copy_1.bz2
test_b_copy_2.bz2}}

_test_b data was inserted 3 times, which is not the desired behavior in this instance._


*Proposal:*
Add _IF NOT EXISTS_ flag to indicate copy semantics.  If the the log file does not exist in
the table/partition, the log would go in normally.  If the log does exist in the table/partition
hive would return an error and return an exit code.


*Proposed HiveQL Example:*
{{LOAD DATA LOCAL IF NOT EXISTS INPATH 'test_a.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')}}


    
> LOAD DATA IF NOT EXISTS functionality
> -------------------------------------
>
>                 Key: HIVE-2889
>                 URL: https://issues.apache.org/jira/browse/HIVE-2889
>             Project: Hive
>          Issue Type: Improvement
>          Components: Import/Export
>    Affects Versions: 0.8.1
>            Reporter: Sean McNamara
>             Fix For: 0.9.0
>
>
> *Background:*
> The behavior of LOAD DATA LOCAL INPATH has changed.  It used to give you an error when
trying to copy in a log that already existed.  Now it re-names the file with copy_1 so the
file always goes into hdfs.
> *Original discussion:*
> http://mail-archives.apache.org/mod_mbox/hive-user/201203.mbox/%3CCB8D2849.14F69%25sean.mcnamara%40webtrends.com%3E
> *Issue:*
> There is no longer an atomic way to insert files into hive and guarantee that the file
won't go in twice.  Using OVERWRITE will cause other logs in the table/partition to be deleted.
> *Example:*
> {{usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_a.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"}}
> {{/usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_b.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"}}
> {{/usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_b.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"}}
> {{/usr/local/hive/bin/hive -e "LOAD DATA LOCAL INPATH 'test_b.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')"}}
> *Result:*
> {{test_a.bz2
> test_b.bz2
> test_b_copy_1.bz2
> test_b_copy_2.bz2}}
> _test_b data was inserted 3 times, which is not the desired behavior in this instance._
> *Proposal:*
> Add _IF NOT EXISTS_ flag to indicate copy semantics.  If the the log file does not exist
in the table/partition, the log would go in normally.  If the log does exist in the table/partition
hive would return an error and return an exit code.
> *Proposed HiveQL Example:*
> {{LOAD DATA LOCAL IF NOT EXISTS INPATH 'test_a.bz2' INTO TABLE logs PARTITION(ds='2012-03-19',
hr='23')}}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message