hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Wong (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-3104) Predicate pushdown doesn't work with multi-insert statements using LATERAL VIEW
Date Thu, 29 Aug 2013 21:59:52 GMT

     [ https://issues.apache.org/jira/browse/HIVE-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steven Wong updated HIVE-3104:
------------------------------

    Description: 
Predicate pushdown seems to work for single-insert queries using LATERAL VIEW. It also seems
to work for multi-insert queries *not* using LATERAL VIEW. However, it doesn't work for multi-insert
queries using LATERAL VIEW: It errors out right away with 'FAILED: SemanticException [Error
10041]: No partition predicate found for Alias "test" Table "test"'.

Here are some examples. In the below examples, I make use of the fact that a query with no
partition filtering when run under "hive.mapred.mode=strict" fails.

--Table creation and population
DROP TABLE IF EXISTS test;
CREATE TABLE test (col1 array<int>, col2 int)  PARTITIONED BY (part_col int);
INSERT OVERWRITE TABLE test PARTITION (part_col=1) SELECT array(1,2), count(*) FROM test;
INSERT OVERWRITE TABLE test PARTITION (part_col=2) SELECT array(2,4,6), count(*) FROM test;

-- Query 1
-- This succeeds (using LATERAL VIEW with single insert)
set hive.mapred.mode=strict;
FROM test
LATERAL VIEW explode(col1) tmp AS exp_col1
INSERT OVERWRITE DIRECTORY '/test/1'
    SELECT exp_col1
    WHERE (part_col=2);

-- Query 2
-- This succeeds (NOT using LATERAL VIEW with multi-insert)
set hive.mapred.mode=strict;
FROM test
INSERT OVERWRITE DIRECTORY '/test/1'
    SELECT col1
    WHERE (part_col=2)
INSERT OVERWRITE DIRECTORY '/test/2'
    SELECT col1
    WHERE (part_col=2);

-- Query 3
-- This fails (using LATERAL VIEW with multi-insert)
set hive.mapred.mode=strict;
FROM test
LATERAL VIEW explode(col1) tmp AS exp_col1
INSERT OVERWRITE DIRECTORY '/test/1'
    SELECT exp_col1
    WHERE (part_col=2)
INSERT OVERWRITE DIRECTORY '/test/2'
    SELECT exp_col1
    WHERE (part_col=2);


  was:
Predicate pushdown seems to work for single-insert queries using LATERAL VIEW. It also seems
to work for multi-insert queries *not* using LATERAL VIEW. However, it doesn't work for multi-insert
queries using LATERAL VIEW.

Here are some examples. In the below examples, I make use of the fact that a query with no
partition filtering when run under "hive.mapred.mode=strict" fails.

--Table creation and population
DROP TABLE IF EXISTS test;
CREATE TABLE test (col1 array<int>, col2 int)  PARTITIONED BY (part_col int);
INSERT OVERWRITE TABLE test PARTITION (part_col=1) SELECT array(1,2), count(*) FROM test;
INSERT OVERWRITE TABLE test PARTITION (part_col=2) SELECT array(2,4,6), count(*) FROM test;

-- Query 1
-- This succeeds (using LATERAL VIEW with single insert)
set hive.mapred.mode=strict;
FROM test
LATERAL VIEW explode(col1) tmp AS exp_col1
INSERT OVERWRITE DIRECTORY '/test/1'
    SELECT exp_col1
    WHERE (part_col=2);

-- Query 2
-- This succeeds (NOT using LATERAL VIEW with multi-insert)
set hive.mapred.mode=strict;
FROM test
INSERT OVERWRITE DIRECTORY '/test/1'
    SELECT col1
    WHERE (part_col=2)
INSERT OVERWRITE DIRECTORY '/test/2'
    SELECT col1
    WHERE (part_col=2);

-- Query 3
-- This fails (using LATERAL VIEW with multi-insert)
set hive.mapred.mode=strict;
FROM test
LATERAL VIEW explode(col1) tmp AS exp_col1
INSERT OVERWRITE DIRECTORY '/test/1'
    SELECT exp_col1
    WHERE (part_col=2)
INSERT OVERWRITE DIRECTORY '/test/2'
    SELECT exp_col1
    WHERE (part_col=2);


    
> Predicate pushdown doesn't work with multi-insert statements using LATERAL VIEW
> -------------------------------------------------------------------------------
>
>                 Key: HIVE-3104
>                 URL: https://issues.apache.org/jira/browse/HIVE-3104
>             Project: Hive
>          Issue Type: Bug
>          Components: CLI
>    Affects Versions: 0.9.0
>         Environment: Apache Hive 0.9.0, Apache Hadoop 0.20.205.0
>            Reporter: Mark Grover
>            Assignee: Xuefu Zhang
>
> Predicate pushdown seems to work for single-insert queries using LATERAL VIEW. It also
seems to work for multi-insert queries *not* using LATERAL VIEW. However, it doesn't work
for multi-insert queries using LATERAL VIEW: It errors out right away with 'FAILED: SemanticException
[Error 10041]: No partition predicate found for Alias "test" Table "test"'.
> Here are some examples. In the below examples, I make use of the fact that a query with
no partition filtering when run under "hive.mapred.mode=strict" fails.
> --Table creation and population
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (col1 array<int>, col2 int)  PARTITIONED BY (part_col int);
> INSERT OVERWRITE TABLE test PARTITION (part_col=1) SELECT array(1,2), count(*) FROM test;
> INSERT OVERWRITE TABLE test PARTITION (part_col=2) SELECT array(2,4,6), count(*) FROM
test;
> -- Query 1
> -- This succeeds (using LATERAL VIEW with single insert)
> set hive.mapred.mode=strict;
> FROM test
> LATERAL VIEW explode(col1) tmp AS exp_col1
> INSERT OVERWRITE DIRECTORY '/test/1'
>     SELECT exp_col1
>     WHERE (part_col=2);
> -- Query 2
> -- This succeeds (NOT using LATERAL VIEW with multi-insert)
> set hive.mapred.mode=strict;
> FROM test
> INSERT OVERWRITE DIRECTORY '/test/1'
>     SELECT col1
>     WHERE (part_col=2)
> INSERT OVERWRITE DIRECTORY '/test/2'
>     SELECT col1
>     WHERE (part_col=2);
> -- Query 3
> -- This fails (using LATERAL VIEW with multi-insert)
> set hive.mapred.mode=strict;
> FROM test
> LATERAL VIEW explode(col1) tmp AS exp_col1
> INSERT OVERWRITE DIRECTORY '/test/1'
>     SELECT exp_col1
>     WHERE (part_col=2)
> INSERT OVERWRITE DIRECTORY '/test/2'
>     SELECT exp_col1
>     WHERE (part_col=2);

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message