hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
Date Tue, 04 Sep 2018 19:43:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16603497#comment-16603497
] 

Prasanth Jayachandran commented on HIVE-18873:
----------------------------------------------

{quote}Under the assumption that Hive is ensuring the results are accurate anyways? (e.g.
we'd want to check that all storage handlers work this way)
{quote}
Orc PPD works this way, if orc cannot handle the filter it will drop the filter and continue
with rest of the filters. The filters will still remain in hive operator tree which will get
reapplied anyways.

 

If the test failure is unrelated to the patch and is already happening in the master, I think
we can go ahead and commit the latest patch and track the test failure in a separate ticket. 

+1, comment failing test, track it separately. 

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers
to produce erroneous result
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-18873
>                 URL: https://issues.apache.org/jira/browse/HIVE-18873
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: Ankit Singhal
>            Assignee: Ankit Singhal
>            Priority: Major
>         Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && mrwork.getAliases().size()
> 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't believe that
predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message