hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14680) retain consistent splits /during/ (as opposed to across) LLAP failures on top of HIVE-14589
Date Tue, 20 Sep 2016 18:22:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15507337#comment-15507337
] 

Siddharth Seth commented on HIVE-14680:
---------------------------------------

Really think we're better off fixing this within Orc itself - instead of working around it
in the split generator (which at some point will handle different file types). Can the ORC
getSplits not deal with this in BI mode?

> retain consistent splits /during/ (as opposed to across) LLAP failures on top of HIVE-14589
> -------------------------------------------------------------------------------------------
>
>                 Key: HIVE-14680
>                 URL: https://issues.apache.org/jira/browse/HIVE-14680
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>             Fix For: 2.2.0
>
>         Attachments: HIVE-14680.01.patch, HIVE-14680.02.patch, HIVE-14680.03.patch, HIVE-14680.patch
>
>
> see HIVE-14589.
> Basic idea (spent about 7 minutes thinking about this based on RB comment ;)) is to return
locations for all slots to HostAffinitySplitLocationProvider, the missing slots being inactive
locations (based solely on the last slot actually present). For the splits mapped to these
locations, fall back via different hash functions, or some sort of probing.
> This still doesn't handle all the cases, namely when the last slots are gone (consistent
hashing is supposed to be good for this?); however for that we'd need more involved coordination
between nodes or a central updater to indicate the number of nodes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message