phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Samarth Jain (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-2154) Failure of one mapper should not affect other mappers in MR index build
Date Thu, 20 Aug 2015 06:33:45 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704376#comment-14704376
] 

Samarth Jain commented on PHOENIX-2154:
---------------------------------------

Reducers could start running before all the mappers are complete. However, the reduce step
in a reducer is not executed till all the mappers are done. Ravi, it looks like you are updating
the index state in the setUp method. Do you know when is setUp executed? Is it executed before
the shuffle phase of a reducer? If yes, it probably makes sense to move the code you have
in setUp to the reduce method instead. 

> Failure of one mapper should not affect other mappers in MR index build
> -----------------------------------------------------------------------
>
>                 Key: PHOENIX-2154
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-2154
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>            Assignee: maghamravikiran
>         Attachments: IndexTool.java, PHOENIX-2154-WIP.patch
>
>
> Once a mapper in the MR index job succeeds, it should not need to be re-done in the event
of the failure of one of the other mappers. The initial population of an index is based on
a snapshot in time, so new rows getting *after* the index build has started and/or failed
do not impact it.
> Also, there's a 1:1 correspondence between index rows and table rows, so there's really
no need to dedup. However, the index rows will have a different row key than the data table,
so I'm not sure how the HFiles are split. Will they potentially overlap and is this an issue?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message