hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-7421) Move processing of postponed over-replicated blocks to a background task
Date Fri, 23 Jan 2015 19:36:36 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Aaron T. Myers resolved HDFS-7421.
    Resolution: Duplicate

> Move processing of postponed over-replicated blocks to a background task
> ------------------------------------------------------------------------
>                 Key: HDFS-7421
>                 URL: https://issues.apache.org/jira/browse/HDFS-7421
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: ha, namenode
>    Affects Versions: 2.6.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
> In an HA environment, we postpone sending block invalidates to DNs until all DNs holding
a given block have done at least one block report to the NN after it became active. When that
first block report after becoming active does occur, we attempt to reprocess all postponed
misreplicated blocks inline with the block report RPC. In the case where there are many postponed
misreplicated blocks, this can cause block report RPCs to take an inordinately long time to
complete, sometimes on the order of minutes, which has the potential to tie up RPC handlers,
block incoming RPCs, etc. There's no need to hurriedly process all postponed misreplicated
blocks so that we can quickly send invalidate commands back to DNs, so let's move this processing
outside of the RPC handler context and into a background thread.

This message was sent by Atlassian JIRA

View raw message