hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Krogen (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-13898) Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode
Date Tue, 11 Sep 2018 19:48:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611150#comment-16611150
] 

Erik Krogen edited comment on HDFS-13898 at 9/11/18 7:47 PM:
-------------------------------------------------------------

Nice find [~csun]! The change to production code LGTM. For the tests:
* You don't need to make the changes to {{MiniQJMHACluster}}. You can set the number of DataNodes
like:
{code}
builder.getDfsBuilder().numDataNodes(3);
{code}
I also don't really understand why we need to tweak the number of DNs if the BlockManager
is mocked anyway?
* I don't really find the BlockManager mocking to be very clean. I think we should be able
to achieve something similar by using a real BlockManager, but injecting some fake blocks:
{code}
NameNodeAdapter.getNamesystem(namenodes[2]).getBlockManager().addBlockCollection(...)
{code}
or creating real blocks, but then corrupting them:
{code}
dfsCluster.corruptBlockOnDataNodes(...)
{code}


was (Author: xkrogen):
Nice find [~csun]! The change to production code LGTM. For the tests:
* You don't need to make the changes to {{MiniQJMHACluster}}. You can set the number of DataNodes
like:
{code}
builder.getDfsBuilder().numDataNodes(3);
{code}
I also don't really understand why we need to tweak the number of DNs if the BlockManager
is mocked anyway?
* I don't really find the BlockManager mocking to be very clean. I think we should be able
to achieve something similar by using a real BlockManager, but injecting some fake blocks:
{code}
NameNodeAdapter.getNamesystem(namenodes[2]).getBlockManager().addBlockCollection(...)
{code}
or creating real blocks, but then corrupting them:
{code}
dfsCluster.corruptBlockOnDataNodes()
{code}

> Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode
> ------------------------------------------------------------------------------------
>
>                 Key: HDFS-13898
>                 URL: https://issues.apache.org/jira/browse/HDFS-13898
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Chao Sun
>            Assignee: Chao Sun
>            Priority: Major
>         Attachments: HDFS-13898-HDFS-12943.000.patch
>
>
> When ObserverNameNode is in safe mode, {{getBlockLocations}} may throw safe mode exception
if the given file doesn't have any block yet. 
> {code}
>     try {
>       checkOperation(OperationCategory.READ);
>       res = FSDirStatAndListingOp.getBlockLocations(
>           dir, pc, srcArg, offset, length, true);
>       if (isInSafeMode()) {
>         for (LocatedBlock b : res.blocks.getLocatedBlocks()) {
>           // if safemode & no block locations yet then throw safemodeException
>           if ((b.getLocations() == null) || (b.getLocations().length == 0)) {
>             SafeModeException se = newSafemodeException(
>                 "Zero blocklocations for " + srcArg);
>             if (haEnabled && haContext != null &&
>                 haContext.getState().getServiceState() == HAServiceState.ACTIVE) {
>               throw new RetriableException(se);
>             } else {
>               throw se;
>             }
>           }
>         }
>       }
> {code}
> It only throws {{RetriableException}} for active NN so requests on observer may just
fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message