hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sameer Paranjpye (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1093) NNBench generates millions of NotReplicatedYetException in Namenode log
Date Fri, 16 Mar 2007 05:54:09 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12481486

Sameer Paranjpye commented on HADOOP-1093:

So Nigel and I were poking around the Namenode logs, trying to trace all the events in the
lifetime of a block. We have a theory that seems to fit all the symptoms.

If you look through the life story of blk_3903667269962316732, the following events appear:

2007-03-16 01:21:05,248 DEBUG org.apache.hadoop.dfs.StateChange: *BLOCK* NameNode.reportWrittenBlock:
blk_391490390916935261 is written to 1 locations
2007-03-16 01:21:05,248 DEBUG org.apache.hadoop.dfs.StateChange: *BLOCK* NameNode.blockReport:
from 2627 blocks
2007-03-16 01:21:05,250 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.blockReceived:
blk_391490390916935261 is received from
2007-03-16 01:21:05,250 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.addStoredBlock:
blockMap updated: is added to blk_391490390916935261
2007-03-16 01:21:05,254 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.processReport:
from 2627 blocks
2007-03-16 01:21:05,255 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.removeStoredBlock:
blk_391490390916935261 from

What seems to be happening is a race between a client reporting a block and a block report
from a Datanode:

- the client reports a written block
- immediately after this the NameNode receives a block report from the DataNode which *doesn't*
contain this block because the block report was generated while the block was in flight
- the NameNode trusts the block report and blithely removes the block from block map
- the next attempt by the client to allocate a block causes a NotReplicatedYetException

Other tells for this condition:
- The NRYE happens mostly in the Namenode benchmark, which generates lots and lots and lots
of blocks with exactly 1 replica. With 'n > 1' replicas, 'n' block reports have to coincide
with the client report in order for this to happen.

- The NRYE shows up 1 hour into the benchmark and 1 hour is of course the block report interval.
We validated this by setting the block report interval to 5 minutes and sure enough NRYEs
showed up in 5 minutes.

We think the checksum patch caused this because it enforced a 1-byte blocksize on .crc files
in NNbench. Earlier, setting the block size on a file had no effect on .crc files. This certainly
needs fixing, but doesn't appear to be a blocker for 0.12.1.

> NNBench generates millions of NotReplicatedYetException in Namenode log
> -----------------------------------------------------------------------
>                 Key: HADOOP-1093
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1093
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.12.0
>            Reporter: Nigel Daley
>         Assigned To: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.12.1
> Running NNBench on latest trunk (0.12.1 candidate) on a few hundred nodes yielded 2.3
million of these exceptions in the NN log:
>    2007-03-08 09:23:03,053 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on
8020 call error:
>    org.apache.hadoop.dfs.NotReplicatedYetException: Not replicated yet
>         at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:803)
>         at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:309)
>         at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:336)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:559)
> I run NNBench to create files with block size set to 1 and replication set to 1.  NNBench
then writes 1 byte to the file.  Minimum replication for the cluster is the default, ie 1.
 If it encounters an exception while trying to do either the create or write operations, it
loops and tries again.  Multiply this by 1000 files per node and a few hundred nodes.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message