hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1093) NNBench generates millions of NotReplicatedYetException in Namenode log
Date Fri, 09 Mar 2007 19:11:09 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12479688
] 

dhruba borthakur commented on HADOOP-1093:
------------------------------------------

I have two questions regarding two snippets of code in method readBuffer() in ChecksumFilesystem.java.

1.  There is a loop of the following form:

        boolean endOfFile=false;
        while (read < len && !endOfFile) {
          int count = datas.read(b, off + read, len - read);
          if (count < 0)
            endOfFile = true;
          else
            read += count;
        }

Now, is there a possibility that datas.read() returns 0? If so, then the above might loop
indefinitely.

2. Another portion of code:

            if (seekToNewSource(oldPos)) {
              throw ce;
            } else {
              retry = true;
            }

Isn't it true that seekTonewSource returns true if there is another possible source for this
block. In that case, should not the above check be reversed? if my observation is correct,
the above code might result in an infinite loop.


   


> NNBench generates millions of NotReplicatedYetException in Namenode log
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-1093
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1093
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.12.0
>            Reporter: Nigel Daley
>         Assigned To: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.12.1
>
>
> Running NNBench on latest trunk (0.12.1 candidate) on a few hundred nodes yielded 2.3
million of these exceptions in the NN log:
>    2007-03-08 09:23:03,053 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on
8020 call error:
>    org.apache.hadoop.dfs.NotReplicatedYetException: Not replicated yet
>         at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:803)
>         at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:309)
>         at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:336)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:559)
> I run NNBench to create files with block size set to 1 and replication set to 1.  NNBench
then writes 1 byte to the file.  Minimum replication for the cluster is the default, ie 1.
 If it encounters an exception while trying to do either the create or write operations, it
loops and tries again.  Multiply this by 1000 files per node and a few hundred nodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message