hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError
Date Sat, 23 Jun 2007 06:41:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12507565
] 

dhruba borthakur commented on HADOOP-1520:
------------------------------------------

The DFSClient retries the ClientProtocol.complete() call untill it is successful. I am assuming
that a previous complete() call was successfully executed at the NameNode but the client decided
to declare that call as "timedout" and retried. The retried call experiences this Exception
over and over again.

One generic solution is a have a unique sequence number per RPC that identifies that RPC.
The NameNode remembers the last few thousands RPCs that it executed. If a client sends a RPC
with a sequence number that exists in the NameNode cache, the NameNode returns the results
that were produced by the previous execution of the same RPC.

Can you pl send a pointer to a namenode log file that shows this problem?

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.14.0
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020,
call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212)
from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index:
0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile:
failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks()
is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020,
call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212)
from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc
by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc
by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the
way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message