hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1085) hftp read failing silently
Date Tue, 06 Apr 2010 22:59:33 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854237#action_12854237
] 

Koji Noguchi commented on HDFS-1085:
------------------------------------

On the source datanode side, there was a log with 
{quote}
2010-04-06 17:56:42,994 ERROR org.mortbay.log: /streamFile
java.nio.channels.CancelledKeyException
        at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
        at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
        at org.mortbay.io.nio.SelectChannelEndPoint.updateKey(SelectChannelEndPoint.java:324)
        at org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:278)
        at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:542)
        at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
        at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
        at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
        at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
        at org.apache.hadoop.hdfs.server.namenode.StreamFile.doGet(StreamFile.java:68)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
        at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:663)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:324)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-04-06 17:56:43,007 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /99.99.99.99:50010, dest: /88.88.88.88:49774, bytes: 61292544, op: HDFS_READ, cliID:
DFSClient_-563231950, offset: 0, srvID: DS-1411469763-98.136.245.125-50010-1268765357890,
blockid: blk_-8819444732305373616_6553266, duration: 95558730000
{quote}

It was a distcp, so it should have read the whole 1 block(128MB) instead of bytes: 61292544.
{quote}
1. blk_-8819444732305373616_6553266 len=134217728
{quote}

Nicholas pointed out to me that this is very similar to error observed in HDFS-1024.

For distcp, thanks to the extra size checking from HADOOP-3294 we are ok, but does this mean
I cannot trust the result of hadoop '-get' through hftp ?
{quote}
% hadoop dfs -get hftp://somehost/somepath/file1  .
{quote}
I have a  feeling that this command can *silently* fail with incomplete output.


> hftp read  failing silently
> ---------------------------
>
>                 Key: HDFS-1085
>                 URL: https://issues.apache.org/jira/browse/HDFS-1085
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>
> When performing a massive distcp through hftp, we saw many tasks fail with 
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 :
java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
>         at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
>         at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
>         at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>         at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message