hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Boudnik (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1523) TestLargeBlock is failing on trunk
Date Wed, 01 Dec 2010 04:23:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965569#action_12965569
] 

Konstantin Boudnik commented on HDFS-1523:
------------------------------------------

Here's the scenario of the test:
- read the file in chunks of 134217728 (128Mb)
- after last full read there are 513 bytes to be read
- 512 bytes of those have to be read from the first block
- 1 byte is going to be read from the last block (second one)

When test passes before reading last 513 bytes call to FSNameSystem.getBlockLocationsInternal
returns last block of the file (size=1)

{noformat}
2010-11-30 19:44:49,439 DEBUG namenode.FSNamesystem (FSNamesystem.java:getBlockLocationsInternal(866))
- blocks = [blk_-6779333650185181528_1001, blk_-3599
865432887782445_1001]
2010-11-30 19:44:49,440 DEBUG namenode.FSNamesystem (FSNamesystem.java:getBlockLocationsInternal(881))
- last = blk_-3599865432887782445_1001
2010-11-30 19:44:49,457 INFO  FSNamesystem.audit (FSNamesystem.java:logAuditEvent(148)) -
ugi=cos       ip=/127.0.0.1   cmd=open        src=/home/cos/work/
H0.23/git/hdfs/build/test/data/2147484160.dat        dst=null        perm=null
2010-11-30 19:44:49,459 DEBUG hdfs.DFSClient (DFSInputStream.java:openInfo(113)) - newInfo
= LocatedBlocks{
  fileLength=2147484161
  underConstruction=false
  blocks=[LocatedBlock{blk_-6779333650185181528_1001; getBlockSize()=2147484160; corrupt=false;
offset=0; locs=[127.0.0.1:35608]}]
  lastLocatedBlock=LocatedBlock{blk_-3599865432887782445_1001; getBlockSize()=1; corrupt=false;
offset=2147484160; locs=[127.0.0.1:35608]}
  isLastBlockComplete=true}
...
2010-11-30 19:45:23,880 INFO  DataNode.clienttrace (BlockSender.java:sendBlock(491)) - src:
/127.0.0.1:35608, dest: /127.0.0.1:51640, bytes: 2164261380, op
: HDFS_READ, cliID: DFSClient_1273505070, offset: 0, srvID: DS-212336177-192.168.102.126-35608-1291175019763,
blockid: blk_-6779333650185181528_1001, durat
ion: 34361753463
2010-11-30 19:45:24,030 DEBUG namenode.FSNamesystem (FSNamesystem.java:getBlockLocationsInternal(866))
- blocks = [blk_-6779333650185181528_1001, blk_-3599
865432887782445_1001]
2010-11-30 19:45:24,030 DEBUG namenode.FSNamesystem (FSNamesystem.java:getBlockLocationsInternal(881))
- last = blk_-3599865432887782445_1001
2010-11-30 19:45:24,031 INFO  FSNamesystem.audit (FSNamesystem.java:logAuditEvent(148)) -
ugi=cos       ip=/127.0.0.1   cmd=open        src=/home/cos/work/H0.23/git/hdfs/build/test/data/2147484160.dat
       dst=null        perm=null
2010-11-30 19:45:24,032 DEBUG datanode.DataNode (DataXceiver.java:<init>(86)) - Number
of active connections is: 2
2010-11-30 19:45:24,099 DEBUG datanode.DataNode (DataXceiver.java:run(135)) - DatanodeRegistration(127.0.0.1:35608,
storageID=DS-212336177-192.168.102.126-35608-1291175019763, infoPort=46218, ipcPort=38099):Number
of active connections is: 3
2010-11-30 19:45:24,099 DEBUG datanode.DataNode (BlockSender.java:<init>(140)) - block=blk_-3599865432887782445_1001,
replica=FinalizedReplica, blk_-3599865432887782445_1001, FINALIZED
  getNumBytes()     = 1
  getBytesOnDisk()  = 1
  getVisibleLength()= 1
  getVolume()       = /home/cos/work/H0.23/git/hdfs/build/test/data/dfs/data/data2/current/finalized
  getBlockFile()    = /home/cos/work/H0.23/git/hdfs/build/test/data/dfs/data/data2/current/finalized/blk_-3599865432887782445
  unlinked=false
2010-11-30 19:45:24,101 DEBUG datanode.DataNode (BlockSender.java:<init>(231)) - replica=FinalizedReplica,
blk_-3599865432887782445_1001, FINALIZED
  getNumBytes()     = 1
  getBytesOnDisk()  = 1
  getVisibleLength()= 1
  getVolume()       = /home/cos/work/H0.23/git/hdfs/build/test/data/dfs/data/data2/current/finalized
  getBlockFile()    = /home/cos/work/H0.23/git/hdfs/build/test/data/dfs/data/data2/current/finalized/blk_-3599865432887782445
  unlinked=false
2010-11-30 19:45:24,103 INFO  DataNode.clienttrace (BlockSender.java:sendBlock(491)) - src:
/127.0.0.1:35608, dest: /127.0.0.1:51644, bytes: 5, op: HDFS_READ, cliID: DFSClient_1273505070,
offset: 0, srvID: DS-212336177-192.168.102.126-35608-1291175019763, blockid: blk_-3599865432887782445_1001,
duration: 1854472
{noformat}

In case of failure:
{noformat}
2010-11-30 19:35:49,426 DEBUG namenode.FSNamesystem (FSNamesystem.java:getBlockLocationsInternal(866))
- blocks = [blk_1170274882140601397_1001, blk_289191
6181488413346_1001]
2010-11-30 19:35:49,426 DEBUG namenode.FSNamesystem (FSNamesystem.java:getBlockLocationsInternal(881))
- last = blk_2891916181488413346_1001
2010-11-30 19:35:49,427 INFO  FSNamesystem.audit (FSNamesystem.java:logAuditEvent(148)) -
ugi=cos       ip=/127.0.0.1   cmd=open        src=/home/cos/work/
hadoop/git/hdfs/build/test/data/2147484160.dat       dst=null        perm=null
2010-11-30 19:35:49,428 DEBUG hdfs.DFSClient (DFSInputStream.java:openInfo(113)) - newInfo
= LocatedBlocks{
  fileLength=2147484161
  underConstruction=false
  blocks=[LocatedBlock{blk_1170274882140601397_1001; getBlockSize()=2147484160; corrupt=false;
offset=0; locs=[127.0.0.1:35644]}]
  lastLocatedBlock=LocatedBlock{blk_2891916181488413346_1001; getBlockSize()=1; corrupt=false;
offset=2147484160; locs=[127.0.0.1:35644]}
  isLastBlockComplete=true}
...
2010-11-30 19:36:16,761 INFO  DataNode.clienttrace (BlockSender.java:sendBlock(491)) - src:
/127.0.0.1:35644, dest: /127.0.0.1:52290, bytes: 2164194816, op
: HDFS_READ, cliID: DFSClient_635470834, offset: 0, srvID: DS-514949605-127.0.0.1-35644-1291174495289,
blockid: blk_1170274882140601397_1001, duration: 273
20364567
2010-11-30 19:36:16,761 DEBUG datanode.DataNode (DataXceiver.java:run(135)) - DatanodeRegistration(127.0.0.1:35644,
storageID=DS-514949605-127.0.0.1-35644-
1291174495289, infoPort=56924, ipcPort=60439):Number of active connections is: 2
2010-11-30 19:36:16,794 DEBUG datanode.DataNode (DataXceiver.java:<init>(86)) - Number
of active connections is: 1
2010-11-30 19:36:16,794 DEBUG datanode.DataNode (BlockSender.java:<init>(140)) - block=blk_1170274882140601397_1001,
replica=FinalizedReplica, blk_11702748
82140601397_1001, FINALIZED
  getNumBytes()     = 2147484160
  getBytesOnDisk()  = 2147484160
  getVisibleLength()= 2147484160
  getVolume()       = /home/cos/work/hadoop/git/hdfs/build/test/data/dfs/data/data1/current/finalized
  getBlockFile()    = /home/cos/work/hadoop/git/hdfs/build/test/data/dfs/data/data1/current/finalized/blk_1170274882140601397
  unlinked=false
2010-11-30 19:36:16,795 DEBUG datanode.DataNode (BlockSender.java:<init>(231)) - replica=FinalizedReplica,
blk_1170274882140601397_1001, FINALIZED
  getNumBytes()     = 2147484160
  getBytesOnDisk()  = 2147484160
  getVisibleLength()= 2147484160
  getVolume()       = /home/cos/work/hadoop/git/hdfs/build/test/data/dfs/data/data1/current/finalized
  getBlockFile()    = /home/cos/work/hadoop/git/hdfs/build/test/data/dfs/data/data1/current/finalized/blk_1170274882140601397
  unlinked=false
2010-11-30 19:36:17,276 INFO  DataNode.clienttrace (BlockSender.java:sendBlock(491)) - src:
/127.0.0.1:35644, dest: /127.0.0.1:52296, bytes: 135200256, op:
 HDFS_READ, cliID: DFSClient_635470834, offset: 2013265920, srvID: DS-514949605-127.0.0.1-35644-1291174495289,
blockid: blk_1170274882140601397_1001, durat
ion: 480762241
2010-11-30 19:36:17,276 DEBUG datanode.DataNode (DataXceiver.java:run(135)) - DatanodeRegistration(127.0.0.1:35644,
storageID=DS-514949605-127.0.0.1-35644-
1291174495289, infoPort=56924, ipcPort=60439):Number of active connections is: 2
2010-11-30 19:36:17,290 WARN  hdfs.DFSClient (DFSInputStream.java:readBuffer(486)) - Exception
while reading from blk_1170274882140601397_1001 of /home/cos
/work/hadoop/git/hdfs/build/test/data/2147484160.dat from 127.0.0.1:35644: java.io.IOException:
Premature EOF from inputStream
        at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:118)
        at org.apache.hadoop.hdfs.BlockReader.readChunk(BlockReader.java:275)
{noformat}

so it seems like the test fails because wrong block is being read or something.

> TestLargeBlock is failing on trunk
> ----------------------------------
>
>                 Key: HDFS-1523
>                 URL: https://issues.apache.org/jira/browse/HDFS-1523
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>
> TestLargeBlock is failing for more than a week not on 0.22 and trunk with
> {noformat}
> java.io.IOException: Premeture EOF from inputStream
> 	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:118)
> 	at org.apache.hadoop.hdfs.BlockReader.readChunk(BlockReader.java:275)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message