hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "takeshi.miao (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-4881) fine tune "Access token verification failed" error msg in datanode log
Date Wed, 05 Jun 2013 08:38:20 GMT
takeshi.miao created HDFS-4881:
----------------------------------

             Summary: fine tune "Access token verification failed" error msg in datanode log
                 Key: HDFS-4881
                 URL: https://issues.apache.org/jira/browse/HDFS-4881
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: datanode
    Affects Versions: 1.0.0
         Environment: CentOS-5.3, java-version-1.6.0_26
            Reporter: takeshi.miao
            Priority: Trivial


I'd like to issue this ticket is due to we suffered a datanode access token verification failure
issue recently. The client is HBase who is accessing the local datanode via DFSClient. The
details log snippets as follows...
*regionserver log*
{code}
...
[2013-05-24 08:33:37,553][regionserver8120-compactions-1369288874174][INFO ][org.apache.hadoop.hbase.regionserver.Store]:
Started compaction of 1 file(s) in cf=ho, hasReferences=true, into hdfs://sjdc-s-hdd-001.sjdc.ispn.trendmicro.com:8020/user/SPN-hbase/spn.guidcensus.ho/f99c6fb26f488034bf0e6ddd7a647ba4/.tmp,
seqid=3, totalSize=4.2g
[2013-05-24 08:33:37,554][regionserver8120-compactions-1369288874174][INFO ][org.apache.hadoop.hdfs.DFSClient]:
Access token was invalid when connecting to /10.31.6.49:1004 : org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException:
Got access token error for OP_READ_BLOCK, self=/10.31.6.49:36530, remote=/10.31.6.49:1004,
for file /user/SPN-hbase/spn.guidcensus.ho/a565dd142933e3abf9bec33d59210d1b/ho/c5b37b9dd8801275c8fb160c0fb32ce5c48b56f4,
for block 4549293737579979499_205814042
...
{code}

*datanode log*
{code}
...
[2013-05-24 08:33:37,554][DataXceiver for client /10.31.6.49:36530 [Waiting for operation
#1]][ERROR][org.apache.hadoop.hdfs.server.datanode.DataNode]: DatanodeRegistration(10.31.6.49:1004,
storageID=DS-1953102179-10.31.6.49-1004-       1342490559943, infoPort=1006, ipcPort=50020):DataXceiver
java.io.IOException: Access token verification failed, for client /10.31.6.49:36530 for OP_READ_BLOCK
for block blk_4549293737579979499_205814042
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:252)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:175)
...
{code}

After trace o.a.h.hdfs.security.token.block.BlockTokenSecretManager.java, I found that there
are more further details error description written in code.
*o.a.h.hdfs.security.token.block.BlockTokenSecretManager.java*
{code}
public void checkAccess(BlockTokenIdentifier id, String userId, Block block,
      AccessMode mode) throws InvalidToken {
    if (LOG.isDebugEnabled()) {
      LOG.debug("Checking access for user=" + userId + ", block=" + block
          + ", access mode=" + mode + " using " + id.toString());
    }
    if (userId != null && !userId.equals(id.getUserId())) {
      throw new InvalidToken("Block token with " + id.toString()
          + " doesn't belong to user " + userId);
    }
    if (id.getBlockId() != block.getBlockId()) {
      throw new InvalidToken("Block token with " + id.toString()
          + " doesn't apply to block " + block);
    }
    if (isExpired(id.getExpiryDate())) {
      throw new InvalidToken("Block token with " + id.toString()
          + " is expired.");
    }
    if (!id.getAccessModes().contains(mode)) {
      throw new InvalidToken("Block token with " + id.toString()
          + " doesn't have " + mode + " permission");
    }
  }
{code}

But actually, this InvalidTokenException will not be handled further (but caught), so I can
not trace what kind of this access block token verification is...
*o.a.h.hdfs.server.datanode.DataXceiver.java*
{code}
...
if (datanode.isBlockTokenEnabled) {
      try {
        datanode.blockTokenSecretManager.checkAccess(accessToken, null, block,
            BlockTokenSecretManager.AccessMode.READ);
      } catch (InvalidToken e) {
        // the e object not handled further...
        try {
          out.writeShort(DataTransferProtocol.OP_STATUS_ERROR_ACCESS_TOKEN);
          out.flush();
          throw new IOException("Access token verification failed, for client "
              + remoteAddress + " for OP_READ_BLOCK for block " + block); 
        } finally {
          IOUtils.closeStream(out);
        }   
      }   
    }
...
{code}


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message