hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "takeshi.miao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-4881) fine tune "Access token verification failed" error msg in datanode log
Date Wed, 05 Jun 2013 10:29:20 GMT

     [ https://issues.apache.org/jira/browse/HDFS-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

takeshi.miao updated HDFS-4881:
-------------------------------

    Attachment: HDFS-4881-branch-1.patch

I submitted a patch for branch-1 due to can not pass the CI check
                
> fine tune "Access token verification failed" error msg in datanode log
> ----------------------------------------------------------------------
>
>                 Key: HDFS-4881
>                 URL: https://issues.apache.org/jira/browse/HDFS-4881
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 1.0.0
>         Environment: CentOS-5.3, java-version-1.6.0_26
>            Reporter: takeshi.miao
>            Priority: Trivial
>             Fix For: 1.0.0
>
>         Attachments: HDFS-4881-branch-1.0.patch, HDFS-4881-branch-1.0-v1.patch, HDFS-4881-branch-1.patch
>
>
> I'd like to issue this ticket is due to we suffered a datanode access token verification
failure issue recently. The client is HBase who is accessing the local datanode via DFSClient.
The details log snippets as follows...
> *regionserver log*
> {code}
> ...
> [2013-05-24 08:33:37,553][regionserver8120-compactions-1369288874174][INFO ][org.apache.hadoop.hbase.regionserver.Store]:
Started compaction of 1 file(s) in cf=ho, hasReferences=true, into hdfs://sjdc-s-hdd-001.sjdc.ispn.trendmicro.com:8020/user/SPN-hbase/spn.guidcensus.ho/f99c6fb26f488034bf0e6ddd7a647ba4/.tmp,
seqid=3, totalSize=4.2g
> [2013-05-24 08:33:37,554][regionserver8120-compactions-1369288874174][INFO ][org.apache.hadoop.hdfs.DFSClient]:
Access token was invalid when connecting to /10.31.6.49:1004 : org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException:
Got access token error for OP_READ_BLOCK, self=/10.31.6.49:36530, remote=/10.31.6.49:1004,
for file /user/SPN-hbase/spn.guidcensus.ho/a565dd142933e3abf9bec33d59210d1b/ho/c5b37b9dd8801275c8fb160c0fb32ce5c48b56f4,
for block 4549293737579979499_205814042
> ...
> {code}
> *datanode log*
> {code}
> ...
> [2013-05-24 08:33:37,554][DataXceiver for client /10.31.6.49:36530 [Waiting for operation
#1]][ERROR][org.apache.hadoop.hdfs.server.datanode.DataNode]: DatanodeRegistration(10.31.6.49:1004,
storageID=DS-1953102179-10.31.6.49-1004-       1342490559943, infoPort=1006, ipcPort=50020):DataXceiver
> java.io.IOException: Access token verification failed, for client /10.31.6.49:36530 for
OP_READ_BLOCK for block blk_4549293737579979499_205814042
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:252)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:175)
> ...
> {code}
> After trace o.a.h.hdfs.security.token.block.BlockTokenSecretManager.java, I found that
there are more further details error description written in code.
> *o.a.h.hdfs.security.token.block.BlockTokenSecretManager.java*
> {code}
> public void checkAccess(BlockTokenIdentifier id, String userId, Block block,
>       AccessMode mode) throws InvalidToken {
>     if (LOG.isDebugEnabled()) {
>       LOG.debug("Checking access for user=" + userId + ", block=" + block
>           + ", access mode=" + mode + " using " + id.toString());
>     }
>     if (userId != null && !userId.equals(id.getUserId())) {
>       throw new InvalidToken("Block token with " + id.toString()
>           + " doesn't belong to user " + userId);
>     }
>     if (id.getBlockId() != block.getBlockId()) {
>       throw new InvalidToken("Block token with " + id.toString()
>           + " doesn't apply to block " + block);
>     }
>     if (isExpired(id.getExpiryDate())) {
>       throw new InvalidToken("Block token with " + id.toString()
>           + " is expired.");
>     }
>     if (!id.getAccessModes().contains(mode)) {
>       throw new InvalidToken("Block token with " + id.toString()
>           + " doesn't have " + mode + " permission");
>     }
>   }
> {code}
> But actually, this InvalidTokenException will not be handled further (but caught), so
I can not trace what kind of this access block token verification is...
> *o.a.h.hdfs.server.datanode.DataXceiver.java*
> {code}
> ...
> if (datanode.isBlockTokenEnabled) {
>       try {
>         datanode.blockTokenSecretManager.checkAccess(accessToken, null, block,
>             BlockTokenSecretManager.AccessMode.READ);
>       } catch (InvalidToken e) {
>         // the e object not handled further...
>         try {
>           out.writeShort(DataTransferProtocol.OP_STATUS_ERROR_ACCESS_TOKEN);
>           out.flush();
>           throw new IOException("Access token verification failed, for client "
>               + remoteAddress + " for OP_READ_BLOCK for block " + block); 
>         } finally {
>           IOUtils.closeStream(out);
>         }   
>       }   
>     }
> ...
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message