hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhaofei Meng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6032) scm cleaner task should rm InMemorySCMStore some cachedResources which does not exists in hdfs fs
Date Wed, 28 Dec 2016 06:10:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15782180#comment-15782180
] 

Zhaofei Meng commented on YARN-6032:
------------------------------------

We should modify use interface in ClientProtocolService to verify fs if or not exist in hdfs.

public UseSharedCacheResourceResponse use(
      UseSharedCacheResourceRequest request) throws YarnException,
      IOException {

    UseSharedCacheResourceResponse response =
        recordFactory.newRecordInstance(UseSharedCacheResourceResponse.class);

    UserGroupInformation callerUGI;
    try {
      callerUGI = UserGroupInformation.getCurrentUser();
    } catch (IOException ie) {
      LOG.info("Error getting UGI ", ie);
      throw RPCUtil.getRemoteException(ie);
    }

    String fileName =
        this.store.addResourceReference(request.getResourceKey(),
            new SharedCacheResourceReference(request.getAppId(),
                callerUGI.getShortUserName()));

    if (fileName != null) {
      if(fs.exists(new Path(fs.getHomeDirectory(),fileName))){
        response
                .setPath(getCacheEntryFilePath(request.getResourceKey(), fileName));
        this.metrics.incCacheHitCount();
      }else{
        this.store.removeResource(request.getResourceKey());
      }
    } else {
      this.metrics.incCacheMissCount();
    }

    return response;
  }

>  scm cleaner task should rm InMemorySCMStore some cachedResources which does not exists
in hdfs fs
> --------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6032
>                 URL: https://issues.apache.org/jira/browse/YARN-6032
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.7.1
>            Reporter: Zhaofei Meng
>             Fix For: 2.7.1
>
>
> If cacheresources exist in scm but not exist in hdfs,the cacheresources  whill not rm
from scm until restart scm.So we shoult add check funcion in cleaner task that  rm the cachedResources
which does not exists in hdfs fs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message