hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (YARN-9029) yarn-ats hbase expects OneFS(namenode and datanode) to have a yarn principle
Date Fri, 16 Nov 2018 04:58:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-9029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rohith Sharma K S resolved YARN-9029.
-------------------------------------
    Resolution: Invalid

This is vendor specific deployment issue. Please contact the vendor!

> yarn-ats hbase expects OneFS(namenode and datanode) to have a yarn principle
> ----------------------------------------------------------------------------
>
>                 Key: YARN-9029
>                 URL: https://issues.apache.org/jira/browse/YARN-9029
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: ATSv2
>    Affects Versions: 3.0.1
>         Environment: Ambari 2.7.1
> HDP 3.0.1
> Isilon OneFS 8.1.2
>            Reporter: Kirankumar Bhusanurmath
>            Priority: Major
>
> *Background on ats-hbase start success*:: We had setStoragePolicy exception on OneFS
for yarn-ats hbase region server and below messages respectively on HDP and OneFS cluster.
> *HDP Host* *è* */var/log/hadoop-yarn/embedded-yarn-ats-hbase/*
> 2018-11-02 05:07:21,027 WARN  [StoreOpener-1588230740-1] util.CommonFSUtils: Unable
to set storagePolicy=HOT for path=hdfs://kb-hdp-1.west.isilon.com:8020/atsv2/hbase/data/data/hbase/meta/1588230740/info.
DEBUG log level might have more details.
> java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  
> *OneFS HDFS log for (unsecured)*
> 2018-11-01T10:25:19Z <30.6> kbhusan-2oq1zs7-1 hdfs[2604]: [hdfs] RPC V9 
> user: yarn-ats exception: org.apache.hadoop.ipc.RpcNoSuchMethodException
> cause: Unknown RPC: setStoragePolicy
>  
> *OneFS HDFS log for (secured)*
> 2018-11-01T12:39:56Z <30.6> kbhusan-2oq1zs7-1 hdfs[2604]: [hdfs] RPC V9 
> user: yarn-ats-hbase/[krb-hdp301.solarch.emc.com@SOLARCH.EMC.COM|mailto:krb-hdp301.solarch.emc.com@SOLARCH.EMC.COM]
exception:
> org.apache.hadoop.ipc.RpcNoSuchMethodException cause: Unknown RPC:
> setStoragePolicy
>  
>  
> As a resolution, we set below 2 properties in custom hbase-site.xml file and setstoragePolicy
exception got resolved.
>  *Custom yarn-hbase-site.xml*
> 'hbase.wal.storage.policy' to ‘NONE’
> ‘hbase.hstore.block.storage.policy’ to ‘NONE’
>  
>  
> *********
> *NOW*, we have a new issue that is OneFS HDFS log and /var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-regionserver-****.log
gets flooded with below error messages.*OneFS HDFS Log*
> -------------------
> 2018-11-06T06:25:43Z <30.6> kbhusan-edl9tdc-1 hdfs[2542]: [hdfs] RPC V9
> complete user:
> yarn-ats-hbase/[rduvnode343393.west.isilon.com@WEST.ISILON.COM|mailto:rduvnode343393.west.isilon.com@WEST.ISILON.COM]
exception:
> java.io.FileNotFoundException cause: File not found:
> /atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541478673646/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541478673646.meta.1541478709060.meta
> <FS Exception at CreateHandleFromPath():448 status:
> STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 Path:
> atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541478673646/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541478673646.meta.1541478709060.meta>
> ---------------
>  */var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-regionserver-****.log*
> --------------
> 2018-11-06 04:01:12,863 INFO  [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=1.12 MB, freeSize=1.25 GB, max=1.25 GB,
> blockCount=24, accesses=287, hits=263, hitRatio=91.64%, ,
> cachingAccesses=287, cachingHits=263, cachingHitsRatio=91.64%,
> evictions=419, evicted=0, evictedPerRun=0.0
>  
> 2018-11-06 04:01:48,763 WARN 
> [LeaseRenewer:[yarn-ats@kb-hdp301-2.west.isilon.com|mailto:yarn-ats@kb-hdp301-2.west.isilon.com]:8020]
impl.LeaseRenewer:
> Failed to renew lease for [DFSClient_NONMAPREDUCE_-533598397_1] for 30
> seconds.  Will retry shortly ...
>  
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> Client auth method 'SIMPLE', mechanism '' is not supported on zone:1
> auth:'kerberos_only'
>  
>         at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
>  
>         at org.apache.hadoop.ipc.Client.call(Client.java:1443)
>  
>         at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>  
>                        “
>                        “
>                        “
>                        “
> 2018-11-06 12:51:02,843 WARN  [Close-WAL-Writer-1] asyncfs.FanOutOneBlockAsyncDFSOutputHelper:
complete file /atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541488822956/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541488822956.1541492429802
failed, retry = 637
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File not found:
/atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541488822956/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541488822956.1541492429802
>         at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1443)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>  
> --------------
> On checking the ats-hbase app logs, found hbase region server is trying to flush the
old edits to free the WAL and it fails due to authentication issue. 
> *Why yarn-ats hbase expects OneFS(namenode and datanode) to have a yarn principle?*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message