hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siyao Meng (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDDS-1892) HA failover attempt log level should be set to DEBUG
Date Fri, 02 Aug 2019 00:02:00 GMT

     [ https://issues.apache.org/jira/browse/HDDS-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Siyao Meng updated HDDS-1892:
-----------------------------
    Description: 
Current HA failover log level is INFO:
{code:bash}
$ ozone sh volume create /volume
2019-08-01 20:55:03 INFO RetryInvocationHandler:411 — com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.NotLeader Exception):
OM om2 is not the leader. Could not determine the leader node.
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotleaderException(OzoneManagerProtocolServerSideTranslatorPB.java:183)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:172)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:94)
        at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
, while invoking $Proxy19.submitRequest over nodeld=om2,nodeAddress=om2:9862 after 1 failover
attempts. Trying to failover immediately.
2019-08-01 20:55:03 INFO RpcClient:288 — Creating Volume: volume, with hadoop as owner.
VOLUME_ALREADY_EXISTS Volume already exists

$ ozone sh bucket create /volume/bucket
2019-08-01 20:55:23 INFO RetryInvocationHandler:411 — com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.Notleader Exception):
OM om2 is not the leader. Could not determine the leader node.
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotLeaderException(OzoneManagerProtocolServerSideTranslatorPB.java:183)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:172)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:94)
        at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
, while invoking $Proxy19.submitRequest over nodeId.om2,nodeAddress.om2:9862 after 1 failover
attempts. Trying to failover immediately.
2019-08-01 20:55:24 INFO RpcClient:425 — Creating Bucket: volume/bucket, with Versioning
false and Storage Type set to DISK and Encryption set to false
{code}

The client will print out every failover attempt.

Suppress it by setting the log level to DEBUG.

  was:
Current HA failover log level is INFO:
{code:bash}
$ ozone sh volume create /volume
2019-08-01 20:55:03 INFO RetrylnvocationHandler:411 — com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.NotLeader Exception):
OM om2 is not the leader. Could not determine the leader node.
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotleaderException(OzoneManagerProtocolServerSideTranslatorPB.java:183)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:172)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:94)
        at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
, while invoking $Proxy19.submitRequest over nodeld=om2,nodeAddress=om2:9862 after 1 failover
attempts. Trying to failover immediately.
2019-08-01 20:55:03 INFO RpcClient:288 — Creating Volume: volume, with hadoop as owner.
VOLUME_ALREADY_EXISTS Volume already exists

$ ozone sh bucket create /volume/bucket
2019-08-01 20:55:23 INFO RetrylnvocationHandler:411 — com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.Notleader Exception):
OM om2 is not the leader. Could not determine the leader node.
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotLeaderException(OzoneManagerProtocolServerSideTranslatorPB.java:183)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:172)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:94)
        at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
, while invoking $Proxy19.submitRequest over nodeId.om2,nodeAddress.om2:9862 after 1 failover
attempts. Trying to failover immediately.
2019-08-01 20:55:24 INFO RpcClient:425 — Creating Bucket: volume/bucket, with Versioning
false and Storage Type set to DISK and Encryption set to false
{code}

The client will print out every failover attempt.

Suppress it by setting the log level to DEBUG.


> HA failover attempt log level should be set to DEBUG
> ----------------------------------------------------
>
>                 Key: HDDS-1892
>                 URL: https://issues.apache.org/jira/browse/HDDS-1892
>             Project: Hadoop Distributed Data Store
>          Issue Type: Sub-task
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>
> Current HA failover log level is INFO:
> {code:bash}
> $ ozone sh volume create /volume
> 2019-08-01 20:55:03 INFO RetryInvocationHandler:411 — com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.NotLeader Exception):
OM om2 is not the leader. Could not determine the leader node.
>         at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotleaderException(OzoneManagerProtocolServerSideTranslatorPB.java:183)
>         at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:172)
>         at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:94)
>         at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
>         at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
>         at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> , while invoking $Proxy19.submitRequest over nodeld=om2,nodeAddress=om2:9862 after 1
failover attempts. Trying to failover immediately.
> 2019-08-01 20:55:03 INFO RpcClient:288 — Creating Volume: volume, with hadoop as owner.
> VOLUME_ALREADY_EXISTS Volume already exists
> $ ozone sh bucket create /volume/bucket
> 2019-08-01 20:55:23 INFO RetryInvocationHandler:411 — com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.Notleader Exception):
OM om2 is not the leader. Could not determine the leader node.
>         at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotLeaderException(OzoneManagerProtocolServerSideTranslatorPB.java:183)
>         at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:172)
>         at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:94)
>         at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
>         at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
>         at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> , while invoking $Proxy19.submitRequest over nodeId.om2,nodeAddress.om2:9862 after 1
failover attempts. Trying to failover immediately.
> 2019-08-01 20:55:24 INFO RpcClient:425 — Creating Bucket: volume/bucket, with Versioning
false and Storage Type set to DISK and Encryption set to false
> {code}
> The client will print out every failover attempt.
> Suppress it by setting the log level to DEBUG.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message