hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jing Zhao <j...@hortonworks.com>
Subject Re: Hadoop HA error "JOURNAL is not supported in state standby"
Date Thu, 29 Aug 2013 20:40:36 GMT
Looks like you have some incompatibility between your client side and
the server side? Are you also running 2.0.5 in your client side?

As Harsh mentioned, the NN side warning msg is not related to your
InvalidProtocolBufferException. The warning msg indicates that both of
your NN are in the Standby state.

Thanks,
-Jing

On Thu, Aug 29, 2013 at 1:36 PM, orahad bigdata <oraclehad@gmail.com> wrote:
> Thanks  Harsh,
>
> I don't have auto failover configuration, but also I have tried to do
> this manually but didn't get success.
>
> hdfs haadmin -transitionToActive node1
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required
> fields: callId, status; Host Details : local host is: "clone2/XX.XX.XX.XX";
> destination host is: "clone1":8020;
>
> So is there any alternative to resolve this issue?.
>
> Thanks
>
> On 8/30/13, Harsh J <harsh@cloudera.com> wrote:
>> On the actual issue though: Do you also have auto-failover configured?
>>
>> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <oraclehad@gmail.com>
>> wrote:
>>> Hi,
>>>
>>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
>>> the NameNode started in standby mode and not changing the state.
>>>
>>> When I tried to do health check through  "hdfs haadmin -checkhealth
>>> <service id> " it's giving me below error.
>>>
>>> Failed on local exception:
>>> com.google.protobuf.InvalidProtocolBufferException: Message missing
>>> required fields: callId, status; Host Details : local host is:
>>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>>>
>>>  I checked the logs at NN side.
>>>
>>> 2013-08-30 00:49:16,074 ERROR
>>> org.apache.hadoop.security.UserGroupInformation:
>>> PriviledgedActionException as:hadoop (auth:SIMPLE)
>>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
>>> JOURNAL is not supported in state standby
>>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 1 on 8020, call
>>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
>>> from 192.168.126.31:48266: error:
>>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
>>> not supported in state standby
>>> 2013-08-30 00:49:32,391 INFO
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
>>> log roll on remote NameNode clone2:8020
>>> 2013-08-30 00:49:32,403 WARN
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
>>> trigger a roll of the active NN
>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>>> Operation category JOURNAL is not supported in state standby
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
>>>         at
>>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
>>>         at
>>> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
>>>         at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
>>>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
>>>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
>>>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>>         at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
>>>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>>>
>>>         at org.apache.hadoop.ipc.Client.call(Client.java:1235)
>>>         at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>>>         at $Proxy11.rollEditLog(Unknown Source)
>>>         at
>>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
>>>         at
>>> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>>>
>>> Did I missed something?
>>>
>>> Thanks
>>
>>
>>
>> --
>> Harsh J
>>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message