hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankil Doshi <forpan...@gmail.com>
Subject Re: org.apache.hadoop.hbase.exceptions.DeserializationException: Missing pb magic PBUF prefix
Date Fri, 23 Oct 2015 18:28:12 GMT
One other thing which is different in my setup is I am using filesystem for
hbase rather hbase-indexer needs hdfs setup to be fully functional. So I
need to change that.

Also, do you have local hbase setup or hbase cluster mode ?

Pankil

On Fri, Oct 23, 2015 at 11:23 AM, Pankil Doshi <forpankil@gmail.com> wrote:

> Hi Beeshma,
>
> Thanks for your response.
>
> I am running zookeeper locally. but I am not managing it with hbase i.e I
> have this set:
> export HBASE_MANAGES_ZK=false
>
> and also:
> <property>
>                 <name>hbase.cluster.distributed</name>
>                 <value>true</value>
>         </property>
>
> even though I have everything running locally in standalone mode.
>
> If I dont set "hbase.cluster.distributed"  I am seeing zookeeper being
> started with start of my hbase. I am not sure if there is any other good
> way not to start or stop zookeeper with hbase as by only setting
>  (HBASE_MANAGES_ZK=false) it doesnt work.
>
> Were you able to setup hbase-indexer at all ?
>
> Pankil
>
>
> On Fri, Oct 23, 2015 at 10:51 AM, beeshma r <beeshma48@gmail.com> wrote:
>
>> Hi Pankil,
>>
>> Are you sure your hbase is running with external zookeeper ensemble ?
>>
>> As per documentation on Hbase Replication
>>
>>
>> http://www.cloudera.com/content/www/en-us/documentation/archive/cdh/4-x/4-2-0/CDH4-Installation-Guide/cdh4ig_topic_20_11.html
>>
>> zookeeper must not be managed by HBase,.But i havent tried this
>>
>> On Fri, Oct 23, 2015 at 9:55 AM, Ashish Singhi <
>> ashish.singhi.apache@gmail.com> wrote:
>>
>> > Hi Pankil.
>> >
>> > A similar issue was reported few days back (
>> >
>> >
>> http://search-hadoop.com/m/YGbbknQt52rKBDS1&subj=HRegionServer+failed+due+to+replication
>> > ).
>> >
>> > May be this is due to hbase-indexer code ?
>> > One more Q, did you upgrade hbase from 0.94 and you see this issue ?
>> >
>> > Regards,
>> > Ashish Singhi
>> >
>> > On Fri, Oct 23, 2015 at 2:47 AM, Pankil Doshi <forpankil@gmail.com>
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > I am using hbase-0.98.15-hadoop2 and hbase-indexer from lily (
>> > > http://ngdata.github.io/hbase-indexer/).
>> > >
>> > > I am seeing below error when I add my indexer:
>> > >
>> > >
>> > > 2015-10-22 14:08:27,468 INFO  [regionserver60020-EventThread]
>> > > replication.ReplicationTrackerZKImpl: /hbase/replication/peers znode
>> > > expired, triggering peerListChanged event
>> > >
>> > > 2015-10-22 14:08:27,473 ERROR [regionserver60020-EventThread]
>> > > regionserver.ReplicationSourceManager: Error while adding a new peer
>> > >
>> > > org.apache.hadoop.hbase.replication.ReplicationException: Error adding
>> > peer
>> > > with id=Indexer_newtest2
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createAndAddPeer(ReplicationPeersZKImpl.java:386)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.peerAdded(ReplicationPeersZKImpl.java:358)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.peerListChanged(ReplicationSourceManager.java:514)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$PeersWatcher.nodeChildrenChanged(ReplicationTrackerZKImpl.java:189)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:468)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>> > >
>> > > at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>> > >
>> > > Caused by: org.apache.hadoop.hbase.replication.ReplicationException:
>> > Error
>> > > starting the peer state tracker for peerId=Indexer_newtest2
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:454)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createAndAddPeer(ReplicationPeersZKImpl.java:384)
>> > >
>> > > ... 6 more
>> > >
>> > > Caused by:
>> > org.apache.zookeeper.KeeperException$DataInconsistencyException:
>> > > KeeperErrorCode = DataInconsistency
>> > >
>> > > at org.apache.hadoop.hbase.zookeeper.ZKUtil.convert(ZKUtil.java:2063)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.startStateTracker(ReplicationPeerZKImpl.java:85)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:452)
>> > >
>> > > ... 7 more
>> > >
>> > > Caused by:
>> org.apache.hadoop.hbase.exceptions.DeserializationException:
>> > > Missing pb magic PBUF prefix
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.protobuf.ProtobufUtil.expectPBMagicPrefix(ProtobufUtil.java:270)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.parseStateFrom(ReplicationPeerZKImpl.java:243)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.isStateEnabled(ReplicationPeerZKImpl.java:232)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.readPeerStateZnode(ReplicationPeerZKImpl.java:90)
>> > >
>> > > at
>> > >
>> > >
>> >
>> org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.startStateTracker(ReplicationPeerZKImpl.java:83)
>> > >
>> > > ... 8 more
>> > >
>> > >
>> > >
>> > > My Hbase-site.xml:
>> > >
>> > >
>> > > <?xml version="1.0"?>
>> > >
>> > > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> > >
>> > > <!--
>> > >
>> > > /**
>> > >
>> > >  * Copyright 2010 The Apache Software Foundation
>> > >
>> > >  *
>> > >
>> > >  * Licensed to the Apache Software Foundation (ASF) under one
>> > >
>> > >  * or more contributor license agreements.  See the NOTICE file
>> > >
>> > >  * distributed with this work for additional information
>> > >
>> > >  * regarding copyright ownership.  The ASF licenses this file
>> > >
>> > >  * to you under the Apache License, Version 2.0 (the
>> > >
>> > >  * "License"); you may not use this file except in compliance
>> > >
>> > >  * with the License.  You may obtain a copy of the License at
>> > >
>> > >  *
>> > >
>> > >  *     http://www.apache.org/licenses/LICENSE-2.0
>> > >
>> > >  *
>> > >
>> > >  * Unless required by applicable law or agreed to in writing, software
>> > >
>> > >  * distributed under the License is distributed on an "AS IS" BASIS,
>> > >
>> > >  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>> > implied.
>> > >
>> > >  * See the License for the specific language governing permissions and
>> > >
>> > >  * limitations under the License.
>> > >
>> > >  */
>> > >
>> > > -->
>> > >
>> > > <configuration>
>> > >
>> > > <!--<property>
>> > >
>> > >                 <name>zookeeper.znode.parent</name>
>> > >
>> > >                 <value>/hbase-unsecure</value>
>> > >
>> > >         </property>-->
>> > >
>> > >
>> > >         <property>
>> > >
>> > >                 <name>hbase.cluster.distributed</name>
>> > >
>> > >                 <value>true</value>
>> > >
>> > >         </property>
>> > >
>> > > //Here you have to set the path where you want HBase to store its
>> files.
>> > >
>> > >    <property>
>> > >
>> > >       <name>hbase.rootdir</name>
>> > >
>> > >       <value>file:/tmp/HBase/HFiles</value>
>> > >
>> > >    </property>
>> > >
>> > > <property>
>> > >
>> > >       <name>hbase.zookeeper.property.clientPort</name>
>> > >
>> > >       <value>2181</value>
>> > >
>> > >       <description>Property from ZooKeeper's config zoo.cfg.
>> > >
>> > >       The port at which the clients will connect.
>> > >
>> > >       </description>
>> > >
>> > >     </property>
>> > >
>> > >     <property>
>> > >
>> > >       <name>hbase.zookeeper.quorum</name>
>> > >
>> > >       <value>localhost</value>
>> > >
>> > >       <description>Comma separated list of servers in the ZooKeeper
>> > Quorum.
>> > >
>> > >       For example, "host1.mydomain.com,host2.mydomain.com,
>> > > host3.mydomain.com
>> > > ".
>> > >
>> > >       By default this is set to localhost for local and
>> > pseudo-distributed
>> > > modes
>> > >
>> > >       of operation. For a fully-distributed setup, this should be set
>> to
>> > a
>> > > full
>> > >
>> > >       list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in
>> > > hbase-env.sh
>> > >
>> > >       this is the list of servers which we will start/stop ZooKeeper
>> on.
>> > >
>> > >       </description>
>> > >
>> > >     </property>
>> > >
>> > > <property>
>> > >
>> > >    <name>hbase.zookeeper.property.dataDir</name>
>> > >
>> > >    <value>/tmp/zookeeper</value>
>> > >
>> > >    <description>Property from ZooKeeper config zoo.cfg.
>> > >
>> > >    The directory where the snapshot is stored.
>> > >
>> > >    </description>
>> > >
>> > > </property>
>> > >
>> > > // FOR HBASE INDEXER
>> > >
>> > >
>> > > <!-- SEP is basically replication, so enable it -->
>> > >
>> > >   <property>
>> > >
>> > >     <name>hbase.replication</name>
>> > >
>> > >     <value>true</value>
>> > >
>> > >   </property>
>> > >
>> > >   <!-- Source ratio of 100% makes sure that each SEP consumer is
>> actually
>> > >
>> > >        used (otherwise, some can sit idle, especially with small
>> > clusters)
>> > > -->
>> > >
>> > >   <property>
>> > >
>> > >     <name>replication.source.ratio</name>
>> > >
>> > >     <value>1.0</value>
>> > >
>> > >   </property>
>> > >
>> > >   <!-- Maximum number of hlog entries to replicate in one go. If this
>> is
>> > >
>> > >        large, and a consumer takes a while to process the events, the
>> > >
>> > >        HBase rpc call will time out. -->
>> > >
>> > >   <property>
>> > >
>> > >     <name>replication.source.nb.capacity</name>
>> > >
>> > >     <value>1000</value>
>> > >
>> > >   </property>
>> > >
>> > >   <!-- A custom replication source that fixes a few things and adds
>> > >
>> > >        some functionality (doesn't interfere with normal replication
>> > >
>> > >        usage). -->
>> > >
>> > >   <property>
>> > >
>> > >     <name>replication.replicationsource.implementation</name>
>> > >
>> > >     <value>com.ngdata.sep.impl.SepReplicationSource</value>
>> > >
>> > >   </property>
>> > >
>> > > </configuration>
>> > >
>> > >
>> > > All research so far shows me one can face problem while upgrading
>> hbase.
>> > > But in my case, I am not upgrading.
>> > >
>> > > Any ideas/suggestions will be very helpful.
>> > >
>> > > Thanks,
>> > > Pankil
>> > >
>> >
>>
>>
>>
>> --
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message