hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish Singhi <ashish.singhi.apa...@gmail.com>
Subject Re: org.apache.hadoop.hbase.exceptions.DeserializationException: Missing pb magic PBUF prefix
Date Sat, 24 Oct 2015 07:24:52 GMT
As you mentioned in one of the previous mail that this issue is due to
hbase-indexer code.
Can you post your problem on ngdata forum.

Regards,
Ashish Singhi

On Sat, Oct 24, 2015 at 2:31 AM, Pankil Doshi <forpankil@gmail.com> wrote:

> I tried setting up using hdfs, still I have the same issue.
>
> On Fri, Oct 23, 2015 at 11:28 AM, Pankil Doshi <forpankil@gmail.com>
> wrote:
>
> > One other thing which is different in my setup is I am using filesystem
> > for hbase rather hbase-indexer needs hdfs setup to be fully functional.
> So
> > I need to change that.
> >
> > Also, do you have local hbase setup or hbase cluster mode ?
> >
> > Pankil
> >
> > On Fri, Oct 23, 2015 at 11:23 AM, Pankil Doshi <forpankil@gmail.com>
> > wrote:
> >
> >> Hi Beeshma,
> >>
> >> Thanks for your response.
> >>
> >> I am running zookeeper locally. but I am not managing it with hbase i.e
> I
> >> have this set:
> >> export HBASE_MANAGES_ZK=false
> >>
> >> and also:
> >> <property>
> >>                 <name>hbase.cluster.distributed</name>
> >>                 <value>true</value>
> >>         </property>
> >>
> >> even though I have everything running locally in standalone mode.
> >>
> >> If I dont set "hbase.cluster.distributed"  I am seeing zookeeper being
> >> started with start of my hbase. I am not sure if there is any other good
> >> way not to start or stop zookeeper with hbase as by only setting
> >>  (HBASE_MANAGES_ZK=false) it doesnt work.
> >>
> >> Were you able to setup hbase-indexer at all ?
> >>
> >> Pankil
> >>
> >>
> >> On Fri, Oct 23, 2015 at 10:51 AM, beeshma r <beeshma48@gmail.com>
> wrote:
> >>
> >>> Hi Pankil,
> >>>
> >>> Are you sure your hbase is running with external zookeeper ensemble ?
> >>>
> >>> As per documentation on Hbase Replication
> >>>
> >>>
> >>>
> http://www.cloudera.com/content/www/en-us/documentation/archive/cdh/4-x/4-2-0/CDH4-Installation-Guide/cdh4ig_topic_20_11.html
> >>>
> >>> zookeeper must not be managed by HBase,.But i havent tried this
> >>>
> >>> On Fri, Oct 23, 2015 at 9:55 AM, Ashish Singhi <
> >>> ashish.singhi.apache@gmail.com> wrote:
> >>>
> >>> > Hi Pankil.
> >>> >
> >>> > A similar issue was reported few days back (
> >>> >
> >>> >
> >>>
> http://search-hadoop.com/m/YGbbknQt52rKBDS1&subj=HRegionServer+failed+due+to+replication
> >>> > ).
> >>> >
> >>> > May be this is due to hbase-indexer code ?
> >>> > One more Q, did you upgrade hbase from 0.94 and you see this issue
?
> >>> >
> >>> > Regards,
> >>> > Ashish Singhi
> >>> >
> >>> > On Fri, Oct 23, 2015 at 2:47 AM, Pankil Doshi <forpankil@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Hi,
> >>> > >
> >>> > > I am using hbase-0.98.15-hadoop2 and hbase-indexer from lily (
> >>> > > http://ngdata.github.io/hbase-indexer/).
> >>> > >
> >>> > > I am seeing below error when I add my indexer:
> >>> > >
> >>> > >
> >>> > > 2015-10-22 14:08:27,468 INFO  [regionserver60020-EventThread]
> >>> > > replication.ReplicationTrackerZKImpl: /hbase/replication/peers
> znode
> >>> > > expired, triggering peerListChanged event
> >>> > >
> >>> > > 2015-10-22 14:08:27,473 ERROR [regionserver60020-EventThread]
> >>> > > regionserver.ReplicationSourceManager: Error while adding a new
> peer
> >>> > >
> >>> > > org.apache.hadoop.hbase.replication.ReplicationException: Error
> >>> adding
> >>> > peer
> >>> > > with id=Indexer_newtest2
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createAndAddPeer(ReplicationPeersZKImpl.java:386)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.peerAdded(ReplicationPeersZKImpl.java:358)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.peerListChanged(ReplicationSourceManager.java:514)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$PeersWatcher.nodeChildrenChanged(ReplicationTrackerZKImpl.java:189)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:468)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> >>> > >
> >>> > > at
> >>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> >>> > >
> >>> > > Caused by:
> org.apache.hadoop.hbase.replication.ReplicationException:
> >>> > Error
> >>> > > starting the peer state tracker for peerId=Indexer_newtest2
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:454)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createAndAddPeer(ReplicationPeersZKImpl.java:384)
> >>> > >
> >>> > > ... 6 more
> >>> > >
> >>> > > Caused by:
> >>> > org.apache.zookeeper.KeeperException$DataInconsistencyException:
> >>> > > KeeperErrorCode = DataInconsistency
> >>> > >
> >>> > > at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.convert(ZKUtil.java:2063)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.startStateTracker(ReplicationPeerZKImpl.java:85)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:452)
> >>> > >
> >>> > > ... 7 more
> >>> > >
> >>> > > Caused by:
> >>> org.apache.hadoop.hbase.exceptions.DeserializationException:
> >>> > > Missing pb magic PBUF prefix
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.expectPBMagicPrefix(ProtobufUtil.java:270)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.parseStateFrom(ReplicationPeerZKImpl.java:243)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.isStateEnabled(ReplicationPeerZKImpl.java:232)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.readPeerStateZnode(ReplicationPeerZKImpl.java:90)
> >>> > >
> >>> > > at
> >>> > >
> >>> > >
> >>> >
> >>>
> org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.startStateTracker(ReplicationPeerZKImpl.java:83)
> >>> > >
> >>> > > ... 8 more
> >>> > >
> >>> > >
> >>> > >
> >>> > > My Hbase-site.xml:
> >>> > >
> >>> > >
> >>> > > <?xml version="1.0"?>
> >>> > >
> >>> > > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >>> > >
> >>> > > <!--
> >>> > >
> >>> > > /**
> >>> > >
> >>> > >  * Copyright 2010 The Apache Software Foundation
> >>> > >
> >>> > >  *
> >>> > >
> >>> > >  * Licensed to the Apache Software Foundation (ASF) under one
> >>> > >
> >>> > >  * or more contributor license agreements.  See the NOTICE file
> >>> > >
> >>> > >  * distributed with this work for additional information
> >>> > >
> >>> > >  * regarding copyright ownership.  The ASF licenses this file
> >>> > >
> >>> > >  * to you under the Apache License, Version 2.0 (the
> >>> > >
> >>> > >  * "License"); you may not use this file except in compliance
> >>> > >
> >>> > >  * with the License.  You may obtain a copy of the License at
> >>> > >
> >>> > >  *
> >>> > >
> >>> > >  *     http://www.apache.org/licenses/LICENSE-2.0
> >>> > >
> >>> > >  *
> >>> > >
> >>> > >  * Unless required by applicable law or agreed to in writing,
> >>> software
> >>> > >
> >>> > >  * distributed under the License is distributed on an "AS IS"
> BASIS,
> >>> > >
> >>> > >  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
or
> >>> > implied.
> >>> > >
> >>> > >  * See the License for the specific language governing permissions
> >>> and
> >>> > >
> >>> > >  * limitations under the License.
> >>> > >
> >>> > >  */
> >>> > >
> >>> > > -->
> >>> > >
> >>> > > <configuration>
> >>> > >
> >>> > > <!--<property>
> >>> > >
> >>> > >                 <name>zookeeper.znode.parent</name>
> >>> > >
> >>> > >                 <value>/hbase-unsecure</value>
> >>> > >
> >>> > >         </property>-->
> >>> > >
> >>> > >
> >>> > >         <property>
> >>> > >
> >>> > >                 <name>hbase.cluster.distributed</name>
> >>> > >
> >>> > >                 <value>true</value>
> >>> > >
> >>> > >         </property>
> >>> > >
> >>> > > //Here you have to set the path where you want HBase to store
its
> >>> files.
> >>> > >
> >>> > >    <property>
> >>> > >
> >>> > >       <name>hbase.rootdir</name>
> >>> > >
> >>> > >       <value>file:/tmp/HBase/HFiles</value>
> >>> > >
> >>> > >    </property>
> >>> > >
> >>> > > <property>
> >>> > >
> >>> > >       <name>hbase.zookeeper.property.clientPort</name>
> >>> > >
> >>> > >       <value>2181</value>
> >>> > >
> >>> > >       <description>Property from ZooKeeper's config zoo.cfg.
> >>> > >
> >>> > >       The port at which the clients will connect.
> >>> > >
> >>> > >       </description>
> >>> > >
> >>> > >     </property>
> >>> > >
> >>> > >     <property>
> >>> > >
> >>> > >       <name>hbase.zookeeper.quorum</name>
> >>> > >
> >>> > >       <value>localhost</value>
> >>> > >
> >>> > >       <description>Comma separated list of servers in the
ZooKeeper
> >>> > Quorum.
> >>> > >
> >>> > >       For example, "host1.mydomain.com,host2.mydomain.com,
> >>> > > host3.mydomain.com
> >>> > > ".
> >>> > >
> >>> > >       By default this is set to localhost for local and
> >>> > pseudo-distributed
> >>> > > modes
> >>> > >
> >>> > >       of operation. For a fully-distributed setup, this should
be
> >>> set to
> >>> > a
> >>> > > full
> >>> > >
> >>> > >       list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is
set
> in
> >>> > > hbase-env.sh
> >>> > >
> >>> > >       this is the list of servers which we will start/stop
> ZooKeeper
> >>> on.
> >>> > >
> >>> > >       </description>
> >>> > >
> >>> > >     </property>
> >>> > >
> >>> > > <property>
> >>> > >
> >>> > >    <name>hbase.zookeeper.property.dataDir</name>
> >>> > >
> >>> > >    <value>/tmp/zookeeper</value>
> >>> > >
> >>> > >    <description>Property from ZooKeeper config zoo.cfg.
> >>> > >
> >>> > >    The directory where the snapshot is stored.
> >>> > >
> >>> > >    </description>
> >>> > >
> >>> > > </property>
> >>> > >
> >>> > > // FOR HBASE INDEXER
> >>> > >
> >>> > >
> >>> > > <!-- SEP is basically replication, so enable it -->
> >>> > >
> >>> > >   <property>
> >>> > >
> >>> > >     <name>hbase.replication</name>
> >>> > >
> >>> > >     <value>true</value>
> >>> > >
> >>> > >   </property>
> >>> > >
> >>> > >   <!-- Source ratio of 100% makes sure that each SEP consumer
is
> >>> actually
> >>> > >
> >>> > >        used (otherwise, some can sit idle, especially with small
> >>> > clusters)
> >>> > > -->
> >>> > >
> >>> > >   <property>
> >>> > >
> >>> > >     <name>replication.source.ratio</name>
> >>> > >
> >>> > >     <value>1.0</value>
> >>> > >
> >>> > >   </property>
> >>> > >
> >>> > >   <!-- Maximum number of hlog entries to replicate in one go.
If
> >>> this is
> >>> > >
> >>> > >        large, and a consumer takes a while to process the events,
> the
> >>> > >
> >>> > >        HBase rpc call will time out. -->
> >>> > >
> >>> > >   <property>
> >>> > >
> >>> > >     <name>replication.source.nb.capacity</name>
> >>> > >
> >>> > >     <value>1000</value>
> >>> > >
> >>> > >   </property>
> >>> > >
> >>> > >   <!-- A custom replication source that fixes a few things
and adds
> >>> > >
> >>> > >        some functionality (doesn't interfere with normal
> replication
> >>> > >
> >>> > >        usage). -->
> >>> > >
> >>> > >   <property>
> >>> > >
> >>> > >     <name>replication.replicationsource.implementation</name>
> >>> > >
> >>> > >     <value>com.ngdata.sep.impl.SepReplicationSource</value>
> >>> > >
> >>> > >   </property>
> >>> > >
> >>> > > </configuration>
> >>> > >
> >>> > >
> >>> > > All research so far shows me one can face problem while upgrading
> >>> hbase.
> >>> > > But in my case, I am not upgrading.
> >>> > >
> >>> > > Any ideas/suggestions will be very helpful.
> >>> > >
> >>> > > Thanks,
> >>> > > Pankil
> >>> > >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message