Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of kevin.odell@cloudera.com
 designates 209.85.216.49 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPQV63WpX9PZZv=1maYc9a0iiZYPdi++N9_4rbgWu5DgV6+_nQ@mail.gmail.com>
References: <loom.20131210T142636-17@post.gmane.org>
	<CAPQV63WpX9PZZv=1maYc9a0iiZYPdi++N9_4rbgWu5DgV6+_nQ@mail.gmail.com>
Date: Tue, 10 Dec 2013 09:17:17 -0500
Message-ID: 
 <CAGngS9euETES4JVWgd2YsxdxGA=Ht-xZc1wVvce2eVcfg=8qZA@mail.gmail.com>
Subject: Re: Problem When Prefetch META table
From: "Kevin O'dell" <kevin.odell@cloudera.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c2bd38146c6004ed2ec5e2

--001a11c2bd38146c6004ed2ec5e2
Content-Type: text/plain; charset=ISO-8859-1

Hey Andrea,

  That is very weird, I hit this similar extremely strange one off issue
that I could never reproduce.  After dropping a table doing X amount of
inserts, I would receive the same error and my META table would be
corrupted.  What I did to correct this issue was to sideline the META table
and then use offlineMetaRepair to rebuild it, and then I never ran into it
again.  It was quite strange and I am sorry I don't have more to offer.


On Tue, Dec 10, 2013 at 8:34 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> Hi Andrea,
>
> Which version of HBase are you testing with?
>
> JM
>
>
> 2013/12/10 Andrea <andrea.gandini.89@gmail.com>
>
> > Hi, I encounter this problem:
> >
> > 13/12/10 13:25:15 WARN
> client.HConnectionManager$HConnectionImplementation:
> > Encountered problems when prefetch META table:
> > java.io.IOException: HRegionInfo was null or empty in Meta for usertable,
> > row=usertable,user4300292263938371081,99999999999999
> >         at
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:170)
> >         at
> >
> org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:54)
> >         at
> >
> org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
> >         at
> >
> org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager
> > .java:360)
> >         at
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130)
> >         at
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > prefetchRegionCache(HConnectionManager.java:918)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > locateRegionInMeta(HConnectionManager.java:973)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > locateRegion(HConnectionManager.java:860)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > locateRegion(HConnectionManager.java:817)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > processBatchCallback(HConnectionManager.java:1507)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > processBatch(HConnectionManager.java:1392)
> >         at
> > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:918)
> >         at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:774)
> >         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:749)
> >         at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:328)
> >         at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:357)
> >         at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148)
> >         at
> > com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461)
> >         at com.yahoo.ycsb.ClientThread.run(Client.java:269)
> >
> > It will happen when, on a netowrk with one zookeeper, one master and one
> > regionserver I try to load a lot of data. This is the number of
> operations
> > before the error:
> >
> > 200 sec: 888708 operations; 7767.62 current ops/sec; [UPDATE
> > AverageLatency(us)=27089963] [INSERT AverageLatency(us)=9738.99] [CLEANUP
> > AverageLatency(us)=27089976]
> >  210 sec: 893742 operations; 502.7 current ops/sec; [UPDATE
> > AverageLatency(us)=14887298.5] [INSERT AverageLatency(us)=6937.27]
> [CLEANUP
> > AverageLatency(us)=14887312.5]
> >  221 sec: 928277 operations; 3226.97 current ops/sec; [UPDATE
> > AverageLatency(us)=27006074] [INSERT AverageLatency(us)=8529.21] [CLEANUP
> > AverageLatency(us)=27006089]
> >  231 sec: 930079 operations; 180.18 current ops/sec;  [INSERT
> > AverageLatency(us)=457.08]
> > 13/12/10 13:25:13 WARN
> client.HConnectionManager$HConnectionImplementation:
> > Encountered problems when prefetch META table: ...
> >
> > Does this depend on the number of thread in whereby I insert the data?
> >
> >
>


-- 
Kevin O'Dell
Systems Engineer, Cloudera

--001a11c2bd38146c6004ed2ec5e2--