Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of vivekrishna@gmail.com
 designates 74.125.82.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=fRgzSRrmvUzhkhl7fsQ6vxTQe7GLLstu53RlOtvJxaSOJSyWK5wlhi4tfuqponPO2u
         MOfqyrTOJ5gBGZIK/8kNPwSQLeugvaGs4b40kLX2LPtIdIpND+l5/rMMVk5C+bk88tl/
         oKiXx2F53RxlH6AzNy5sq+f6xE2RCRaIo/u2U=
MIME-Version: 1.0
In-Reply-To: <AANLkTikLoZtgpAakSJXTcpK-Be93V4Ly0AwYf4LKqGNW@mail.gmail.com>
References: <AANLkTim5HEPObuRUAMrFBuaGd3GaFNSTRNyHZH4za21Q@mail.gmail.com>
 <AANLkTin=kpkPcjohj6egxiEvU3SJxM7+DySe8n6heU-B@mail.gmail.com>
 <4D812189.9010207@qualtrics.com>
 <AANLkTikr9G_zRBV0KD3MWYvqQ5Bw5dcgPqHhW+L5sp5t@mail.gmail.com>
 <AANLkTi==G7-p2cbn8B1_CPvGt8ZALFL7v99yKk9HvytF@mail.gmail.com>
 <AANLkTikLoZtgpAakSJXTcpK-Be93V4Ly0AwYf4LKqGNW@mail.gmail.com>
From: Vivek Krishna <vivekrishna@gmail.com>
Date: Wed, 16 Mar 2011 18:18:01 -0400
Message-ID: <AANLkTimR394H0Lqm7YBBn1s1dVCUBm7tuh65O8fpUnsO@mail.gmail.com>
Subject: Re: Row Counters
To: user@hbase.apache.org
Cc: Ted Yu <yuzhihong@gmail.com>
Content-Type: multipart/alternative; boundary=00504502d9e82de423049ea0ec65

--00504502d9e82de423049ea0ec65
Content-Type: text/plain; charset=ISO-8859-1

Oops. sorry about the environment.

I am using hadoop-0.20.2-CDH3B4, and hbase-0.90.1-CDH3B4
and zookeeper-3.3.2-CDH3B4.

I was able to configure jars and run the command,

hadoop jar /usr/lib/hbase/hbase-0.90.1-CDH3B4.jar rowcounter test,

but I get

java.io.IOException: Cannot create a record reader because of a
previous error. Please look at the previous logs lines from the task's
full log for more details.
	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:98)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:240)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
	at org.apache.hadoop.mapred.Child.main(Child.java:234)


The previous error in the task's full log is ..

2011-03-16 21:41:03,367 ERROR
org.apache.hadoop.hbase.mapreduce.TableInputFormat:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:292)
	at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:155)
	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
	at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:91)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:240)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
	at org.apache.hadoop.mapred.Child.main(Child.java:234)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:986)
	... 15 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
	... 16 more


find I am pretty sure zookeeper master is running in the same machine at
port 2181.  Not sure why the connection loss occurs.  Do I need
HBASE-3578<https://issues.apache.org/jira/browse/HBASE-3578>by any
chance?

Viv


On Wed, Mar 16, 2011 at 5:36 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> In the future, describe your environment a bit.
>
> The way I approach this is:
> find the correct commandline from
> src/main/java/org/apache/hadoop/hbase/mapreduce/package-info.java
>
> Then I issue:
> [hadoop@us01-ciqps1-name01 hbase]$
> HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase
> classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-0.90.1.jar
> rowcounter packageindex
>
> Then I check the map/reduce task on job tracker URL
>
> On Wed, Mar 16, 2011 at 1:59 PM, Vivek Krishna <vivekrishna@gmail.com
> >wrote:
>
> > I guess it is using the mapred class
> >
> > 11/03/16 20:58:27 INFO mapred.JobClient: Task Id :
> > attempt_201103161245_0005_m_000004_0, Status : FAILED
> > java.io.IOException: Cannot create a record reader because of a previous
> > error. Please look at the previous logs lines from the task's full log
> for
> > more details.
> >  at
> >
> >
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:98)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:240)
> >  at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> >  at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> > at org.apache.hadoop.mapred.Child.main(Child.java:234)
> >
> > How do I use mapreduce class?
> > Viv
> >
> >
> >
> > On Wed, Mar 16, 2011 at 4:52 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > Since we have lived so long without this information, I guess we can
> hold
> > > for longer :-)
> > > Another issue I am working on is to reduce memory footprint. See the
> > > following discussion thread:
> > > One of the regionserver aborted, then the master shut down itself
> > >
> > > We have to bear in mind that there would be around 10K regions or more
> in
> > > production.
> > >
> > > Cheers
> > >
> > > On Wed, Mar 16, 2011 at 1:46 PM, Jeff Whiting <jeffw@qualtrics.com>
> > wrote:
> > >
> > > > Just a random thought.  What about keeping a per region row count?
> >  Then
> > > if
> > > > you needed to get a row count for a table you'd just have to query
> each
> > > > region once and sum.  Seems like it wouldn't be too expensive because
> > > you'd
> > > > just have a row counter variable.  It maybe more complicated than I'm
> > > making
> > > > it out to be though...
> > > >
> > > > ~Jeff
> > > >
> > > >
> > > > On 3/16/2011 2:40 PM, Stack wrote:
> > > >
> > > >> On Wed, Mar 16, 2011 at 1:35 PM, Vivek Krishna<
> vivekrishna@gmail.com>
> > > >>  wrote:
> > > >>
> > > >>> 1.  How do I count rows fast in hbase?
> > > >>>
> > > >>> First I tired count 'test'  , takes ages.
> > > >>>
> > > >>> Saw that I could use RowCounter, but looks like it is deprecated.
> > > >>>
> > > >> It is not.  Make sure you are using the one from mapreduce package
> as
> > > >> opposed to mapred package.
> > > >>
> > > >>
> > > >>  I just need to verify the total counts.  Is it possible to see
> > > somewhere
> > > >>> in
> > > >>> the web interface or ganglia or by any other means?
> > > >>>
> > > >>>  We don't keep a current count on a table.  Too expensive.  Run the
> > > >> rowcounter MR job.  This page may be of help:
> > > >>
> > > >>
> > >
> >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description
> > > >>
> > > >> Good luck,
> > > >> St.Ack
> > > >>
> > > >
> > > > --
> > > > Jeff Whiting
> > > > Qualtrics Senior Software Engineer
> > > > jeffw@qualtrics.com
> > > >
> > > >
> > >
> >
>

--00504502d9e82de423049ea0ec65--