Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates
 209.85.221.188 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=d7t6RkWD5PqR3mFoIVBgTgra1zbKR3SrF86zRnxM2xdyeHvx8bdsQdAphxBeeR6Lsc
         8P9JNDf8Hf0AnqljpHuuQCYiQ2w4DTRDsUGuFZOwYutztvuKcNkxqsYWEqf/7Cgy6zFP
         8sDwzfpIQWcmXikPvSP4HlMoa6NB6QZhHtR7w=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <24977400.post@talk.nabble.com>
References: <24920888.post@talk.nabble.com>
	 <00df01ca1aa7$bdc3b220$394b1660$@com> <24927386.post@talk.nabble.com>
	 <78568af10908112045r3fefb83ehb70f564fda02a4b9@mail.gmail.com>
	 <4A824533.3040404@lifeless.net> <24939515.post@talk.nabble.com>
	 <7c962aed0908120906j5d20fb19g9a153cb32ca7dbf3@mail.gmail.com>
	 <24975031.post@talk.nabble.com>
	 <7c962aed0908141002s5f53ea19q3b81664ade17b5ff@mail.gmail.com>
	 <24977400.post@talk.nabble.com>
Date: Fri, 14 Aug 2009 14:08:36 -0700
Message-ID: <7c962aed0908141408q1d719377xc9067db90e5f01f7@mail.gmail.com>
Subject: Re: HBase in a real world application
From: stack <stack@duboce.net>
To: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363ba25aa4fa68047120749b

--0016363ba25aa4fa68047120749b
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Thanks for trying.  Looks like that region is now gone (split is my guess).
Check the master log for mentions of this region to see its history.  Can
you correlate the client failure with an event on this region in master
log?  It looks like client was being pig-headed fixated on the parent of a
split.  You could check your table is healthy?  Run the rowcounter program
to make sure no holes in table?

St.Ack

On Fri, Aug 14, 2009 at 12:41 PM, llpind <sonny_heer@hotmail.com> wrote:

>
> hbase(main):003:0> get '.META.', 'TestTable,0001749889,1250092414985',
> {COLUMNS =3D>'info'}
> 09/08/14 12:28:10 DEBUG client.HConnectionManager$TableServers: Cache hit
> for row <> in tableName .META.: location server 192.168.0.196:60020,
> location region name .META.,,1
> NativeException: java.lang.NullPointerException: null
>        from org/apache/hadoop/hbase/client/HTable.java:789:in `get'
>        from org/apache/hadoop/hbase/client/HTable.java:769:in `get'
>        from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
>        from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
>        from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>        from java/lang/reflect/Method.java:597:in `invoke'
>        from org/jruby/javasupport/JavaMethod.java:298:in
> `invokeWithExceptionHandling'
>        from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
>        from org/jruby/java/invokers/InstanceMethodInvoker.java:30:in `cal=
l'
>        from org/jruby/runtime/callsite/CachingCallSite.java:30:in `call'
>        from org/jruby/ast/CallManyArgsNode.java:59:in `interpret'
>        from org/jruby/ast/LocalAsgnNode.java:123:in `interpret'
>        from org/jruby/ast/NewlineNode.java:104:in `interpret'
>        from org/jruby/ast/IfNode.java:112:in `interpret'
>        from org/jruby/ast/NewlineNode.java:104:in `interpret'
>        from org/jruby/ast/IfNode.java:114:in `interpret'
> ... 115 levels...
>        from
> home/hadoop/hbase_minus_0_dot_20_dot_0/bin/$_dot_dot_/bin/hirb#start:-1:i=
n
> `call'
>        from org/jruby/internal/runtime/methods/DynamicMethod.java:226:in
> `call'
>        from org/jruby/internal/runtime/methods/CompiledMethod.java:211:in
> `call'
>        from org/jruby/internal/runtime/methods/CompiledMethod.java:71:in
> `call'
>        from org/jruby/runtime/callsite/CachingCallSite.java:253:in
> `cacheAndCall'
>        from org/jruby/runtime/callsite/CachingCallSite.java:72:in `call'
>        from
> home/hadoop/hbase_minus_0_dot_20_dot_0/bin/$_dot_dot_/bin/hirb.rb:487:in
> `__file__'
>        from
> home/hadoop/hbase_minus_0_dot_20_dot_0/bin/$_dot_dot_/bin/hirb.rb:-1:in
> `load'
>        from org/jruby/Ruby.java:577:in `runScript'
>        from org/jruby/Ruby.java:480:in `runNormally'
>        from org/jruby/Ruby.java:354:in `runFromMain'
>        from org/jruby/Main.java:229:in `run'
>        from org/jruby/Main.java:110:in `run'
>        from org/jruby/Main.java:94:in `main'
>        from /home/hadoop/hbase-0.20.0/bin/../bin/hirb.rb:384:in `get'
>        from (hbase):4hbase(main):004:0> get '.META.',
> 'TestTable,0001749889,1250092414985'
> 09/08/14 12:28:13 DEBUG client.HConnectionManager$TableServers: Cache hit
> for row <> in tableName .META.: location server 192.168.0.196:60020,
> location region name .META.,,1
> COLUMN                       CELL
>  historian:assignment        timestamp=3D1250108456441, value=3DRegion as=
signed
> to server server195,60020,12501083767
>                             79
>  historian:compaction        timestamp=3D1250109313965, value=3DRegion
> compaction completed in 35sec
>  historian:open              timestamp=3D1250108459484, value=3DRegion op=
ened
> on
> server : server195
>  historian:split             timestamp=3D1250092447915, value=3DRegion sp=
lit
> from: TestTable,0001634945,1250035163
>                             027
>  info:regioninfo             timestamp=3D1250109315260, value=3DREGION =
=3D> {NAME
> =3D> 'TestTable,0001749889,125009241
>                             4985', STARTKEY =3D> '0001749889', ENDKEY =3D=
>
> '0001866010', ENCODED =3D> 1707908074, O
>                             FFLINE =3D> true, TABLE =3D> {{NAME =3D> 'Tes=
tTable',
> FAMILIES =3D> [{NAME =3D> 'info', VER
>                             SIONS =3D> '3', COMPRESSION =3D> 'NONE', TTL =
=3D>
> '2147483647', BLOCKSIZE =3D> '65536', I
>                             N_MEMORY =3D> 'false', BLOCKCACHE =3D> 'true'=
}]}}
>
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> stack-3 wrote:
> >
> > Is that region offline?
> >
> > Do a:
> >
> > hbase> get ".META.", "TestTable,0001749889,1250092414985", {COLUMNS =3D=
>
> > "info"}.
> >
> > If so, can you get its history so we can figure how it went offline? (S=
ee
> > region history in UI or grep it in master logs?)
> >
> > St.Ack
> >
> >
> > On Fri, Aug 14, 2009 at 9:55 AM, llpind <sonny_heer@hotmail.com> wrote:
> >
> >>
> >> Hey Stack,  I tried the following command:
> >>
> >> hadoop-0.20.0/bin/hadoop jar hbase-0.20.0/hbase-0.20.0-test.jar
> >>  randomWrite
> >> 10
> >>
> >> running a map/reduce job, it failed with the following exceptions in
> each
> >> node:
> >>
> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> >> contact
> >> region server Some server for region , row '0001753186', but failed
> after
> >> 11
> >> attempts.
> >> Exceptions:
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >>
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionL=
ocationForRowWithRetries(HConnectionManager.java:995)
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBat=
chOfRows(HConnectionManager.java:1025)
> >>        at
> >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584)
> >>        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450)
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest.testRow(Per=
formanceEvaluation.java:497)
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvalua=
tion.java:406)
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEva=
luation.java:627)
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(Perfo=
rmanceEvaluation.java:194)
> >>        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >>        at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356)
> >>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >>
> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >>
> >> This appears to be this issue:
> >>
> >> http://issues.apache.org/jira/browse/HBASE-1603
> >>
> >>
> >> Has this been fixed in .20?  Thanks.
> >>
> >>
> >>
> >> stack-3 wrote:
> >> >
> >> > On Wed, Aug 12, 2009 at 8:58 AM, llpind <sonny_heer@hotmail.com>
> wrote:
> >> >
> >> >>
> >> >> Playing with the HBase perfomanceEval Class, but it seems to take a
> >> long
> >> >> time to run =93sequentialWrite 2=94 (~20 minutes).  If I simply emu=
late 1
> >> >> clients in a simple program, I can do 1 Million Puts in about 3
> >> minutes
> >> >> (non
> >> >> mapred).  The sequential write is writing 2 million with 2 clients.
> >> >> Please
> >> >> help in understanding how to use the performanceEvaluation Class.
> >> >>
> >> >
> >> > If the number of clients is > 1, unless you add the '--nomapred' (sp=
?)
> >> > argument, PE launches a mapreduce program of N tasks.  Each task put=
s
> >> up
> >> a
> >> > client writing 1M rows (IIRC).  Try N where N =3D=3D number_of_map_s=
lots
> >> and
> >> > see
> >> > what that does?  N =3D=3D 2 probably won't tell you much.  You could=
 also
> >> set
> >> > an
> >> > N > 1 and use the '--nomapred'.  This will run PE clients in a
> distinct
> >> > thread.  For small numbers of N, this can put up heavier loading tha=
n
> >> MR
> >> > with its setup and teardown cost.
> >> >
> >> > St.Ack
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/HBase-in-a-real-world-application-tp24920888p249750=
31.html
> >> Sent from the HBase User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/HBase-in-a-real-world-application-tp24920888p249774=
00.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>

--0016363ba25aa4fa68047120749b--