hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: HBase in a real world application
Date Fri, 14 Aug 2009 21:08:36 GMT
Thanks for trying.  Looks like that region is now gone (split is my guess).
Check the master log for mentions of this region to see its history.  Can
you correlate the client failure with an event on this region in master
log?  It looks like client was being pig-headed fixated on the parent of a
split.  You could check your table is healthy?  Run the rowcounter program
to make sure no holes in table?

St.Ack

On Fri, Aug 14, 2009 at 12:41 PM, llpind <sonny_heer@hotmail.com> wrote:

>
> hbase(main):003:0> get '.META.', 'TestTable,0001749889,1250092414985',
> {COLUMNS =>'info'}
> 09/08/14 12:28:10 DEBUG client.HConnectionManager$TableServers: Cache hit
> for row <> in tableName .META.: location server 192.168.0.196:60020,
> location region name .META.,,1
> NativeException: java.lang.NullPointerException: null
>        from org/apache/hadoop/hbase/client/HTable.java:789:in `get'
>        from org/apache/hadoop/hbase/client/HTable.java:769:in `get'
>        from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
>        from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
>        from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>        from java/lang/reflect/Method.java:597:in `invoke'
>        from org/jruby/javasupport/JavaMethod.java:298:in
> `invokeWithExceptionHandling'
>        from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
>        from org/jruby/java/invokers/InstanceMethodInvoker.java:30:in `call'
>        from org/jruby/runtime/callsite/CachingCallSite.java:30:in `call'
>        from org/jruby/ast/CallManyArgsNode.java:59:in `interpret'
>        from org/jruby/ast/LocalAsgnNode.java:123:in `interpret'
>        from org/jruby/ast/NewlineNode.java:104:in `interpret'
>        from org/jruby/ast/IfNode.java:112:in `interpret'
>        from org/jruby/ast/NewlineNode.java:104:in `interpret'
>        from org/jruby/ast/IfNode.java:114:in `interpret'
> ... 115 levels...
>        from
> home/hadoop/hbase_minus_0_dot_20_dot_0/bin/$_dot_dot_/bin/hirb#start:-1:in
> `call'
>        from org/jruby/internal/runtime/methods/DynamicMethod.java:226:in
> `call'
>        from org/jruby/internal/runtime/methods/CompiledMethod.java:211:in
> `call'
>        from org/jruby/internal/runtime/methods/CompiledMethod.java:71:in
> `call'
>        from org/jruby/runtime/callsite/CachingCallSite.java:253:in
> `cacheAndCall'
>        from org/jruby/runtime/callsite/CachingCallSite.java:72:in `call'
>        from
> home/hadoop/hbase_minus_0_dot_20_dot_0/bin/$_dot_dot_/bin/hirb.rb:487:in
> `__file__'
>        from
> home/hadoop/hbase_minus_0_dot_20_dot_0/bin/$_dot_dot_/bin/hirb.rb:-1:in
> `load'
>        from org/jruby/Ruby.java:577:in `runScript'
>        from org/jruby/Ruby.java:480:in `runNormally'
>        from org/jruby/Ruby.java:354:in `runFromMain'
>        from org/jruby/Main.java:229:in `run'
>        from org/jruby/Main.java:110:in `run'
>        from org/jruby/Main.java:94:in `main'
>        from /home/hadoop/hbase-0.20.0/bin/../bin/hirb.rb:384:in `get'
>        from (hbase):4hbase(main):004:0> get '.META.',
> 'TestTable,0001749889,1250092414985'
> 09/08/14 12:28:13 DEBUG client.HConnectionManager$TableServers: Cache hit
> for row <> in tableName .META.: location server 192.168.0.196:60020,
> location region name .META.,,1
> COLUMN                       CELL
>  historian:assignment        timestamp=1250108456441, value=Region assigned
> to server server195,60020,12501083767
>                             79
>  historian:compaction        timestamp=1250109313965, value=Region
> compaction completed in 35sec
>  historian:open              timestamp=1250108459484, value=Region opened
> on
> server : server195
>  historian:split             timestamp=1250092447915, value=Region split
> from: TestTable,0001634945,1250035163
>                             027
>  info:regioninfo             timestamp=1250109315260, value=REGION => {NAME
> => 'TestTable,0001749889,125009241
>                             4985', STARTKEY => '0001749889', ENDKEY =>
> '0001866010', ENCODED => 1707908074, O
>                             FFLINE => true, TABLE => {{NAME => 'TestTable',
> FAMILIES => [{NAME => 'info', VER
>                             SIONS => '3', COMPRESSION => 'NONE', TTL =>
> '2147483647', BLOCKSIZE => '65536', I
>                             N_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
>
>
> =====================================
>
> stack-3 wrote:
> >
> > Is that region offline?
> >
> > Do a:
> >
> > hbase> get ".META.", "TestTable,0001749889,1250092414985", {COLUMNS =>
> > "info"}.
> >
> > If so, can you get its history so we can figure how it went offline? (See
> > region history in UI or grep it in master logs?)
> >
> > St.Ack
> >
> >
> > On Fri, Aug 14, 2009 at 9:55 AM, llpind <sonny_heer@hotmail.com> wrote:
> >
> >>
> >> Hey Stack,  I tried the following command:
> >>
> >> hadoop-0.20.0/bin/hadoop jar hbase-0.20.0/hbase-0.20.0-test.jar
> >>  randomWrite
> >> 10
> >>
> >> running a map/reduce job, it failed with the following exceptions in
> each
> >> node:
> >>
> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> >> contact
> >> region server Some server for region , row '0001753186', but failed
> after
> >> 11
> >> attempts.
> >> Exceptions:
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >> TestTable,0001749889,1250092414985
> >>
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocationForRowWithRetries(HConnectionManager.java:995)
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1025)
> >>        at
> >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584)
> >>        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450)
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest.testRow(PerformanceEvaluation.java:497)
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:406)
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:627)
> >>        at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:194)
> >>        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >>        at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356)
> >>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >>
> >> ================================================================
> >>
> >> This appears to be this issue:
> >>
> >> http://issues.apache.org/jira/browse/HBASE-1603
> >>
> >>
> >> Has this been fixed in .20?  Thanks.
> >>
> >>
> >>
> >> stack-3 wrote:
> >> >
> >> > On Wed, Aug 12, 2009 at 8:58 AM, llpind <sonny_heer@hotmail.com>
> wrote:
> >> >
> >> >>
> >> >> Playing with the HBase perfomanceEval Class, but it seems to take a
> >> long
> >> >> time to run “sequentialWrite 2” (~20 minutes).  If I simply emulate
1
> >> >> clients in a simple program, I can do 1 Million Puts in about 3
> >> minutes
> >> >> (non
> >> >> mapred).  The sequential write is writing 2 million with 2 clients.
> >> >> Please
> >> >> help in understanding how to use the performanceEvaluation Class.
> >> >>
> >> >
> >> > If the number of clients is > 1, unless you add the '--nomapred' (sp?)
> >> > argument, PE launches a mapreduce program of N tasks.  Each task puts
> >> up
> >> a
> >> > client writing 1M rows (IIRC).  Try N where N == number_of_map_slots
> >> and
> >> > see
> >> > what that does?  N == 2 probably won't tell you much.  You could also
> >> set
> >> > an
> >> > N > 1 and use the '--nomapred'.  This will run PE clients in a
> distinct
> >> > thread.  For small numbers of N, this can put up heavier loading than
> >> MR
> >> > with its setup and teardown cost.
> >> >
> >> > St.Ack
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/HBase-in-a-real-world-application-tp24920888p24975031.html
> >> Sent from the HBase User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/HBase-in-a-real-world-application-tp24920888p24977400.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message