hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Sela <am...@infolinks.com>
Subject Re: RegionServer shutdown with ScanWildcardColumnTracker exception
Date Wed, 17 Apr 2013 11:47:41 GMT
I scanned over this counter with and without column specification and all
looks OK now.
I have no CPs in this table.
Is there some kind of a hint mechanism in HBase' internal scan ? because
it's weird that ScanWildcardColumnTracker.checkColumn says that column is
smaller than previous column: *imprersions_ALL_2013041617*. there is no
imprersions only impressions and r is indeed smaller than s, could it be
some kind of hint bug ? I don't think I know enough of HBase internals to
fully understand that...



On Wed, Apr 17, 2013 at 1:42 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> Hi Amit
>
> Checking the code this is possible when the qualifiers are not sorted.  Do
> you have any CPs in your path which tries to play with the KVs?
>
> Seems to be a very weird thing.
> Can you try doing a scan on the KV just before this happens.  That will tel
> you the existing kvs that are present.
>
> Even now if you can have the cluster you can try scanning for the region
> for which the flush happened.  That will give us some more info.
>
> Regards
> Ram
>
>
> On Wed, Apr 17, 2013 at 2:36 PM, Amit Sela <amits@infolinks.com> wrote:
>
> > The cluster runs Hadoop 1.0.4 and HBase 0.94.2
> >
> > I have three families in this table: weekly, daily, hourly. each family
> has
> > the following qualifiers:
> > Weekly - impressions_{countrycode}_{week#} - country code is 0, 1 or ALL
> > (aggregation of both 0 and 1)
> > Daily and hourly are the same but with yyyyMMdd and yyyyMMddhh
> > respectively.
> >
> > Just before the exception the regionserver StoreFile executes the
> > following:
> >
> > 2013-04-16 17:56:06,769 [regionserver8041.cacheFlusher] INFO
> > org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom
> filter
> > type for hdfs://
> > hadoop-master.infolinks.com:8000/hbase/URL_COUNTERS/af2760e
> > 4d04a9e3025d1fb53bdba8acf/.tmp/dc4ce516887f4e0bbaf6201d69ba90bc:
> > CompoundBloomFilterWriter
> > 2013-04-16 17:56:07,331 [regionserver8041.cacheFlusher] INFO
> > org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO
> > DeleteFamily was added to HFile (hdfs://hbase-master-address:8000/hbase
> > /URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
> > /.tmp/dc4ce516887f4e0bbaf6201d69ba90bc)
> > 2013-04-16 17:56:07,331 [regionserver8041.cacheFlusher] INFO
> > org.apache.hadoop.hbase.regionserver.Store: Flushed ,
> sequenceid=210517246,
> > memsize=39.3m, into tmp file hdfs://hbase-master:8000/hbase
> > /URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
> > /.tmp/dc4ce516887f4e0bbaf6201d69ba90bc
> > 2013-04-16 17:56:07,357 [regionserver8041.cacheFlusher] INFO
> > org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom
> filter
> > type for hdfs://hbase-master:8000/hbase/URL_COUNTERS/*af2760e*
> > *4d04a9e3025d1fb53bdba8acf*/.tmp/3fa7993dcb294be1bca5e4d7357f4003:
> > CompoundBloomFilterWriter
> > 2013-04-16 17:56:07,608 [regionserver8041.cacheFlusher] INFO
> > org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO
> > DeleteFamily was added to HFile (hdfs://hbase-master:8000/hbase
> > /URL_COUNTERS/*af2760e4d04a9e3025d1fb53bdba8acf*
> > /.tmp/3fa7993dcb294be1bca5e4d7357f4003)
> > 2013-04-16 17:56:07,608 [regionserver8041.cacheFlusher] FATAL
> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
> server
> > region-server-address,8041,1364993168088: Replay of HLog required
> > . Forcing server shutdown
> > DroppedSnapshotException: region: TABLE,ROWKEY,1364317591568.*
> > af2760e4d04a9e3025d1fb53bdba8acf*.
> > ....
> > ....
> > ...
> >
> >
> > On Wed, Apr 17, 2013 at 11:47 AM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> > > Seems interesting.  Can  you tell us what are the families and the
> > > qualifiers available in your schema.
> > >
> > > Any other interesting logs that you can see before this?
> > >
> > > BTW the version of HBase is also needed?  If we can track it out we can
> > > then file a JIRA if it is a bug.
> > >
> > > Regards
> > > RAm
> > >
> > >
> > > On Wed, Apr 17, 2013 at 2:00 PM, Amit Sela <amits@infolinks.com>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I had a regionserver crushed during counters increment. Looking at
> the
> > > > regionserver log I saw:
> > > >
> > > > org.apache.hadoop.hbase.DroppedSnapshotException: region: TABLE_NAME,
> > > > ROW_KEY...at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1472)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1351)
> > > >         at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1292)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:406)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:380)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:243)
> > > >         at java.lang.Thread.run(Thread.java:722)
> > > > Caused by: java.io.IOException: ScanWildcardColumnTracker.checkColumn
> > ran
> > > > into a column actually smaller than the previous column: *QUALIFIER*
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.ScanWildcardColumnTracker.checkColumn(ScanWildcardColumnTracker.java:104)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:354)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:362)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:311)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:738)
> > > >         at
> > > > org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:673)
> > > >         at
> > > > org.apache.hadoop.hbase.regionserver.Store.access$400(Store.java:108)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:2276)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1447)
> > > >
> > > > The strange thing is that the *QUALIFER* name as it appears in the
> log
> > is
> > > > misspelled.... there is no, and never was such qualifier name.
> > > >
> > > > Thanks,
> > > >
> > > > Amit.
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message