hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clint Morgan <clint.mor...@troove.net>
Subject Re: Replaying of HLog
Date Thu, 04 Jun 2009 17:15:40 GMT
I did not properly close the log, just did a kill -9 on the regionserver.
But I was using local filesystem so I thought the WAL appends would get
through. It appears some did...

I'm not worried about the META.

The debug log does sound like it sees my changes in the WAL (to the BaseLog
table, and its index), but it does not apply them appropriately...


On Thu, Jun 4, 2009 at 10:04 AM, stack <stack@duboce.net> wrote:

> Clint, as J-D suggests, you sure the edit made into a properly closed log
> first (I'm guessing you did)?  If not, you can make it happen by tuning
> down
> the time between log flushes -- its new config
> hbase.regionserver.logroll.period.
> St.Ack
>
> On Thu, Jun 4, 2009 at 9:35 AM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
>
> > Clint,
> >
> > Don't dig too deep, META edits will always be missed until we can
> > flush every edits.
> >
> > J-D
> >
> > On Thu, Jun 4, 2009 at 12:31 PM, Clint Morgan <cm-public@troove.net>
> > wrote:
> > > I the meta edits did not get applied correctly. The only changes were
> > > region->server assignments, and after replay it still had the old
> > > assignments.
> > >
> > > I had made one explicit change to another table before the hard
> restart,
> > and
> > > that change did not get recovered either.
> > >
> > > I'll dig deeper today...
> > > -clint
> > >
> > > On Wed, Jun 3, 2009 at 7:22 PM, stack <stack@duboce.net> wrote:
> > >
> > >> On Wed, Jun 3, 2009 at 5:53 PM, Clint Morgan <cm-public@troove.net>
> > wrote:
> > >>
> > >> > Which seems weird because the hlog path points to a table's
> directory
> > >> which
> > >> > is a different table than the given region. Does this seem right?
> > >> >
> > >> >
> > >> No.  It seems badly wrong.  I took a quick look at HLog#splitLog and
> how
> > >> its
> > >> making up name of the file to write to.  I don't see anything obvious.
> >  Its
> > >> using the HLogKey it got out of the commit log.  This in turn looks
> like
> > it
> > >> has right table name going in -- it gets it from region whose commit
> is
> > >> being logged.
> > >>
> > >> I see in below that it made queues for -ROOT- and .META.   Did these
> > edits
> > >> get split into the right location?
> > >>
> > >> St.Ack
> > >>
> > >>
> > >>
> > >> > A larger portion of the log is below.
> > >> >
> > >> > [03/06/09 16:35:34] 530059 [        HMaster] INFO
> > >> > hadoop.hbase.regionserver.HLog  - Splitting 1 log(s) in
> > >> > file:/opt/troove-data/hadoop/hbase/log_10.0.1.196_1244071611670_7802
> > >> > [03/06/09 16:35:34] 530060 [        HMaster] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Splitting hlog 1 of 1:
> > >> >
> > >> >
> > >>
> >
> file:/opt/troove-data/hadoop/hbase/log_10.0.1.196_1244071611670_7802/hlog.dat.1244071612170,
> > >> > length=45056
> > >> > [03/06/09 16:35:35] 530106 [        HMaster] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Adding queue for -ROOT-,,0
> > >> > [03/06/09 16:35:35] 530110 [        HMaster] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Adding queue for .META.,,1
> > >> > [03/06/09 16:35:35] 530120 [        HMaster] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
> > >> > TransactionStatus,,1244059490418
> > >> > [03/06/09 16:35:35] 530121 [        HMaster] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
> > >> > net_troove_coin_project_Project,,1244059485284
> > >> > [03/06/09 16:35:35] 530122 [        HMaster] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
> > >> > net_troove_security_ApplicationUser,,1244059486102
> > >> > [03/06/09 16:35:35] 530124 [        HMaster] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
> > >> > net_troove_coin_timeexpense_BaseLog,,1244059487257
> > >> > [03/06/09 16:35:35] 530124 [        HMaster] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
> > >> >
> > >>
> >
> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520
> > >> > [03/06/09 16:35:35] 530133 [pool-1-thread-1] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Thread got 2 to process
> > >> > [03/06/09 16:35:35] 530137 [pool-1-thread-3] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Thread got 7 to process
> > >> > [03/06/09 16:35:35] 530137 [pool-1-thread-2] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Thread got 220 to process
> > >> > [03/06/09 16:35:35] 530144 [pool-1-thread-4] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Thread got 1 to process
> > >> > [03/06/09 16:35:35] 530149 [pool-1-thread-5] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Thread got 18 to process
> > >> > [03/06/09 16:35:35] 530157 [pool-1-thread-6] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Thread got 10 to process
> > >> > [03/06/09 16:35:35] 530209 [pool-1-thread-7] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Thread got 10 to process
> > >> > [03/06/09 16:35:35] 530325 [pool-1-thread-3] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog file writer for
> > path
> > >> >
> > >> >
> > >>
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/228384672/oldlogfile.log
> > >> > and region TransactionStatus,,1244059490418
> > >> > [03/06/09 16:35:35] 530326 [pool-1-thread-3] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Applied 7 total edits to
> > >> > TransactionStatus,,1244059490418 in 187ms
> > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-1] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog file writer for
> > path
> > >> >
> > >> >
> > >>
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/70236052/oldlogfile.log
> > >> > and region -ROOT-,,0
> > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-5] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog file writer for
> > path
> > >> >
> > >> >
> > >>
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/767391690/oldlogfile.log
> > >> > and region net_troove_coin_timeexpense_BaseLog,,1244059487257
> > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-5] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Applied 18 total edits to
> > >> > net_troove_coin_timeexpense_BaseLog,,1244059487257 in 173ms
> > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-6] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog file writer for
> > path
> > >> >
> > >> >
> > >>
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/364454454/oldlogfile.log
> > >> > and region
> > >> >
> > >>
> >
> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520
> > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-6] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Applied 10 total edits to
> > >> >
> > >>
> >
> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520
> > >> > in 154ms
> > >> > [03/06/09 16:35:35] 530330 [pool-1-thread-2] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog file writer for
> > path
> > >> >
> > >> >
> > >>
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/1028785192/oldlogfile.log
> > >> > and region .META.,,1
> > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-4] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog file writer for
> > path
> > >> >
> > >> >
> > >>
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/124273138/oldlogfile.log
> > >> > and region net_troove_coin_project_Project,,1244059485284
> > >> > [03/06/09 16:35:35] 530331 [pool-1-thread-4] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Applied 1 total edits to
> > >> > net_troove_coin_project_Project,,1244059485284 in 187ms
> > >> > [03/06/09 16:35:35] 530330 [pool-1-thread-7] DEBUG
> > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog file writer for
> > path
> > >> >
> > >> >
> > >>
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/395668855/oldlogfile.log
> > >> > and region net_troove_security_ApplicationUser,,1244059486102
> > >> >
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message