hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: Replaying of HLog
Date Thu, 04 Jun 2009 22:08:26 GMT
We should re-new the objects on every iteration instead.  I'll patch it up
for 0.20.

omg!


On Thu, Jun 4, 2009 at 2:38 PM, Clint Morgan <clint.morgan@troove.net>wrote:

> Ok, I tracked down the issue:
>
> HLog:803
>                  queue.push(new HLogEntry(val, key));
>
> The same val and key references are used in the iterator, so they keep
> changing their contents as we .next() through the loop.
>
> Thus, when we are done, all the HLogEntries in logEntries will have the
> same
> HLogEdits and HLogKeys! I don't think this would have ever worked unless
> there is only one region in the log being split.
>
> I guess the solution is to deep copy those val and key contents...
>
> -clint
>
> On Thu, Jun 4, 2009 at 10:38 AM, stack <stack@duboce.net> wrote:
>
> > If you can figure whats up, that'd be great Clint.
> > St.Ack
> >
> > On Thu, Jun 4, 2009 at 10:15 AM, Clint Morgan <clint.morgan@troove.net
> > >wrote:
> >
> > > I did not properly close the log, just did a kill -9 on the
> regionserver.
> > > But I was using local filesystem so I thought the WAL appends would get
> > > through. It appears some did...
> > >
> > > I'm not worried about the META.
> > >
> > > The debug log does sound like it sees my changes in the WAL (to the
> > BaseLog
> > > table, and its index), but it does not apply them appropriately...
> > >
> > >
> > > On Thu, Jun 4, 2009 at 10:04 AM, stack <stack@duboce.net> wrote:
> > >
> > > > Clint, as J-D suggests, you sure the edit made into a properly closed
> > log
> > > > first (I'm guessing you did)?  If not, you can make it happen by
> tuning
> > > > down
> > > > the time between log flushes -- its new config
> > > > hbase.regionserver.logroll.period.
> > > > St.Ack
> > > >
> > > > On Thu, Jun 4, 2009 at 9:35 AM, Jean-Daniel Cryans <
> > jdcryans@apache.org
> > > > >wrote:
> > > >
> > > > > Clint,
> > > > >
> > > > > Don't dig too deep, META edits will always be missed until we can
> > > > > flush every edits.
> > > > >
> > > > > J-D
> > > > >
> > > > > On Thu, Jun 4, 2009 at 12:31 PM, Clint Morgan <
> cm-public@troove.net>
> > > > > wrote:
> > > > > > I the meta edits did not get applied correctly. The only changes
> > were
> > > > > > region->server assignments, and after replay it still had
the old
> > > > > > assignments.
> > > > > >
> > > > > > I had made one explicit change to another table before the hard
> > > > restart,
> > > > > and
> > > > > > that change did not get recovered either.
> > > > > >
> > > > > > I'll dig deeper today...
> > > > > > -clint
> > > > > >
> > > > > > On Wed, Jun 3, 2009 at 7:22 PM, stack <stack@duboce.net>
wrote:
> > > > > >
> > > > > >> On Wed, Jun 3, 2009 at 5:53 PM, Clint Morgan <
> > cm-public@troove.net>
> > > > > wrote:
> > > > > >>
> > > > > >> > Which seems weird because the hlog path points to a
table's
> > > > directory
> > > > > >> which
> > > > > >> > is a different table than the given region. Does this
seem
> > right?
> > > > > >> >
> > > > > >> >
> > > > > >> No.  It seems badly wrong.  I took a quick look at HLog#splitLog
> > and
> > > > how
> > > > > >> its
> > > > > >> making up name of the file to write to.  I don't see anything
> > > obvious.
> > > > >  Its
> > > > > >> using the HLogKey it got out of the commit log.  This in
turn
> > looks
> > > > like
> > > > > it
> > > > > >> has right table name going in -- it gets it from region
whose
> > commit
> > > > is
> > > > > >> being logged.
> > > > > >>
> > > > > >> I see in below that it made queues for -ROOT- and .META.
  Did
> > these
> > > > > edits
> > > > > >> get split into the right location?
> > > > > >>
> > > > > >> St.Ack
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> > A larger portion of the log is below.
> > > > > >> >
> > > > > >> > [03/06/09 16:35:34] 530059 [        HMaster] INFO
> > > > > >> > hadoop.hbase.regionserver.HLog  - Splitting 1 log(s)
in
> > > > > >> >
> > > file:/opt/troove-data/hadoop/hbase/log_10.0.1.196_1244071611670_7802
> > > > > >> > [03/06/09 16:35:34] 530060 [        HMaster] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Splitting hlog 1
of 1:
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/log_10.0.1.196_1244071611670_7802/hlog.dat.1244071612170,
> > > > > >> > length=45056
> > > > > >> > [03/06/09 16:35:35] 530106 [        HMaster] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
-ROOT-,,0
> > > > > >> > [03/06/09 16:35:35] 530110 [        HMaster] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
.META.,,1
> > > > > >> > [03/06/09 16:35:35] 530120 [        HMaster] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
> > > > > >> > TransactionStatus,,1244059490418
> > > > > >> > [03/06/09 16:35:35] 530121 [        HMaster] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
> > > > > >> > net_troove_coin_project_Project,,1244059485284
> > > > > >> > [03/06/09 16:35:35] 530122 [        HMaster] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
> > > > > >> > net_troove_security_ApplicationUser,,1244059486102
> > > > > >> > [03/06/09 16:35:35] 530124 [        HMaster] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
> > > > > >> > net_troove_coin_timeexpense_BaseLog,,1244059487257
> > > > > >> > [03/06/09 16:35:35] 530124 [        HMaster] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Adding queue for
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520
> > > > > >> > [03/06/09 16:35:35] 530133 [pool-1-thread-1] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Thread got 2 to process
> > > > > >> > [03/06/09 16:35:35] 530137 [pool-1-thread-3] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Thread got 7 to process
> > > > > >> > [03/06/09 16:35:35] 530137 [pool-1-thread-2] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Thread got 220 to
process
> > > > > >> > [03/06/09 16:35:35] 530144 [pool-1-thread-4] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Thread got 1 to process
> > > > > >> > [03/06/09 16:35:35] 530149 [pool-1-thread-5] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Thread got 18 to
process
> > > > > >> > [03/06/09 16:35:35] 530157 [pool-1-thread-6] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Thread got 10 to
process
> > > > > >> > [03/06/09 16:35:35] 530209 [pool-1-thread-7] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Thread got 10 to
process
> > > > > >> > [03/06/09 16:35:35] 530325 [pool-1-thread-3] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog
file
> writer
> > > for
> > > > > path
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/228384672/oldlogfile.log
> > > > > >> > and region TransactionStatus,,1244059490418
> > > > > >> > [03/06/09 16:35:35] 530326 [pool-1-thread-3] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Applied 7 total edits
to
> > > > > >> > TransactionStatus,,1244059490418 in 187ms
> > > > > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-1] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog
file
> writer
> > > for
> > > > > path
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/70236052/oldlogfile.log
> > > > > >> > and region -ROOT-,,0
> > > > > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-5] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog
file
> writer
> > > for
> > > > > path
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/767391690/oldlogfile.log
> > > > > >> > and region net_troove_coin_timeexpense_BaseLog,,1244059487257
> > > > > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-5] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Applied 18 total
edits to
> > > > > >> > net_troove_coin_timeexpense_BaseLog,,1244059487257
in 173ms
> > > > > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-6] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog
file
> writer
> > > for
> > > > > path
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/364454454/oldlogfile.log
> > > > > >> > and region
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520
> > > > > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-6] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Applied 10 total
edits to
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520
> > > > > >> > in 154ms
> > > > > >> > [03/06/09 16:35:35] 530330 [pool-1-thread-2] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog
file
> writer
> > > for
> > > > > path
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/1028785192/oldlogfile.log
> > > > > >> > and region .META.,,1
> > > > > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-4] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog
file
> writer
> > > for
> > > > > path
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/124273138/oldlogfile.log
> > > > > >> > and region net_troove_coin_project_Project,,1244059485284
> > > > > >> > [03/06/09 16:35:35] 530331 [pool-1-thread-4] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Applied 1 total edits
to
> > > > > >> > net_troove_coin_project_Project,,1244059485284 in 187ms
> > > > > >> > [03/06/09 16:35:35] 530330 [pool-1-thread-7] DEBUG
> > > > > >> > hadoop.hbase.regionserver.HLog  - Creating new hlog
file
> writer
> > > for
> > > > > path
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/395668855/oldlogfile.log
> > > > > >> > and region net_troove_security_ApplicationUser,,1244059486102
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message