Return-Path: Delivered-To: apmail-hadoop-hbase-dev-archive@minotaur.apache.org Received: (qmail 7226 invoked from network); 4 Jun 2009 17:38:51 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 4 Jun 2009 17:38:51 -0000 Received: (qmail 46992 invoked by uid 500); 4 Jun 2009 17:39:02 -0000 Delivered-To: apmail-hadoop-hbase-dev-archive@hadoop.apache.org Received: (qmail 46942 invoked by uid 500); 4 Jun 2009 17:39:02 -0000 Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-dev@hadoop.apache.org Delivered-To: mailing list hbase-dev@hadoop.apache.org Received: (qmail 46932 invoked by uid 99); 4 Jun 2009 17:39:02 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Jun 2009 17:39:02 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.92.26 as permitted sender) Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Jun 2009 17:38:52 +0000 Received: by qw-out-2122.google.com with SMTP id 3so543720qwe.35 for ; Thu, 04 Jun 2009 10:38:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to :content-type; bh=kahHERCipypcB1A9AV5HZTPQjVNgi93vRR1D+LENLFo=; b=JgL16l/Ztl0wPmsOKM1bEjOirOERkbvN5gbjEdDyiKBaq1U24ZLMWfBmIubq3UJdR7 lLBjF0IbvrZcCeYP6Xqen1iAOlDF8jKvsNVKQJfRTSfYmXUlswb/mI3E4gRSB1/YsKrP U09V+FB4orsuFu2UgfbeoBVXcserZcKUS6hF4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; b=VXusi7EGnsgKHnBQ5TixHiNFdpt742S9DqKU4631UEhSbZ12AZ04GY0TeuhBb0Gl9q MP11BABcaczryterAftDQYF8X4m9k40nKeItk8XPpLzPGb0tCk0Hq/kV2PXZB1amLUaT g2eO3eqEs2Sew/el7lHs0KswHI9TECw0jIxrQ= MIME-Version: 1.0 Sender: saint.ack@gmail.com Received: by 10.229.96.1 with SMTP id f1mr680095qcn.103.1244137110947; Thu, 04 Jun 2009 10:38:30 -0700 (PDT) In-Reply-To: References: <7c962aed0906031922u22b27375gf05e1360150a7ac6@mail.gmail.com> <31a243e70906040935g3ac813eev72bb6a8c93d461be@mail.gmail.com> <7c962aed0906041004u40379b95rd4aca0dd6bb5e1b6@mail.gmail.com> Date: Thu, 4 Jun 2009 10:38:29 -0700 X-Google-Sender-Auth: 1741b9af392d2b1e Message-ID: <7c962aed0906041038n764971dev7f30f18669f56314@mail.gmail.com> Subject: Re: Replaying of HLog From: stack To: hbase-dev@hadoop.apache.org Content-Type: multipart/alternative; boundary=0016364270d28cecff046b893ed7 X-Virus-Checked: Checked by ClamAV on apache.org --0016364270d28cecff046b893ed7 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit If you can figure whats up, that'd be great Clint. St.Ack On Thu, Jun 4, 2009 at 10:15 AM, Clint Morgan wrote: > I did not properly close the log, just did a kill -9 on the regionserver. > But I was using local filesystem so I thought the WAL appends would get > through. It appears some did... > > I'm not worried about the META. > > The debug log does sound like it sees my changes in the WAL (to the BaseLog > table, and its index), but it does not apply them appropriately... > > > On Thu, Jun 4, 2009 at 10:04 AM, stack wrote: > > > Clint, as J-D suggests, you sure the edit made into a properly closed log > > first (I'm guessing you did)? If not, you can make it happen by tuning > > down > > the time between log flushes -- its new config > > hbase.regionserver.logroll.period. > > St.Ack > > > > On Thu, Jun 4, 2009 at 9:35 AM, Jean-Daniel Cryans > >wrote: > > > > > Clint, > > > > > > Don't dig too deep, META edits will always be missed until we can > > > flush every edits. > > > > > > J-D > > > > > > On Thu, Jun 4, 2009 at 12:31 PM, Clint Morgan > > > wrote: > > > > I the meta edits did not get applied correctly. The only changes were > > > > region->server assignments, and after replay it still had the old > > > > assignments. > > > > > > > > I had made one explicit change to another table before the hard > > restart, > > > and > > > > that change did not get recovered either. > > > > > > > > I'll dig deeper today... > > > > -clint > > > > > > > > On Wed, Jun 3, 2009 at 7:22 PM, stack wrote: > > > > > > > >> On Wed, Jun 3, 2009 at 5:53 PM, Clint Morgan > > > wrote: > > > >> > > > >> > Which seems weird because the hlog path points to a table's > > directory > > > >> which > > > >> > is a different table than the given region. Does this seem right? > > > >> > > > > >> > > > > >> No. It seems badly wrong. I took a quick look at HLog#splitLog and > > how > > > >> its > > > >> making up name of the file to write to. I don't see anything > obvious. > > > Its > > > >> using the HLogKey it got out of the commit log. This in turn looks > > like > > > it > > > >> has right table name going in -- it gets it from region whose commit > > is > > > >> being logged. > > > >> > > > >> I see in below that it made queues for -ROOT- and .META. Did these > > > edits > > > >> get split into the right location? > > > >> > > > >> St.Ack > > > >> > > > >> > > > >> > > > >> > A larger portion of the log is below. > > > >> > > > > >> > [03/06/09 16:35:34] 530059 [ HMaster] INFO > > > >> > hadoop.hbase.regionserver.HLog - Splitting 1 log(s) in > > > >> > > file:/opt/troove-data/hadoop/hbase/log_10.0.1.196_1244071611670_7802 > > > >> > [03/06/09 16:35:34] 530060 [ HMaster] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Splitting hlog 1 of 1: > > > >> > > > > >> > > > > >> > > > > > > file:/opt/troove-data/hadoop/hbase/log_10.0.1.196_1244071611670_7802/hlog.dat.1244071612170, > > > >> > length=45056 > > > >> > [03/06/09 16:35:35] 530106 [ HMaster] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for -ROOT-,,0 > > > >> > [03/06/09 16:35:35] 530110 [ HMaster] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for .META.,,1 > > > >> > [03/06/09 16:35:35] 530120 [ HMaster] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for > > > >> > TransactionStatus,,1244059490418 > > > >> > [03/06/09 16:35:35] 530121 [ HMaster] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for > > > >> > net_troove_coin_project_Project,,1244059485284 > > > >> > [03/06/09 16:35:35] 530122 [ HMaster] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for > > > >> > net_troove_security_ApplicationUser,,1244059486102 > > > >> > [03/06/09 16:35:35] 530124 [ HMaster] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for > > > >> > net_troove_coin_timeexpense_BaseLog,,1244059487257 > > > >> > [03/06/09 16:35:35] 530124 [ HMaster] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for > > > >> > > > > >> > > > > > > net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520 > > > >> > [03/06/09 16:35:35] 530133 [pool-1-thread-1] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Thread got 2 to process > > > >> > [03/06/09 16:35:35] 530137 [pool-1-thread-3] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Thread got 7 to process > > > >> > [03/06/09 16:35:35] 530137 [pool-1-thread-2] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Thread got 220 to process > > > >> > [03/06/09 16:35:35] 530144 [pool-1-thread-4] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Thread got 1 to process > > > >> > [03/06/09 16:35:35] 530149 [pool-1-thread-5] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Thread got 18 to process > > > >> > [03/06/09 16:35:35] 530157 [pool-1-thread-6] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Thread got 10 to process > > > >> > [03/06/09 16:35:35] 530209 [pool-1-thread-7] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Thread got 10 to process > > > >> > [03/06/09 16:35:35] 530325 [pool-1-thread-3] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer > for > > > path > > > >> > > > > >> > > > > >> > > > > > > file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/228384672/oldlogfile.log > > > >> > and region TransactionStatus,,1244059490418 > > > >> > [03/06/09 16:35:35] 530326 [pool-1-thread-3] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Applied 7 total edits to > > > >> > TransactionStatus,,1244059490418 in 187ms > > > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-1] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer > for > > > path > > > >> > > > > >> > > > > >> > > > > > > file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/70236052/oldlogfile.log > > > >> > and region -ROOT-,,0 > > > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-5] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer > for > > > path > > > >> > > > > >> > > > > >> > > > > > > file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/767391690/oldlogfile.log > > > >> > and region net_troove_coin_timeexpense_BaseLog,,1244059487257 > > > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-5] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Applied 18 total edits to > > > >> > net_troove_coin_timeexpense_BaseLog,,1244059487257 in 173ms > > > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-6] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer > for > > > path > > > >> > > > > >> > > > > >> > > > > > > file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/364454454/oldlogfile.log > > > >> > and region > > > >> > > > > >> > > > > > > net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520 > > > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-6] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Applied 10 total edits to > > > >> > > > > >> > > > > > > net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520 > > > >> > in 154ms > > > >> > [03/06/09 16:35:35] 530330 [pool-1-thread-2] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer > for > > > path > > > >> > > > > >> > > > > >> > > > > > > file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/1028785192/oldlogfile.log > > > >> > and region .META.,,1 > > > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-4] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer > for > > > path > > > >> > > > > >> > > > > >> > > > > > > file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/124273138/oldlogfile.log > > > >> > and region net_troove_coin_project_Project,,1244059485284 > > > >> > [03/06/09 16:35:35] 530331 [pool-1-thread-4] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Applied 1 total edits to > > > >> > net_troove_coin_project_Project,,1244059485284 in 187ms > > > >> > [03/06/09 16:35:35] 530330 [pool-1-thread-7] DEBUG > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer > for > > > path > > > >> > > > > >> > > > > >> > > > > > > file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/395668855/oldlogfile.log > > > >> > and region net_troove_security_ApplicationUser,,1244059486102 > > > >> > > > > >> > > > > > > > > > > --0016364270d28cecff046b893ed7--