Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Message-ID: <1080640403.1214453565269.JavaMail.jira@brutus>
Date: Wed, 25 Jun 2008 21:12:45 -0700 (PDT)
From: "stack (JIRA)" <jira@apache.org>
To: hbase-dev@hadoop.apache.org
Subject: [jira] Assigned: (HBASE-710) If clocks are way off, then we can
 have daughter split come before rather than after its parent in .META.
In-Reply-To: <1660144810.1214441206745.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/HBASE-710?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack reassigned HBASE-710:
---------------------------

    Assignee: stack

> If clocks are way off, then we can have daughter split come before rather=
 than after its parent in .META.
> -------------------------------------------------------------------------=
--------------------------------
>
>                 Key: HBASE-710
>                 URL: https://issues.apache.org/jira/browse/HBASE-710
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.1.3
>
>         Attachments: hbase-710.patch
>
>
> On the Jon Gray cluster, his clocks are skewed badly.  I see weird stuff =
in .META.
> {code}
> 2008-06-25 14:57:57,728 DEBUG org.apache.hadoop.hbase.HMaster: HMaster.me=
taScanner regioninfo: {regionname: items,823ce1e3-d414-474f-ac70-c4081cecef=
0f,1214416614697, startKey: <823ce1e3-d414-474f-ac70-c4081cecef0f>, endKey:=
 <86f8df20-e237-4bb3-9748-88cef892bd70>, encodedName: 1157924217, offline: =
true, split: true, tableDesc: {name: items, families: {cfrecs:=3D{name: cfr=
ecs, max versions: 2, compression: NONE, in memory: false, max length: 2147=
483647, bloom filter: none}, clusters:=3D{name: clusters, max versions: 2, =
compression: NONE, in memory: false, max length: 2147483647, bloom filter: =
none}, content:=3D{name: content, max versions: 2, compression: NONE, in me=
mory: false, max length: 2147483647, bloom filter: none}, readby:=3D{name: =
readby, max versions: 2, compression: NONE, in memory: false, max length: 2=
147483647, bloom filter: none}, receivedby:=3D{name: receivedby, max versio=
ns: 2, compression: NONE, in memory: false, max length: 2147483647, bloom f=
ilter: none}, savedby:=3D{name: savedby, max versions: 2, compression: NONE=
, in memory: false, max length: 2147483647, bloom filter: none}, sentby:=3D=
{name: sentby, max versions: 2, compression: NONE, in memory: false, max le=
ngth: 2147483647, bloom filter: none}}}}, server: 192.168.249.223:60020, st=
artCode: 1214406344634
> 2008-06-25 14:57:57,732 DEBUG org.apache.hadoop.hbase.HMaster: HMaster.me=
taScanner regioninfo: {regionname: items,823ce1e3-d414-474f-ac70-c4081cecef=
0f,1214416641213, startKey: <823ce1e3-d414-474f-ac70-c4081cecef0f>, endKey:=
 <83fca0e2-f324-4f9e-99c1-1fdbeff63b3d>, encodedName: 541300165, tableDesc:=
 {name: items, families: {cfrecs:=3D{name: cfrecs, max versions: 2, compres=
sion: NONE, in memory: false, max length: 2147483647, bloom filter: none}, =
clusters:=3D{name: clusters, max versions: 2, compression: NONE, in memory:=
 false, max length: 2147483647, bloom filter: none}, content:=3D{name: cont=
ent, max versions: 2, compression: NONE, in memory: false, max length: 2147=
483647, bloom filter: none}, readby:=3D{name: readby, max versions: 2, comp=
ression: NONE, in memory: false, max length: 2147483647, bloom filter: none=
}, receivedby:=3D{name: receivedby, max versions: 2, compression: NONE, in =
memory: false, max length: 2147483647, bloom filter: none}, savedby:=3D{nam=
e: savedby, max versions: 2, compression: NONE, in memory: false, max lengt=
h: 2147483647, bloom filter: none}, sentby:=3D{name: sentby, max versions: =
2, compression: NONE, in memory: false, max length: 2147483647, bloom filte=
r: none}}}}, server: 192.168.249.220:60020, startCode: 1214424347649
> 2008-06-25 14:57:57,738 DEBUG org.apache.hadoop.hbase.HMaster: HMaster.me=
taScanner regioninfo: {regionname: items,823ce1e3-d414-474f-ac70-c4081cecef=
0f,1214434560891, startKey: <823ce1e3-d414-474f-ac70-c4081cecef0f>, endKey:=
 <9066d4f3-314b-4d9c-90e8-7aa08a52fdd4>, encodedName: 1673833201, offline: =
true, split: true, tableDesc: {name: items, families: {cfrecs:=3D{name: cfr=
ecs, max versions: 2, compression: NONE, in memory: false, max length: 2147=
483647, bloom filter: none}, clusters:=3D{name: clusters, max versions: 2, =
compression: NONE, in memory: false, max length: 2147483647, bloom filter: =
none}, content:=3D{name: content, max versions: 2, compression: NONE, in me=
mory: false, max length: 2147483647, bloom filter: none}, readby:=3D{name: =
readby, max versions: 2, compression: NONE, in memory: false, max length: 2=
147483647, bloom filter: none}, receivedby:=3D{name: receivedby, max versio=
ns: 2, compression: NONE, in memory: false, max length: 2147483647, bloom f=
ilter: none}, savedby:=3D{name: savedby, max versions: 2, compression: NONE=
, in memory: false, max length: 2147483647, bloom filter: none}, sentby:=3D=
{name: sentby, max versions: 2, compression: NONE, in memory: false, max le=
ngth: 2147483647, bloom filter: none}}}}, server: 192.168.249.221:60020, st=
artCode: 1214406358315
> {code}
> Thats 3 regions with same start code; 2 are offline.
> Looking at the regionids -- these are timestamps -- I see that they don't=
 jibe with how they should be aligned.  Parents should come before daughter=
s in timestamps.
> Looking at clocks on cluster, they are badly skewed:
> {code
> <st^ack>=09[hbase@mb0 logs]$ for i in 0 1 2 3 4 5 6 7 8 9; do ssh hb$i "h=
ostname; date"; done
> =09<st^ack>=09hb0.streamy.com
> =09<st^ack>=09Wed Jun 25 16:47:29 PDT 2008
> =09<st^ack>=09hb1.streamy.com
> =09<st^ack>=09Wed Jun 25 11:47:39 PDT 2008
> =09<st^ack>=09hb2.streamy.com
> =09<st^ack>=09Wed Jun 25 11:47:40 PDT 2008
> =09<st^ack>=09hb3.streamy.com
> =09<st^ack>=09Wed Jun 25 11:47:26 PDT 2008
> =09<st^ack>=09hb4.streamy.com
> =09<st^ack>=09Wed Jun 25 11:47:35 PDT 2008
> =09<st^ack>=09hb5.streamy.com
> =09<st^ack>=09Wed Jun 25 16:47:29 PDT 2008
> =09<st^ack>=09hb6.streamy.com
> =09<st^ack>=09Wed Jun 25 16:47:29 PDT 2008
> =09<st^ack>=09hb7.streamy.com
> =09<st^ack>=09Wed Jun 25 16:47:29 PDT 2008
> =09<st^ack>=09hb8.streamy.com
> =09<st^ack>=09Wed Jun 25 16:47:30 PDT 2008
> =09<st^ack>=09hb9.streamy.com
> =09<st^ack>=09Wed Jun 25 16:47:30 PDT 2008
> {code}
> Looking at split code, looks like the regionserver sets the regionid/time=
stamp on the new daughter regions inside in the HRegionInfo constructor tha=
t gets called when splitting:
> {code}
>     this.regionId =3D System.currentTimeMillis();
> {code}
> Daughters update the .META. table; they need to have a basic check that t=
hey are not inserting with a timestamp that is older than the parent they a=
re splitting.
> This is a bit like HBASE-609

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.