Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of rnigam@decarta.com designates
 208.81.204.160 as permitted sender)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Subject: RE: version mismatch exception
Date: Wed, 17 Aug 2011 13:12:24 -0700
Message-ID: 
 <6C5C1804772DB944BA88A0DC48D338DA0B1D4E1F@dct-mail.sanjose.telcontar.com>
In-Reply-To: 
 <6C5C1804772DB944BA88A0DC48D338DA0B1D4D5B@dct-mail.sanjose.telcontar.com>
Thread-Topic: version mismatch exception
Thread-Index: AcxbxS6CmiAAsKvBQv2ZgbrWGDc4fwAAH2RAAEogZ2AACqMWUA==
References: 
 <6C5C1804772DB944BA88A0DC48D338DA0B0FDA96@dct-mail.sanjose.telcontar.com>
 <CADcMMgGa80yyPNRhZU1EmUHG0HpUjt7b2HYP8jPLrbqwJya78A@mail.gmail.com>
 <6C5C1804772DB944BA88A0DC48D338DA0B0FDB2E@dct-mail.sanjose.telcontar.com>
 <6C5C1804772DB944BA88A0DC48D338DA0B1D4D5B@dct-mail.sanjose.telcontar.com>
From: "Rohit Nigam" <rnigam@decarta.com>
To: "Geoff Hendrey" <ghendrey@decarta.com>,
	"Stack" <stack@duboce.net>,
	<user@hbase.apache.org>
Cc: "Search" <Search@decarta.com>

Hi St.Ack
The region in the file System are good, all I am looking is to change =
the end key of that region in the .META. table so that chaining problem =
goes away .The way I am planning to do is to get the HRegionInfo object =
for that existing region key from the .META. table . Create a new =
HRegionInfo obj with the updated endkey , start key and regionid being =
the same as from the result above and do a put in the .META. table. I =
think I just change the endkey and nothing else it will not create  a =
new row in .META. table and would just update the existing row. Please =
confirm if my theory is right.
Rohit

-----Original Message-----
From: Geoff Hendrey=20
Sent: Wednesday, August 17, 2011 9:55 AM
To: 'Stack'; 'user@hbase.apache.org'
Subject: RE: version mismatch exception

Hi St.Ack,

Keying off of what you said: " Did you update the
info:regioninfo cell so it has a new hregioninfo with same start and
end row?   You know this makes a new region, rather than extend the
range of the previous region? (So the old region will be in the
filesystem still with the old data).  You should also make sure the
original region is closed before you delete the row from .META."

So I take it the best approach is to:
1) close the original region (the region whose .META.'s endkey we want =
to repoint)
2) delete the region's row from .META.
3) Put a new row into .META., the new row's hregioninfo having the =
desired endkey such that there is no more "hole" in .META.

I'm trying to nail down exactly the sequence of steps we should take so =
that we don't have to do scary manual surgery of -ROOT- and .META. like =
we did yesterday. We we're pretty much fumbling around in the dark =
trying to figure out the structure of -ROOT-'s HDFS files, and same for =
.META. after our first failed attempt to update the endrow. We did =
figure it out, and removed the files from .META. and -ROOT- that =
prevented hbase from coming up. Our error was that instead of updating =
the endrow in .META. we inadvertently put a new row into .META. with =
default timestamp, and that basically shot everything to hell. I =
couldn't find docs on the structure of -ROOT- and .META. HDFS files, but =
we sort of pieced it together and we're able to remove the newly created =
files in -ROOT- and .META. based on the their creation times and =
grepping their content, after which hbase was able to come back up =
without error.

So, apologies for going slow on this, and really trying to exactly nail =
down the set of steps we should proceed with in order to avoid another =
self-inflicted corruption.

Best,
geoff


-----Original Message-----
From: Rohit Nigam=20
Sent: Monday, August 15, 2011 8:46 PM
To: Stack; user@hbase.apache.org
Cc: Search
Subject: RE: version mismatch exception

So I got the info:regioninfo value from the .META. table for the key and =
I did a put in the .META. table with the  change in the endkey. Yes I =
did change the info:regioninfo cell with just changing the  endkey. I =
didn't see the change as the master didn't come up and kept throwing the =
version mismatch exception. Somehow the version got changed , no idea =
how? , I had to run the add_table.rb for that table to restore the whole =
thing for the master to be up. So my case is to just  update the endkey =
of a row in the .META. table for a table region because the chain is =
broken ,how do I do that so that this exception doesn't happen.
Rohit

-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of =
Stack
Sent: Monday, August 15, 2011 8:29 PM
To: user@hbase.apache.org
Cc: Search
Subject: Re: version mismatch exception

On Mon, Aug 15, 2011 at 2:45 PM, Rohit Nigam <rnigam@decarta.com> wrote:
> I changed the endkey of one of the records in the '.META.' table =
because
> of the chaining issue we experienced , =A0using the program which gets
> that row and does a put so that the =A0endkey could be changed , now =
when
> I try to view the record in .META. table using shell I get a
>

Tell me how you did this?  You removed the original row and replaced
it with another that has different end key?  Did you update the
info:regioninfo cell so it has a new hregioninfo with same start and
end row?   You know this makes a new region, rather than extend the
range of the previous region? (So the old region will be in the
filesystem still with the old data).  You should also make sure the
original region is closed before you delete the row from .META.


>
>
> ERROR: org.apache.hadoop.io.VersionMismatchException: null
>

Where did you get this from?


> 2011-08-15 14:32:34,639 FATAL [master-doop10.dt.sv4.decarta.com:60000]
> master.HMaster(948): Unhandled exception. Starting shutdown.
>
> A record version mismatch occured. Expecting v0, found v116
>

What you doing when this happens?  It looks like we are deserializing
the wrong content?  Is that possible.

St.Ack