Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <5183CAFD.8070301@neofonie.de>
Date: Fri, 03 May 2013 16:34:37 +0200
From: Dimitri Goldin <dimitri.goldin@neofonie.de>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130308 Thunderbird/17.0.4
MIME-Version: 1.0
To: user@hbase.apache.org
Subject: Re: Eternal RIT problem when RS tries to access wrong region-folder
 on HDFS
References: <5182758B.1060306@neofonie.de>
 <CAGngS9eRcPRJ4UL2QD=_byeNht3ioCX=OPgynb6H=bAXe6Bg=w@mail.gmail.com>
In-Reply-To: 
 <CAGngS9eRcPRJ4UL2QD=_byeNht3ioCX=OPgynb6H=bAXe6Bg=w@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

Hi Kevin,

On 05/03/2013 02:57 PM, Kevin O'dell wrote:
 > That is interesting.  I have seen this before, can you please send a
 > hadoop fs -lsr /hbase/documents?  This is going to be caused by a bad
 > split.  I will let you know what files you need to delete to safely
 > recover from this error.

Thanks for the reply. Earlier today I also determined that it has to
do with a failed region-split and already tried to solve it
on my own.

I found a total of three reference files in the folder and two hfiles.
Unfortunately documents contains more than 5k regions, so it seems a
little impractical to send the listing to the list. Please let me know
if you'd still like to see it and I will send it to you directly.

original contents of /hbase/documents/79c619508659018ff3ef0887611eb8f7/d*:
==
0707b1ec4c6b41cf9174e0d2a1785fe9.5b9c16898a371de58f31f0bdf86b1f8b
47511faae81b4452afd3ca206e28346f.5b9c16898a371de58f31f0bdf86b1f8b
4f01ecd052ce464d81e79a62ea227d6b (116MB)
4f01ecd052ce464d81e79a62ea227d6b.5b9c16898a371de58f31f0bdf86b1f8b
eb7dbb09701d4353be24ca82481c4a7e (951MB)
==

* d is the only Columnfamily

Additionally, there was an 'almost empty' recovered.edits referencing
the old parent region and containing only a CACHEFLUSH.

As mentioned, '5b9c16898a371de58f31f0bdf86b1f8b' did not exist anymore
,.tmp was empty and .META. entry did not contain any splitA/splitB
columns, so I backed up the original region folder, removed the
reference files and kept 4f01ecd052ce464d81e79a62ea227d6b
and eb7dbb09701d4353be24ca82481c4a7e for now to get the table working
again.

I am still trying to locate log entries from the split, but haven't
found them yet.

Do you think this was an appropriate measure? Please let me know if
you had a different approach in mind and I'll see if I can use the
backed-up region. Also, any ideas under which circumstances this
might occur/is there a JIRA I can follow and maybe try to contribute
observations from logs?

Thanks a lot,
	Dimitry
-- 
----------------------------------
Dimitry Goldin
Software Developer

Neofonie GmbH
Robert-Koch-Platz 4
10115 Berlin

T: +49 30 246 27

goldin@neofonie.de <mailto:goldin@neofonie.de>
http://www.neofonie.de

Handelsregister
Berlin-Charlottenburg: HRB 67460

Gesch�ftsf�hrung:
Thomas Kitlitschko