hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Friso van Vollenhoven <fvanvollenho...@xebia.com>
Subject RE: Data loss on clean RS shutdown without WAL?
Date Fri, 21 Jan 2011 17:54:43 GMT
Thanks J-D.

With cleanly, I meant that it did not produce any errors other than the YouAreDeadException.
When a very long pause causes this I often also see HDFS client related errors because of
leases expiring, but this time that was not the case. It's not "cleanly" in the sense that
I issued the shutdown myself.


Friso


________________________________________
Van: jdcryans@gmail.com [jdcryans@gmail.com] namens Jean-Daniel Cryans [jdcryans@apache.org]
Verzonden: vrijdag 21 januari 2011 18:32
Aan: user@hbase.apache.org
Onderwerp: Re: Data loss on clean RS shutdown without WAL?

If the region servers gets YouAreDeadException, it does an "abort" and
won't flush the data since another region server could already be
serving the region. If you're not writing to the WAL, then yes it's
data loss.

Not sure what you mean by "shuts down cleanly" in your case, if you
see a log that starts with "Aborting region server" then it's not
really "clean".

J-D

On Fri, Jan 21, 2011 at 2:38 AM, Friso van Vollenhoven
<fvanvollenhoven@xebia.com> wrote:
> Hi all,
>
> Question: when a regionserver shuts down cleanly after a YouAreDeadException and the
master nicely reassigns the regions, will you loose any data that was written to the memstore
of the dead RS when not using WAL?
>
> There was no hard crash and not a single error in any of the logs (except for the FATAL:
YouAreDeadException). The RS lost its zookeeper session after a timeout, probably GC combined
with some other starvation on heavy load. I think the memstore flushes on shutdown, but I
am not entirely sure what happens in the situation where regions are already opened by other
regionservers when the dying executes the shutdown code. Can I assume that the RS that gets
reassigned a region creates a new HFile and that this will be compacted together with the
one left by the dead RS at the next compaction run?
>
>
> Thanks,
> Friso
>
>

Mime
View raw message