hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: A data loss scenario with a single region server going down
Date Tue, 21 Sep 2010 00:43:55 GMT
hi,

sorry i dont.  i think the current transactional/indexed person is
working on bringing it up to 0.89, perhaps they would enjoy your help
in testing or porting the code?

I'll poke a few people into replying.

-ryan

On Mon, Sep 20, 2010 at 5:19 PM, George P. Stathis <gstathis@traackr.com> wrote:
> On Mon, Sep 20, 2010 at 4:55 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
>> When you say replication what exactly do you mean?  In normal HDFS, as
>> you write the data is sent to 3 nodes yes, but with the flaw I
>> outlined, it doesnt matter because the datanodes and namenode will
>> pretend a data block just didnt exist if it wasnt closed properly.
>>
>
> That's the part I was not understanding. I do now. Thanks.
>
>
>>
>> So even with the most careful white glove handling of hbase, you will
>> eventually have a crash and you will lose data w/o 0.89/CDH3 et. al.
>> You can circumvent this by storing the data elsewhere and spooling
>> into hbase, or perhaps just not minding if you lose data (yes those
>> applications exist).
>>
>> Looking at those JIRAs in question, the first is already on trunk
>> which is 0.89.  The second isn't alas.  At this point the
>> transactional hbase just isnt being actively maintained by any
>> committer and we are reliant on kind people's contributions.  So I
>> can't promise when it will hit 0.89/0.90.
>>
>
> Are you aware of any indexing alternatives in 0.89?
>
>
>>
>> -ryan
>>
>>
>> On Mon, Sep 20, 2010 at 1:21 PM, George P. Stathis <gstathis@traackr.com>
>> wrote:
>> > Thanks for the response Ryan. I have no doubt that 0.89 can be used in
>> > production and that it has strong support. I just wanted to avoid moving
>> to
>> > it now because we have limited resources and it would put a dent in our
>> > roadmap if we were to fast track the migration now. Specifically, we are
>> > using HBASE-2438 and HBASE-2426 to support pagination across indexes. So
>> we
>> > either have to migrate those to 0.89 or somehow go stock and be able to
>> > support pagination across region servers.
>> >
>> > Of course, if the choice is between migrating or losing more data, data
>> > safety comes first. But if we can buy two or three more months of time
>> and
>> > avoid region server crashes (like you did for a year), maybe we can go
>> that
>> > route for now. What do we need to do achieve that?
>> >
>> > -GS
>> >
>> > PS: Out of curiosity, I understand the WAL log append issue for a single
>> > regionserver when it comes to losing the data on a single node. But if
>> that
>> > data is also being replicated on another region server, why wouldn't it
>> be
>> > available there? Or is the WAL log shared across multiple region servers
>> > (maybe that's what I'm missing)?
>> >
>> >
>> > On Mon, Sep 20, 2010 at 3:52 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>> >
>> >> Hey,
>> >>
>> >> The problem is that the stock 0.20 hadoop wont let you read from a
>> >> non-closed file.  It will report that length as 0.  So if a
>> >> regionserver crashes, that last WAL log that is still open becomes 0
>> >> length and the data within in unreadable.  That specifically is the
>> >> problem of data loss.  You could always make it so your regionservers
>> >> rarely crash - this is possible btw and I did it for over a year.
>> >>
>> >> But you will want to run CDH3 or the append-branch releases to get the
>> >> series of patches that fix this hole.  It also happens that only 0.89
>> >> runs on it.  I would like to avoid the hadoop "everyone uses 0.20
>> >> forever" problem and talk about what we could do to help you get on
>> >> 0.89.  Over here at SU we've made a commitment to the future of 0.89
>> >> and are running it in production.  Let us know what else you'd need.
>> >>
>> >> -ryan
>> >>
>> >> On Mon, Sep 20, 2010 at 12:39 PM, George P. Stathis
>> >> <gstathis@traackr.com> wrote:
>> >> > Thanks Todd. We are not quite ready to move to 0.89 yet. We have made
>> >> custom
>> >> > modifications to the transactional contrib sources which are now taken
>> >> out
>> >> > of 0.89. We are planning on moving to 0.90 when it comes out and at
>> that
>> >> > point, either migrate our customizations, or move back to the
>> out-of-the
>> >> box
>> >> > features (which will require a re-write of our code).
>> >> >
>> >> > We are well aware of the CDH distros but at the time we started with
>> >> hbase,
>> >> > there was none that included HBase. I think CDH3 the first one to
>> include
>> >> > HBase, correct? And is 0.89 the only one supported?
>> >> >
>> >> > Moreover, are we saying that there is no way to prevent stock hbase
>> >> 0.20.6
>> >> > and hadoop 0.20.2 from losing data when a single node goes down? It
>> does
>> >> not
>> >> > matter if the data is replicated, it will still get lost?
>> >> >
>> >> > -GS
>> >> >
>> >> > On Sun, Sep 19, 2010 at 5:58 PM, Todd Lipcon <todd@cloudera.com>
>> wrote:
>> >> >
>> >> >> Hi George,
>> >> >>
>> >> >> The data loss problems you mentioned below are known issues when
>> running
>> >> on
>> >> >> stock Apache 0.20.x hadoop.
>> >> >>
>> >> >> You should consider upgrading to CDH3b2, which includes a number
of
>> HDFS
>> >> >> patches that allow HBase to durably store data. You'll also have
to
>> >> upgrade
>> >> >> to HBase 0.89 - we ship a version as part of CDH that will work
well.
>> >> >>
>> >> >> Thanks
>> >> >> -Todd
>> >> >>
>> >> >> On Sun, Sep 19, 2010 at 6:57 AM, George P. Stathis <
>> >> gstathis@traackr.com
>> >> >> >wrote:
>> >> >>
>> >> >> > Hi folks. I'd like to run the following data loss scenario
by you
>> to
>> >> see
>> >> >> if
>> >> >> > we are doing something obviously wrong with our setup here.
>> >> >> >
>> >> >> > Setup:
>> >> >> >
>> >> >> >   - Hadoop 0.20.1
>> >> >> >   - HBase 0.20.3
>> >> >> >   - 1 Master Node running Nameserver, SecondaryNameserver,
>> JobTracker,
>> >> >> >   HMaster and 1 Zookeeper (no zookeeper quorum right now)
>> >> >> >   - 4 child nodes running a Datanode, TaskTracker and RegionServer
>> >> each
>> >> >> >   - dfs.replication is set to 2
>> >> >> >   - Host: Amazon EC2
>> >> >> >
>> >> >> > Up until yesterday, we were frequently experiencing
>> >> >> > HBASE-2077<https://issues.apache.org/jira/browse/HBASE-2077>,
>> >> >> > which kept bringing our RegionServers down. What we realized
though
>> is
>> >> >> that
>> >> >> > we were losing data (a few hours worth) with just one out
of four
>> >> >> > regionservers going down. This is problematic since we are
supposed
>> to
>> >> >> > replicate at x2 out of 4 nodes, so at least one other node
should
>> be
>> >> able
>> >> >> > to
>> >> >> > theoretically serve the data that the downed regionserver
can't.
>> >> >> >
>> >> >> > Questions:
>> >> >> >
>> >> >> >   - When a regionserver goes down unexpectedly, the only
data that
>> >> >> >   theoretically gets lost was whatever didn't make it to
the WAL,
>> >> right?
>> >> >> Or
>> >> >> >   wrong? E.g.
>> >> >> >
>> >> >> >
>> >> >>
>> >>
>> http://www.larsgeorge.com/2010/01/hbase-architecture-101-write-ahead-log.html
>> >> >> >   - We ran a hadoop fsck on our cluster and verified the
>> replication
>> >> >> factor
>> >> >> >   as well as that the were no under replicated blocks. So
why was
>> our
>> >> >> data
>> >> >> > not
>> >> >> >   available from another node?
>> >> >> >   - If the log gets rolled every 60 minutes by default (we
haven't
>> >> >> touched
>> >> >> >   the defaults), how can we lose data from up to 24 hours
ago?
>> >> >> >   - When the downed regionserver comes back up, shouldn't
that data
>> be
>> >> >> >   available again? Ours wasn't.
>> >> >> >   - In such scenarios, is there a recommended approach for
>> restoring
>> >> the
>> >> >> >   regionserver that goes down? We just brought them back
up by
>> logging
>> >> on
>> >> >> > the
>> >> >> >   node itself an manually restarting them first. Now we have
>> automated
>> >> >> > crons
>> >> >> >   that listen for their ports and restart them if they go
down
>> within
>> >> two
>> >> >> >   minutes.
>> >> >> >   - Are there way to recover such lost data?
>> >> >> >   - Are versions 0.89 / 0.90 addressing any of these issues?
>> >> >> >   - Curiosity question: when a regionserver goes down, does
the
>> master
>> >> >> try
>> >> >> >   to replicate that node's data on another node to satisfy
the
>> >> >> > dfs.replication
>> >> >> >   ratio?
>> >> >> >
>> >> >> > For now, we have upgraded our HBase to 0.20.6, which is supposed
to
>> >> >> contain
>> >> >> > the HBASE-2077 <https://issues.apache.org/jira/browse/HBASE-2077>
>> fix
>> >> >> (but
>> >> >> > no one has verified yet). Lars' blog also suggests that Hadoop
>> 0.21.0
>> >> is
>> >> >> > the
>> >> >> > way to go to avoid the  file append issues but it's not production
>> >> ready
>> >> >> > yet. Should we stick to 0.20.1? Upgrade to 0.20.2?
>> >> >> >
>> >> >> > Any tips here are definitely appreciated. I'll be happy to
provide
>> >> more
>> >> >> > information as well.
>> >> >> >
>> >> >> > -GS
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Todd Lipcon
>> >> >> Software Engineer, Cloudera
>> >> >>
>> >> >
>> >>
>> >
>>
>

Mime
View raw message