hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 张铎(Duo Zhang) <palomino...@gmail.com>
Subject Re: Violating strong consistency after the fact
Date Fri, 03 Jul 2020 13:36:30 GMT
You can see my design doc for async dfs output

https://docs.google.com/document/d/11AyWtGhItQs6vsLRIx32PwTxmBY3libXwGXI25obVEY/edit#heading=h.2jvw6cxnmirr


See the footnote below section 3.4. For the current HDFS pipeline
implementation, it could be a problem for replication in HBase, though it
rarely happens.

And now HBase has its own AsyncFSWAL implementation, HBASE-14004 is used to
resolve the problem(although later we make things wrong and HBASE-24625 is
the fix).

And for WAL recovery, it will not be a problem. We will only return success
to client after all the replicas have been successfully committed, so if
DN2 goes offline, we will close the current file and commit it, and open a
new file to write WAL.

Thanks.

Paul Carey <paul.p.carey@gmail.com> 于2020年7月3日周五 下午7:40写道:

> >  If the hdfs write succeeded while u had only one DN available, then the
> other replica on the offline DN would be invalid now.
>
> Interesting, I wasn't aware of this. Are there any docs you could
> point me towards where this is described? I've had a look in Hadoop:
> The Definitive Guide and the official docs, but hadn't come across
> this.
>
> On Fri, Jul 3, 2020 at 11:19 AM Wellington Chevreuil
> <wellington.chevreuil@gmail.com> wrote:
> >
> > This is actually an hdfs consistency question, not hbase. If the hdfs
> write
> > succeeded while u had only one DN available, then the other replica on
> the
> > offline DN would be invalid now. Then what u have is an under replicated
> > block, and of your only available DN goes offline before it could be
> > replicated, the file that block belongs to now is corrupt. If I turn on
> the
> > previous offline DN, it would still be corrupt as the replica it has is
> not
> > valid anymore (NN knows which is the last valid version of the replica),
> so
> > unless u can bring back the DN that has the only valid replica, your
> hfilr
> > is corrupt and your data is lost.
> >
> > On Fri, 3 Jul 2020, 09:12 Paul Carey, <paul.p.carey@gmail.com> wrote:
> >
> > > Hi
> > >
> > > I'd like to understand how HBase deals with the situation where the
> > > only available DataNodes for a given offline Region contain stale
> > > data. Will HBase allow the Region to be brought online again,
> > > effectively making the inconsistency permanent, or will it refuse to
> > > do so?
> > >
> > > My question is motivated from seeing how Kafka and Elasticsearch
> > > handle this scenario. They both allow the inconsistency to become
> > > permanent, Kafka via unclean leader election, and Elasticsearch via
> > > the allocate_stale_primary command.
> > >
> > > To better understand my question, please consider the following
> example:
> > >
> > > - HDFS is configured with `dfs.replication=2` and
> > > `dfs.namenode.replication.min=1`
> > > - DataNodes DN1 and DN2 contain the blocks for Region R1
> > > - DN2 goes offline
> > > - R1 receives a writes which succeeds as it can be written
> successfully to
> > > DN1
> > > - DN1 goes offline before the NameNode can replicate the
> > > under-replicated block containing the write to another DataNode
> > > - At this point the R1 is offline
> > > - DN2 comes back online, but it does not contain the missed write
> > >
> > > There are now two options:
> > >
> > > - R1 is brought back online, violating consistency
> > > - R1 remains offline, indefinitely, until DN1 is brought back online
> > >
> > > How does HBase deal with this situation?
> > >
> > > Many thanks
> > >
> > > Paul
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message