phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geoffrey Jacoby <gjac...@apache.org>
Subject Re: Regarding the Secondary Index write path
Date Tue, 30 Oct 2018 18:25:28 GMT
Abhishek,

You might want to check out Vincent Poon's excellent presentation at this
year's PhoenixCon about recent changes over the past couple of years to the
index pipeline.

https://www.youtube.com/watch?v=VBONDM7sD40

One of those changes is the one you observed. Global mutable index writes
were moved later in the HBase write pipeline to avoid some nasty deadlock
and starvation cases that could occur when making MemStore writes / MVCC
advancement wait on cross-server index RPCs to complete.

Geoffrey


On Tue, Oct 30, 2018 at 10:31 AM Abhishek Talluri
<abhishektalluri@cloudera.com.invalid> wrote:

> Hi All,
>
> I am referring to the presentation that is given in SF Hbase Meetup in
> 2013, attached it for your reference. The write path states that,
> co-processor calculates the index updates and wal edits
> -> Writes it to the WAL (Making it durable)
> -> Then write the index updates
> -> Then proceed to the Memstore.
>
> But after looking at the code base (looked into 4.7 and 4.14), it looks
> like the write to index tables happen in the postBatchMutate phase which is
> after the MemStore write finishes. I wanted to check with the community to
> see if the flowchart is outdated. I feel that series of steps should be:
> co-processor calculates the index updates and wal edits
> -> Writes it to the WAL (Making it durable)
> -> Then proceed to the Memstore.
> -> Then write the index updates
>
> Appreciate any input on this. I want to clarify this because we see a case
> where the write path could be creating some delay between write to WAL and
> MemStore and creating some out of sync issue when using hbase lily indexer.
>
> Thanks,
> Abhishek Talluri
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message