hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enis Söztutar <e...@apache.org>
Subject Re: [DISCUSS] Merging Hybrid Logical Clocks with Master
Date Thu, 24 Aug 2017 01:18:58 GMT
Thanks Amit for continuing on the work.

I am biased, but of course would like to see this merged back to master
(and possibly make it to 2.0) once the issues are ironed out. We can
eliminate a very large class of problems with a mathematically sound
strategy.

Enis

On Tue, Aug 22, 2017 at 1:28 PM, Stack <stack@duboce.net> wrote:

> Look at the pretty HLC timestamps in meta after a run:
>
>  hbase:meta
>           column=table:state, timestamp=1576463246879096833, value=\x08\x00
>  hbase:namespace
>            column=table:state, timestamp=1576460369910562817,
> value=\x08\x00
>  hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
>           column=info:regioninfo, timestamp=1576462593304821761,
> value={ENCODED => 6d365281d5ca8ee0bc0fa12cdf33ecb5, NAME =>
> 'hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.',
> STARTKEY => '', ENDKEY => ''}
>  hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
>           column=info:seqnumDuringOpen, timestamp=1576462593304821761,
> value=\x00\x00\x00\x00\x00\x00\x00\x15
>  hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
>           column=info:server, timestamp=1576462593304821761, value=
> ve0536.halxg.cloudera.com:16020
>  hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
>           column=info:serverstartcode, timestamp=1576462593304821761,
> value=1503431013602
>  hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
>           column=info:sn, timestamp=1576462593050017795, value=
> ve0536.halxg.cloudera.com,16020,1503431013602
>  hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
>           column=info:state, timestamp=1576462593304821761, value=OPEN
>
> St.Ack
>
>
>
> On Tue, Aug 22, 2017 at 1:26 PM, Stack <stack@duboce.net> wrote:
>
> > Nice work Amit. Idea of merging sounds good to me. Our metadata system
> > will run w/ a new rigor as HLC allows us leave behind a whole class of
> > issues.
> >
> > I ran the HLC branch on small cluster for a while w/ chaos. The ITBLL job
> > passed so basically works I'd say. +1 from me on merge.
> >
> > St.Ack
> >
> > On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <amit.patel@cloudera.com>
> > wrote:
> >
> >> Hi everyone,
> >>
> >> I’d like to begin the discussion of merging the current HLC work with
> >> master. Note that the current work is not ready to merge yet since there
> >> are a few remaining issues (see below). For reference, HBASE-14070
> >> <https://issues.apache.org/jira/browse/HBASE-14070> proposes using
> hybrid
> >> logical clocks, which combine both logical and physical clocks to
> capture
> >> not only causality relationship between events, but also physical time
> at
> >> which events occur. HLC can be enabled on a per-table basis and at the
> >> moment just the meta table has HLC enabled by default (the other tables
> >> default to use the system clock). An initial design document created by
> >> Enis Soztutar can be found here
> >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LD
> >> T4e_bXy8P9h6kWC05Bhw/edit#>.
> >> The more recent status document created by Sai Teja Ranuva can be found
> >> here
> >> <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_NuF8TbQ
> >> kZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>.
> >>
> >>
> >> Changes
> >>
> >> The new classes that would be introduced include: a Clock interface with
> >> three clock implementations mentioned above and enums for the clock and
> >> timestamp type.  The three clock implementations are hybrid logical,
> >> system
> >> monotonic, and system clocks and each generate 64-bit timestamps.
> >> Timestamps from the system monotonic and system clock represent physical
> >> time and are directly backwards compatible.
> >>
> >> Users can set the clock type of a table during table creation. A table’s
> >> clock type cannot be changed after the table has been created. For
> tables
> >> whose clock type is HLC, users would not be permitted to set their own
> >> timestamps for mutations. Certain events such as region assignment,
> >> unassignment, and recovery update the clocks upon receiving timestamps
> >> externally.
> >>
> >> Benefits
> >>
> >>
> >>    -
> >>
> >>    Hybrid logical clocks (and even system monotonic clocks) can solve a
> >>    variety of long standing issues related to time highlighted here
> >>    <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4L
> >> DT4e_bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>.
> >>
> >>    -
> >>
> >>    Future work with hybrid logical clocks would include enabling them
> with
> >>    user tables. Ideally we would want HLC to be used everywhere. Further
> >> work
> >>    such as global point-in-time snapshots can leverage HLC
> >>
> >>
> >> Remaining issues
> >>
> >> There are still a few remaining issues to close out before actually
> >> merging, but I wanted to at least start the discussion. Currently I am
> >> working on fixing remaining tests that are either failing or timing out
> in
> >> the public branch <https://builds.apache.org/job/HBASE-14070.HLC/> as
> >> well
> >> as doing local performance tests, and cluster tests. Some of the
> remaining
> >> issues we are tracking are:
> >>
> >>
> >>    -
> >>
> >>    HBASE-18542 (Perf numbers)
> >>    <https://issues.apache.org/jira/browse/HBASE-18542>
> >>    -
> >>
> >>    HBASE-18508 (Fixing unit tests)
> >>    <https://issues.apache.org/jira/browse/HBASE-18508>
> >>    -
> >>
> >>    HBASE-18432 (Clock skew)
> >>    <https://issues.apache.org/jira/browse/HBASE-18432>
> >>
> >> Some brainstorming/discussions we also are tracking:
> >>
> >>    -
> >>
> >>    (HBASE-18643) Adding transaction id to Result
> >>    <https://issues.apache.org/jira/browse/HBASE-18643>
> >>    -
> >>
> >>    (HBASE-18642) Deprecate setting of timestamp in client for HLC
> >>    <https://issues.apache.org/jira/browse/HBASE-18642>
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message