hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18135) Track file archival for low latency space quota with snapshots
Date Fri, 02 Jun 2017 23:29:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035620#comment-16035620

Josh Elser commented on HBASE-18135:

bq. We might be able to push this down to "HBase" itself (use some kind of compareAndSet)
or just synchronize access in the master via some new class.

My thinking is this: for the archive "notices" sent by RS to Master, these can be pushed to
the Quota table using Increments. Row-locks server-side provide the level of exclusion we
want/need. Then, every so often, the SnapshotQuotaObserverChore will recompute a "total" size
for a snapshot which needs to be updated in the table exclusively.

These two process may race, but this can be solved with a RWLock. In this case the "read"
lock would be grabbed by the threads issuing Increments as they can happen in parallel. The
"write" lock would be grabbed by the Chore when it's updating the "total" size of the snapshot
which prevents the "readers" (the Incrementers) from submitting new Increments which may screw
up the total. Just.. ignore the fact that we're writing while holding a "read" lock ;)

> Track file archival for low latency space quota with snapshots
> --------------------------------------------------------------
>                 Key: HBASE-18135
>                 URL: https://issues.apache.org/jira/browse/HBASE-18135
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Josh Elser
>            Assignee: Josh Elser
> Related to the work proposed on HBASE-17748 and building on the same idea as HBASE-18133,
we can make the space quota tracking for HBase snapshots faster to respond.
> When snapshots are in play, the location of a file (whether in the {{data}} or {{archive}}
directory) plays a factor in the realized size of a table. Like flushes, compactions, etc,
moving files from the data directory to the archive directory is done by the RegionServer.
We can hook into this call and send the necessary information to the Master so that it can
more quickly update the size of a table when there are snapshots in play.
> This will require the RegionServer to report the full coordinates of the file being moved
(table+region+family+file) so that the SnapshotQuotaObserverChore running in the master can
avoid HDFS lookups in partial or total to compute the location of a Region's hfiles.
> This may also require some refactoring of the SnapshotQuotaObserverChore to de-couple
the receipt of these file archival reports from RegionServers (e.g. {{HRegionFileSystem.removeStoreFiles(..)}},
and the Master processing the sizes of snapshots.

This message was sent by Atlassian JIRA

View raw message