hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Latham (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
Date Fri, 31 Jul 2015 06:22:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648808#comment-14648808

Dave Latham commented on HBASE-12219:

Sorry I missed this JIRA at the time, but I have a couple of concerns if I'm understanding
this change correctly.  I'd like to check to see if I am.  It looks like the effect is to
back out the directory modtime caching and instead have the master just maintain a persistent
in memory cache.

Previous to this change it was safe to have any processes on the cluster use FSTableDescriptors
to read or write table descriptors.  Updates would be atomic and consistent, immediately available
to all readers.  However, the cost of that was a HDFS NN operation on every table descriptor
read to prove it had not changed.  Since in practice it appears that only the master process
ever updates an existing table descriptor, it should be safe to have the master skip the directory
modtime checks and proactively update its cached copy.  (hbck can also create table descriptors
for orphaned tables but hopefully those don't happen to tables the master has already cached).

This change makes the master descriptor reads faster but imposes the constraint that only
the active master should update table descriptors - any other writers would cause the master
cache to become stale.  It also means that no other processes should use the cache the same
way as the master could change the data and cause stale caches.  Assuming this is the case,
I think we'd be better served by reflecting that in the FSTableDescriptors API and javadoc.
 For example, currently most constructors and usages now default to keeping a persistent cache
as well as allowing updates which sets a bad example for new uses.  There are also no warnings
in the javadoc about the new contract.  Possibly better would be to make the default constructor
be read only and have persistent caching disabled.  Then another constructor for the master
allowing both writes and persistent caching.

This change also seems to remove all table descriptor caching from the region servers (the
old directory modtime caching is gone and the new caching is disabled for region servers).
 Thanks to HBASE-8778 reloading from the FS each time is cheaper than it used to be, but this
change still increases the cost from 1 NN operation (check directory modtime) to 2 NN + 3
DN operations (find current file, get its block locations, open block, read close block).
 This slows things down a bit again for mass assignments/balances on huge tables.  It seems
better for the region servers to retain the directory modtime caching, but simply skip the
modtime check when running inside the master.

Does that understanding of this change sound correct - or did I botch it?  Sorry I missed
it at the time.  If that sounds right, a follow up JIRA may be good, and if I see our table
assignments slower from this and no one else gets to it I can try to put up the changes.

> Cache more efficiently getAll() and get() in FSTableDescriptors
> ---------------------------------------------------------------
>                 Key: HBASE-12219
>                 URL: https://issues.apache.org/jira/browse/HBASE-12219
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.94.24, 0.99.1,
>            Reporter: Esteban Gutierrez
>            Assignee: Esteban Gutierrez
>              Labels: scalability
>             Fix For: 2.0.0, 0.98.8, 0.99.2
>         Attachments: HBASE-12219-0.94.patch, HBASE-12219-0.98.patch, HBASE-12219-0.98.v1.addendum.patch,
HBASE-12219-0.98.v1.patch, HBASE-12219-0.99.addendum.patch, HBASE-12219-0.99.patch, HBASE-12219-0.99.v1.patch,
HBASE-12219-v1.patch, HBASE-12219-v1.patch, HBASE-12219.v0.txt, HBASE-12219.v2.patch, HBASE-12219.v3.patch,
> Currently table descriptors and tables are cached once they are accessed for the first
time. Next calls to the master only require a trip to HDFS to lookup the modified time in
order to reload the table descriptors if modified. However in clusters with a large number
of tables or concurrent clients and this can be too aggressive to HDFS and the master causing
contention to process other requests. A simple solution is to have a TTL based cached for
FSTableDescriptors#getAll() and  FSTableDescriptors#TableDescriptorAndModtime() that can allow
the master to process those calls faster without causing contention without having to perform
a trip to HDFS for every call. to listtables() or getTableDescriptor()

This message was sent by Atlassian JIRA

View raw message