accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-2613) Take advantage of HDFS caching to improve MTTR
Date Wed, 02 Apr 2014 17:31:24 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957915#comment-13957915
] 

Eric Newton commented on ACCUMULO-2613:
---------------------------------------

I don't want to seem argumentative, because I really don't know if using this cache for the
WAL is a good idea, or not.  But I can think of some issues:

 * hopefully, in your clusters, recovery is an unusual operation
 * WAL has to write to disk to survive power loss, making it a bad candidate for RAM-only
storage
 * Others have purposefully turned off caching of WAL data to make memory available for other
things, since reading them at all is unusual

We already know we can improve recovery time by reducing the largest WAL size, parallelizing
read/sort, and computing a more optimal leaseRecovery timeout.  I would strongly suggest a
more in-depth look into recovery before even experimenting with HDFS caching.

> Take advantage of HDFS caching to improve MTTR
> ----------------------------------------------
>
>                 Key: ACCUMULO-2613
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2613
>             Project: Accumulo
>          Issue Type: Improvement
>            Reporter: Sean Busbey
>            Priority: Critical
>              Labels: recovery
>             Fix For: 1.7.0
>
>
> Hadoop 2.3.0 added [HDFS caching|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html].
> We should use this for small internal use tables (like !METADATA) and we should probably
have a configurable option to use it for tables, with a stern warning that it should only
be enabled on small tables that will be frequently used.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message