accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1015) Lock a table to memory
Date Wed, 23 Apr 2014 17:51:14 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978531#comment-13978531
] 

Josh Elser commented on ACCUMULO-1015:
--------------------------------------

bq. How much of this would we get with a combination of HDFS caching and HDFS short circuit
reads?

Well, you'll still be incurring the overhead of the IPC to the datanode and reading the data
from there, so I imagine it is still "significant" as opposed to completely in memory. I think
it would likely depend on the level of implementation of an in-memory table. If we just keep
the HDFS blocks in the tserver memory, that would be fairly "simple" and should still perform
better, but I don't have any idea of actual numbers. That would be a good first step that
someone could evaluate without implementing anything new

in-memory
1. Load some data in a table
2. Warm the caches for that table
3. Test performance

short-circuit reads
1. Turn off caches
2. Ingest data into a table
3. Compact to disk
4. Test performance

> Lock a table to memory
> ----------------------
>
>                 Key: ACCUMULO-1015
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1015
>             Project: Accumulo
>          Issue Type: New Feature
>          Components: tserver
>            Reporter: Josh Elser
>            Priority: Trivial
>
> Index block and data block caching get you part of the way, but it would be nice to have
the ability to lock an entire table to memory. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message