accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-1015) Lock a table to memory
Date Wed, 23 Apr 2014 17:51:14 GMT


Josh Elser commented on ACCUMULO-1015:

bq. How much of this would we get with a combination of HDFS caching and HDFS short circuit

Well, you'll still be incurring the overhead of the IPC to the datanode and reading the data
from there, so I imagine it is still "significant" as opposed to completely in memory. I think
it would likely depend on the level of implementation of an in-memory table. If we just keep
the HDFS blocks in the tserver memory, that would be fairly "simple" and should still perform
better, but I don't have any idea of actual numbers. That would be a good first step that
someone could evaluate without implementing anything new

1. Load some data in a table
2. Warm the caches for that table
3. Test performance

short-circuit reads
1. Turn off caches
2. Ingest data into a table
3. Compact to disk
4. Test performance

> Lock a table to memory
> ----------------------
>                 Key: ACCUMULO-1015
>                 URL:
>             Project: Accumulo
>          Issue Type: New Feature
>          Components: tserver
>            Reporter: Josh Elser
>            Priority: Trivial
> Index block and data block caching get you part of the way, but it would be nice to have
the ability to lock an entire table to memory. 

This message was sent by Atlassian JIRA

View raw message