accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <>
Subject Re: Time based locality groups
Date Thu, 08 Mar 2012 02:05:03 GMT
I was thinking of something like the following.   A locality group in
an rfile would be comprised of arbitrary locality group metadata and
key value pairs.

interface Partitioner {

    void init(LocalityGroupConfig lgc);

    //method for determining what locality groups a compaction should
create in the output RFile
    //this method recieves metadata about locality groups in the files
being compacted
getLocalityGroupsToCreate(List<LocalityGroupMetadata> lgml);

    //the following three methods are used to write data into a new
RFile locality group
    void startLocalityGroup(LocalityGroupInfo lgi);
    //all data is passed throug this method it serves two purposes
decide if data even goes in a locality group
    //and for the data that is accepted build up the metadata for the
locality group being created
    boolean acceptKeyValue(Key k, Value v);
    //once all data is written ask for the metadata and write that to the RFile
    LocalityGroupMetadata finishLocalityGroup();

    //method to select which locality groups in a RFile should be read
by a scan or compaction
    //this method is passed info about the existing locality groups in an RFile
getLocalityGroupsToRead(List<LocalityGroupMetadata> lgml, ScanOptions


On Wed, Mar 7, 2012 at 7:39 PM, Eric Newton <> wrote:
> Something like this:
>    partition, meta = partitioner.choose(key, value, meta)
> The partition can be a string, which is used to look up the partitions'
> configuration.  The meta information can be used by queries to avoid
> including files from the partition in queries.  The metadata would be saved
> at the close of the file.
> During a query, files could be filtered based on some arbitrary query data:
>    files = partitioner.selectFiles(files, query)
> I like it! It might also be nice to indicate some sort of "estimated"
> percent of keys processed, and the type of compaction occurring (flush,
> partial, everything):
>    partition, meta = partitioner.choose(key, value, meta, percent,
> compactionType)
> Is there any other tablet-level information we might want to provide to a
> partitioner?  Perhaps the source partition of the key/value?
> -Eric
> On Wed, Mar 7, 2012 at 6:54 PM, Keith Turner <> wrote:
>> Replying to myself :)
>> The more I think about this, it seems that locality groups could
>> handled by plugins that can parition the data and select locality
>> groups in any way it likes. Want locality groups based on row suffix,
>> go ahead and write the plugin.
>> The plugin would be used for compaction time partitioning and scan
>> time locality group selection.   User could pass options to the
>> locality group plugin at scan time just like options are passed to
>> iterators.    Maybe this is an extension or further generalization of
>> the existing iterator framework, I have not thought through that far
>> enough.
>> Keith
>> On Wed, Mar 7, 2012 at 6:22 PM, Keith Turner <> wrote:
>> > We regularly have questions from users about querying new data and
>> > aging off old data.  I was thinking about how we could better support
>> > this in need in 1.5.  One thing that occurred to me is having locality
>> > groups that were based on timestamp instead of column family.  For
>> > example a locality group for each month.   Alternatively we could have
>> > group for < day old, < week old, < month old, < year old.  Would
>> > a way for users to define these.
>> >
>> > This would make scanning a table for recent data much faster.  Also
>> > dropping old data could be made much faster by just dropping entire
>> > locality groups at compaction time.
>> >
>> > One thing that irks me about this is : Should column family and time
>> > based locality groups be mutually exclusive (i.e. an RFile has one or
>> > the other, not both)?  If they are not then order of which is
>> > partitioned first is important for query performance and would
>> > probably need to be user configurable.
>> >
>> > Thoughts?
>> >
>> > Keith

View raw message