accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-790) RFile should compress using common prefixes of key elements
Date Tue, 20 Nov 2012 18:16:59 GMT


Hudson commented on ACCUMULO-790:

Integrated in Accumulo-Trunk #561 (See [])
    ACCUMULO-790 Added missing version 6 test resource for testing prior version of RFile
with latest code (Revision 1411754)

     Result = SUCCESS
ctubbsii : 
Files : 
* /accumulo/trunk/core/src/test/resources/org/apache/accumulo/core/file/rfile/ver_6.rf

> RFile should compress using common prefixes of key elements
> -----------------------------------------------------------
>                 Key: ACCUMULO-790
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: tserver
>            Reporter: Christopher Tubbs
>            Assignee: Christopher Tubbs
>              Labels: compression, file, hackathon, optimization, rfile
>             Fix For: 1.5.0
> Relative keys have proven themselves as a great way to compress within dimensions of
the key. However, we could probably do better, since we know that our data is sorted lexicographically,
we can make a reasonable assumption that we will get better compression if we only store the
fact that a key (or portion of a key) has a common prefix with the previous key, even if it
is not an exact match.
> Currently, in RFile, unused bits from the delete flag byte are being used to store flags
that show whether an element of the key is exactly the same as the previous, or if it is different.
We can change the semantics of these flags to store three states per element of the key: exact
match as previous key, has a common prefix as previous key, no relative key compression. If
we don't want to add a byte to store 2 bits for 3 states per element, we can just take the
ordinal value of the unused 7 bits of the delete flag field and map it to an enumeration of
relative key flags.
> In the case of a common prefix flag enabled for a given element of the current key when
reading the RFile, we can interpret the first bytes of that element as a VInt expressing the
length of the common prefix relative to the previous key's same element. Because this will
add at least one byte to the the length of that element, we will not want to use the common
prefix compression if the common prefix is less than 2 bytes. For less than 2 bytes in common
(1 or 0 bytes in common), we'd select the no compression flag for that element.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message