accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris McCubbin (JIRA)" <>
Subject [jira] [Updated] (ACCUMULO-1787) support two tier compression codec configuration
Date Sun, 24 Nov 2013 05:24:36 GMT


Chris McCubbin updated ACCUMULO-1787:


Also if it is of interest this is my simple rig to time the length of ingests. It uses the
same APIs to get info as the monitor page. It also computes things like area under the major
compaction chart and area under the load average chart. It's not commented at all so use at
your own risk.

For du in the above chart I used the accumulo shell du command.

> support two tier compression codec configuration
> ------------------------------------------------
>                 Key: ACCUMULO-1787
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>            Reporter: Adam Fuchs
>         Attachments:, ci_file_sizes.png, hybrid.diff
> Given our current configuration of one compression codec per table we have the option
of leaning towards performance with something like snappy or leaning towards smaller footprint
with something like gzip. With a change to the way we configure codecs we might be able to
approach the best of both worlds. Consider the difference between files that have been written
by major or minor compactions and files that exist at any given point in time. For better
footprint on disk we care about the latter, but for total CPU usage over time we care about
the former. The two distributions are distinct because Accumulo deletes files after major
compactions. If we figure out whether a file is going to be long-lived at the time we write
it then we can pick the compression codec that optimizes the relevant concern.
> One way to distinguish is by file size. Accumulo writes many small files and later major
compacts those away, so the distribution of written files is skewed towards smaller files
while the distribution of files existing at any point in time is skewed towards larger files.
I recommend for each table we support a general compression codec and a second codec for files
under a configurable size.

This message was sent by Atlassian JIRA

View raw message