accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: How to speed up table compaction [SEC=UNOFFICIAL]
Date Tue, 06 May 2014 17:03:06 GMT
Yup, that property can be changed on the fly, and future compactions 
will use the new codec (all of the queued compactions will see it). It 
will not actively seek out and re-write files which are not compressed 
with the current codec.

And, yes, depending on the distro of Hadoop you're using, there might be 
some extra bits to install. I believe that your normal Hadoop-2.x.y 
release from Apache will have the native libs available for you and 
might just to install libsnappy via your OS's package manager.

On 5/6/14, 11:52 AM, David Medinets wrote:
> Can that property be changed on the fly? And Snappy needs to be
> installed throughout the cluster before making the configuration change?
>
>
> On Tue, May 6, 2014 at 12:43 AM, Josh Elser <josh.elser@gmail.com
> <mailto:josh.elser@gmail.com>> wrote:
>
>     Depending on the CPU/IO ratio for your system, switching to a
>     different compression codec might help. Snappy tends to be a bit
>     quicker writing out data as opposed to gzip at the cost of being
>     larger on disk. The increase in final size on disk might be
>     prohibitive depending on your requirements though.
>
>     I forget the table property off hand, but, if you haven't changed
>     this already, it will be the property with a defauly value of 'gz' :)
>
>     On May 5, 2014 6:43 PM, "Dickson, Matt MR"
>     <matt.dickson@defence.gov.au <mailto:matt.dickson@defence.gov.au>>
>     wrote:
>
>         __
>
>         *UNOFFICIAL*
>
>         I'm trying to compact a table and have had a queue of 60,000
>         compactions with only 800 running at a time.  This has now been
>         running for 4 days and only decreased the queue by 8,000.
>         I've increased tserver.compaction.major.concurrent.max=12 and
>         stopped all ingest but not seen a change in progress.  Are there
>         other accumulo settings I can alter to improve this?  I also saw
>         tserver.compaction.major.thread.files.open.max=10 should this be
>         increased?
>         Thanks in advance,
>         Matt
>
>

Mime
View raw message