accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-1534) Tablet Server using large number of decompressors during a scan
Date Thu, 05 Dec 2013 22:17:41 GMT


Keith Turner commented on ACCUMULO-1534:

I ran some test adjusting tfile.fs.input.buffer.size and it seems like 32K is a good setting.
 I generated an Rfile with 5 million entries using TestIngest 

The table below shows times for scanning the RFile directly with different buffer sizes. 
The times are averages of running the test many times.  

||buf size||time in ms||

I also tried randomly seeking within the file with different buffer sizes.  

||buffer size||avg seek time in ms||

I also tried scanning through Accumulo w/ 32k and 256k buffers and did not really see a difference.

> Tablet Server using large number of decompressors during a scan
> ---------------------------------------------------------------
>                 Key: ACCUMULO-1534
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>    Affects Versions: 1.4.3
>            Reporter: Mike Drob
>            Assignee: Keith Turner
>            Priority: Critical
>             Fix For: 1.5.1, 1.6.0
> I believe this issue is similar to ACCUMULO-665. We've run into a situation where a complex
iterator tree creates a large number of decompressors from the underlying CodecPool for serving
scans. Each decompressor holds on to a large buffer and the total volume ends up killing the
> We have verified that turning off compression makes this problem go away.

This message was sent by Atlassian JIRA

View raw message