accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1534) Tablet Server using large number of decompressors during a scan
Date Thu, 05 Dec 2013 22:17:41 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840634#comment-13840634
] 

Keith Turner commented on ACCUMULO-1534:
----------------------------------------

I ran some test adjusting tfile.fs.input.buffer.size and it seems like 32K is a good setting.
 I generated an Rfile with 5 million entries using TestIngest 

The table below shows times for scanning the RFile directly with different buffer sizes. 
The times are averages of running the test many times.  

||buf size||time in ms||
|1k|5020|
|2k|4681|
|4k|4463|
|8k|4249|
|16k|4160|
|32k|4073|
|64k|4052|
|128k|4117|
|256k|4179|

I also tried randomly seeking within the file with different buffer sizes.  

||buffer size||avg seek time in ms||
|1k|4.2|
|32k|3.8|
|256k|4.1|

I also tried scanning through Accumulo w/ 32k and 256k buffers and did not really see a difference.

> Tablet Server using large number of decompressors during a scan
> ---------------------------------------------------------------
>
>                 Key: ACCUMULO-1534
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1534
>             Project: Accumulo
>          Issue Type: Bug
>    Affects Versions: 1.4.3
>            Reporter: Mike Drob
>            Assignee: Keith Turner
>            Priority: Critical
>             Fix For: 1.5.1, 1.6.0
>
>
> I believe this issue is similar to ACCUMULO-665. We've run into a situation where a complex
iterator tree creates a large number of decompressors from the underlying CodecPool for serving
scans. Each decompressor holds on to a large buffer and the total volume ends up killing the
tserver.
> We have verified that turning off compression makes this problem go away.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message