accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-624) iterators may open lots of compressors
Date Thu, 21 Jun 2012 19:16:42 GMT


Keith Turner commented on ACCUMULO-624:

The experiment I did was the following :

 * Edit conf/generic_logger.xml and set org.apache.accumulo.core.file.rfile.bcfile to DEBUG
 * Start tservers
 * Look at shard example and run the query command (after indexing)
 * Look at tserver logs, should see messages about allocating and deallocting multiple decompressors
 * Enable block cache on shard table
 * Major compact table or wait a few minutes for file to close, file open by previous query
does not have cache set
 * run shard example query again
 * look at tservers logs, should only see one decompressor used
 * subsequent queries may not show any decompressor use because data is cached

> iterators may open lots of compressors
> --------------------------------------
>                 Key: ACCUMULO-624
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>            Reporter: Eric Newton
>            Assignee: Keith Turner
> A large iterator tree may create many instances of Compressors.  These instances are
pulled from a pool that never decreases in size.  So, if 50 simultaneous queries are run over
dozens of files, each with a complex iterator stack, there will be thousands of compressors
created.  Each of these holds a large buffer.  This can cause the server to run out of memory.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message