incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Leberknight (JIRA)" <>
Subject [jira] [Commented] (BLUR-15) Change the Block Cache to allow for a configurable slab size
Date Tue, 28 Aug 2012 03:58:07 GMT


Scott Leberknight commented on BLUR-15:

The BlockCache has three constructors, two of which simply call other constructors; one of
them does hardcode 128MB as the slab size, but the other of the two takes a slabSize parameter
and hardcodes the block size as 32MB. The third constructor accepts arguments for the totalMemory,
slabSize, and blockSize, and then computes the number of blocks per slab and the number of
slabs. The first two constructors are never called by any other code so far as I can tell.
The third constructor (the one that takes totalMemory, slabSize, and blockSize) is only called
by ThriftBlurShardServer's static createServer method.

So, the BlockCache actually does permit construction using any slab size you like. The ThriftBlurShardServer
has three inputs in the static createServer method: a hardcoded numberOfBlocksPerSlab value
of 16,384; a hardcoded blockSize of 8,192 (from the constant BlockDirectory.BLOCK_SIZE); and
the slabCount from the "blur.shard.blockcache.slab.count" configuration parameter. From those
three inputs it then computes slabSize and totalMemory, and then instantiates a BlockCache
using those two inputs plus the blockSize, and then BlockCache turns around and re-computes
numberOfBlockPerSlab and numberOfSlabs in its constructor.

So it seems a little roundabout the way all these sizes are calculated and passed into the
BlockCache via the ThriftBlurShardServer, but you can in fact instantiate a BlockCache with
whatever slabSize you want. So the question is how this should be changed, if at all. It seems
the same logic is basically in two places, but for different reasons, i.e. because different
values are calculated. It also seems like certain restrictions/invariants should be enforced,
e.g. that (slabSize % blockSize) = 0 and (totalMemory % slabSize) = 0. These probably should
be enforced in BlockCache, which should contain all this logic. Though I see why ThriftBlurShardServer
has a hand in it, which is to catch an OutOfMemoryError in case the configuration results
in that error.

You could basically have three configuration parameters that ThriftBlurShardServer would use:
1. the slab count (this already is a config parameter)
2. the block size
3. the number of blocks per slab

With these three the total memory and slab size can be controlled, and passed to BlockCache
without it being modified (other than to enforce the invariants). Does this make sense?
> Change the Block Cache to allow for a configurable slab size
> ------------------------------------------------------------
>                 Key: BLUR-15
>                 URL:
>             Project: Apache Blur
>          Issue Type: Improvement
>            Reporter: Aaron McCurry
> The current implementation of the block cache has a hard coded slab size of 128MB.  This
should be configurable so that unit testing can use the block cache at a smaller size.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message