incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rahul challapalli <challapallira...@gmail.com>
Subject Re: git commit: Adding first cut at the block cache version 2. Can be enabling in the blur-site via "blur.shard.experimental.block.cache" property.
Date Tue, 03 Sep 2013 21:24:11 GMT
Thanks Aaron. Your clarifications are really helpful. Understanding this
module also helped in enhancing my bitwise operations skill. Didn't know ~0
is -1 :)

- Rahul


On Tue, Sep 3, 2013 at 2:03 PM, Aaron McCurry <amccurry@gmail.com> wrote:

> Rahul these are all great question.  I will attempt to answer them below.
>
>
> On Tue, Sep 3, 2013 at 4:30 PM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Hi Aaron,
> >
> > Below are a few questions that I had while looking at the existing
> > BlockCache functionality. While these may be irrelevant with the new
> cache
> > implementation, some clarifications would still be helpful. Thank You.
> >
> > 1. In BlockCache we have an eviction listener registered but where are we
> > actually removing elements from the cache? If our cache is full and it
> has
> > stale data which has not been used for certain time and new data
> > comes....we are not replacing it. Shouldn't we replace unused blocks from
> > the cache?
> >
>
> Ok, this is a fairly complex implementation (hence the V2).  The map that
> is used in the LRU cache is the ConcurrentLinkedHashMap (from google code)
> and it actually calls the eviction handler.  The key in the LRU are
> basically directory + file + block number and the value is simply a pointer
> to a slab (the array of ByteBuffers) and a position in that slab where the
> 8K of cache data is stored.  Along with the key and value in the LRU there
> are bitsets that mark what blocks are not in use and which ones are.  So
> when an entry in the LRU is evicted the bitset is updated as being not in
> use, then another put entry can make use of that empty slot.
>
>
> > 2. What is the reason to duplicate a slab before returning it in the
> > BlurCache.getSlab() method?
> >
>
> Thread safety, you have to duplicate the ByteBuffer to allow multiple
> threads to read and write to the ByteBuffer at the same time.
>
>
> >
> > 3. Why is BlockLocks not extending OpenBitSet?
> >
>
> Thread safety again, OpenBitSet is not thread safe.
>
>
> >
> > 4. What is the purpose of using AtomicLongArray and wlen instead of a
> byte
> > array in BlockLocks?...With AtomicLongArray if we have to modify a single
> > bit we have update the whole word(64bits) containing the bit. (Note :
> > assuming byte level operations are atomic)
> >
>
> Length, if you use an atomic byte array you can only address 2GB of space
> if you use an atomic long array you can address 16GB (8 bytes per long).
> Probably overkill but it also made the writing of that bitset easier
> because I could just copy the logic from OpenBitSet.
>
>
> >
> > 5. What is the use of BlockCache._lockCounters? Is it just the
> > cardinality(no of bits set) of BlockLocks?
> >
>
> Speed, calculating cardinality requires a sweep of all the memory in the
> bitset.  Keeping the counter makes it faster.
>
>
> >
> > 6. Why are we cloning the BlockCacheKey (BlockCache.store()) before
> putting
> > it into the _cache(Map) for new locations. The code seems to be releasing
> > the old location which might be a result of a concurrent process adding
> the
> > location after the current process has read the cache and found that it
> > does not exist.
> >
>
> BlockCacheKey is cloned because it is likely reused in the calling logic
> (can't trust anyone) and if the object is changed by the caller and not
> cloned the hash map can get messed up.
>
>
> >
> > 7. What is the use of _threadLocalSlabs in BlockCache?
> >
>
> I did some testing and it seems like duplicate on the ByteBuffer is slower
> than just using a ThreadLocal variable and reusing the duplicated
> ByteBuffer over and over again.
>
>
> Hope this helps.  Let me know if you have more questions.
>
> Aaron
>
>
> >
> > - Rahul
> >
> >
> > On Tue, Sep 3, 2013 at 1:17 PM, Aaron McCurry <amccurry@gmail.com>
> wrote:
> >
> > > Whoops, I meant to push this into a branch (Thanks Tim).  Sorry.  I can
> > > move it into a branch tonight if everyone wants me to...  But the code
> is
> > > disabled.
> > >
> > > Aaron
> > >
> > >
> > > On Tue, Sep 3, 2013 at 4:13 PM, Aaron McCurry <amccurry@gmail.com>
> > wrote:
> > >
> > > > I have added the code for a replacement block cache (V2).  It is
> > > currently
> > > > disabled and marked experimental, the original block cache is still
> in
> > > > place and the default for  Blur.  I wanted to add the code in so that
> > > when
> > > > I test the upcoming RC candidate that I can try it out.  I apologize
> > for
> > > > adding it before posting about the work but my laptop was dieing
> (looks
> > > > like the display is dead) and I was afraid of losing the work.  I
> plan
> > on
> > > > posting an explanation at some point tonight when I can get on my
> other
> > > > computer.  Again, this is not effecting any of the current block
> cache.
> > > >
> > > > Aaron
> > > >
> > > >
> > > > On Tue, Sep 3, 2013 at 4:05 PM, rahul challapalli <
> > > > challapallirahul@gmail.com> wrote:
> > > >
> > > >> Could you elaborate on what we are trying to achieve? I was looking
> at
> > > the
> > > >> existing functionality and have a few questions which I will post
> > soon.
> > > >>
> > > >> - Rahul
> > > >>
> > > >>
> > > >> On Tue, Sep 3, 2013 at 1:02 PM, Tim Williams <williamstw@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > in master? what's the plan for this?
> > > >> >
> > > >> > --tim
> > > >> >
> > > >> > On Tue, Sep 3, 2013 at 3:34 PM,  <amccurry@apache.org> wrote:
> > > >> > > Updated Branches:
> > > >> > >   refs/heads/master bc55d21fb -> 5ea9c30e2
> > > >> > >
> > > >> > >
> > > >> > > Adding first cut at the block cache version 2.  Can be enabling
> in
> > > the
> > > >> > blur-site via "blur.shard.experimental.block.cache" property.
> > > >> > >
> > > >> > >
> > > >> > > Project:
> > http://git-wip-us.apache.org/repos/asf/incubator-blur/repo
> > > >> > > Commit:
> > > >> >
> > http://git-wip-us.apache.org/repos/asf/incubator-blur/commit/5ea9c30e
> > > >> > > Tree:
> > > >> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/tree/5ea9c30e
> > > >> > > Diff:
> > > >> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/diff/5ea9c30e
> > > >> > >
> > > >> > > Branch: refs/heads/master
> > > >> > > Commit: 5ea9c30e2172921be492bdb10ccd4a802e398a23
> > > >> > > Parents: bc55d21
> > > >> > > Author: Aaron McCurry <amccurry@gmail.com>
> > > >> > > Authored: Tue Sep 3 15:33:07 2013 -0400
> > > >> > > Committer: Aaron McCurry <amccurry@gmail.com>
> > > >> > > Committed: Tue Sep 3 15:33:07 2013 -0400
> > > >> > >
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > >  .../indexserver/DistributedIndexServer.java     |  15 +-
> > > >> > >  .../blur/thrift/ThriftBlurShardServer.java      | 117
> +++++------
> > > >> > >  .../blur/store/BlockCacheDirectoryFactory.java  |  28 +++
> > > >> > >  .../store/BlockCacheDirectoryFactoryV1.java     |  39 ++++
> > > >> > >  .../store/BlockCacheDirectoryFactoryV2.java     |  62 ++++++
> > > >> > >  .../blur/store/blockcache_v2/BaseCache.java     | 210
> > > >> > +++++++++++++++++++
> > > >> > >  .../apache/blur/store/blockcache_v2/Cache.java  | 138
> > ++++++++++++
> > > >> > >  .../store/blockcache_v2/CacheDirectory.java     | 147
> > +++++++++++++
> > > >> > >  .../store/blockcache_v2/CacheIndexInput.java    | 171
> > > +++++++++++++++
> > > >> > >  .../store/blockcache_v2/CacheIndexOutput.java   | 130
> > ++++++++++++
> > > >> > >  .../blur/store/blockcache_v2/CacheKey.java      |  89 ++++++++
> > > >> > >  .../blur/store/blockcache_v2/CacheValue.java    |  93 ++++++++
> > > >> > >  .../store/blockcache_v2/FileNameBlockSize.java  |  24 +++
> > > >> > >  .../store/blockcache_v2/FileNameFilter.java     |  24 +++
> > > >> > >  .../cachevalue/BaseCacheValue.java              |  83 ++++++++
> > > >> > >  .../cachevalue/ByteArrayCacheValue.java         |  55 +++++
> > > >> > >  .../cachevalue/UnsafeCacheValue.java            |  99 +++++++++
> > > >> > >  .../blockcache_v2/CacheIndexInputTest.java      | 206
> > > >> ++++++++++++++++++
> > > >> > >  .../blockcache_v2/CacheIndexOutputTest.java     | 101 +++++++++
> > > >> > >  .../cachevalue/ByteArrayCacheValueTest.java     |  68 ++++++
> > > >> > >  .../cachevalue/UnsafeCacheValueTest.java        |  68 ++++++
> > > >> > >  .../org/apache/blur/utils/BlurConstants.java    |   1 +
> > > >> > >  .../src/main/resources/blur-default.properties  |   3 +
> > > >> > >  docs/cluster-setup.html                         |   3 +
> > > >> > >  24 files changed, 1907 insertions(+), 67 deletions(-)
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-core/src/main/java/org/apache/blur/manager/indexserver/DistributedIndexServer.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-core/src/main/java/org/apache/blur/manager/indexserver/DistributedIndexServer.java
> > > >> >
> > > >>
> > >
> >
> b/blur-core/src/main/java/org/apache/blur/manager/indexserver/DistributedIndexServer.java
> > > >> > > index 6b7e942..ff3ce79 100644
> > > >> > > ---
> > > >> >
> > > >>
> > >
> >
> a/blur-core/src/main/java/org/apache/blur/manager/indexserver/DistributedIndexServer.java
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-core/src/main/java/org/apache/blur/manager/indexserver/DistributedIndexServer.java
> > > >> > > @@ -63,8 +63,7 @@ import
> org.apache.blur.metrics.AtomicLongGauge;
> > > >> > >  import org.apache.blur.server.IndexSearcherClosable;
> > > >> > >  import org.apache.blur.server.ShardContext;
> > > >> > >  import org.apache.blur.server.TableContext;
> > > >> > > -import org.apache.blur.store.blockcache.BlockDirectory;
> > > >> > > -import org.apache.blur.store.blockcache.Cache;
> > > >> > > +import org.apache.blur.store.BlockCacheDirectoryFactory;
> > > >> > >  import org.apache.blur.store.hdfs.BlurLockFactory;
> > > >> > >  import org.apache.blur.store.hdfs.HdfsDirectory;
> > > >> > >  import org.apache.blur.thrift.generated.ShardState;
> > > >> > > @@ -107,7 +106,7 @@ public class DistributedIndexServer extends
> > > >> > AbstractIndexServer {
> > > >> > >    private Configuration _configuration;
> > > >> > >    private String _nodeName;
> > > >> > >    private int _shardOpenerThreadCount;
> > > >> > > -  private Cache _cache;
> > > >> > > +  private BlockCacheDirectoryFactory
> _blockCacheDirectoryFactory;
> > > >> > >    private ZooKeeper _zookeeper;
> > > >> > >    private String _cluster;
> > > >> > >
> > > >> > > @@ -448,7 +447,7 @@ public class DistributedIndexServer extends
> > > >> > AbstractIndexServer {
> > > >> > >      boolean blockCacheEnabled =
> > > >> > _clusterStatus.isBlockCacheEnabled(_cluster, table);
> > > >> > >      if (blockCacheEnabled) {
> > > >> > >        Set<String> blockCacheFileTypes =
> > > >> > _clusterStatus.getBlockCacheFileTypes(_cluster, table);
> > > >> > > -      dir = new BlockDirectory(table + "_" + shard, directory,
> > > >> _cache,
> > > >> > blockCacheFileTypes);
> > > >> > > +      dir = _blockCacheDirectoryFactory.newDirectory(table +
> "_"
> > +
> > > >> > shard, directory, blockCacheFileTypes);
> > > >> > >      } else {
> > > >> > >        dir = directory;
> > > >> > >      }
> > > >> > > @@ -717,10 +716,6 @@ public class DistributedIndexServer extends
> > > >> > AbstractIndexServer {
> > > >> > >      _shardOpenerThreadCount = shardOpenerThreadCount;
> > > >> > >    }
> > > >> > >
> > > >> > > -  public void setCache(Cache cache) {
> > > >> > > -    _cache = cache;
> > > >> > > -  }
> > > >> > > -
> > > >> > >    public void setZookeeper(ZooKeeper zookeeper) {
> > > >> > >      _zookeeper = zookeeper;
> > > >> > >    }
> > > >> > > @@ -752,4 +747,8 @@ public class DistributedIndexServer extends
> > > >> > AbstractIndexServer {
> > > >> > >    public static AtomicLong getPauseWarmup() {
> > > >> > >      return _pauseWarmup;
> > > >> > >    }
> > > >> > > +
> > > >> > > +  public void
> > > >> setBlockCacheDirectoryFactory(BlockCacheDirectoryFactory
> > > >> > blockCacheDirectoryFactory) {
> > > >> > > +    _blockCacheDirectoryFactory = blockCacheDirectoryFactory;
> > > >> > > +  }
> > > >> > >  }
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-core/src/main/java/org/apache/blur/thrift/ThriftBlurShardServer.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-core/src/main/java/org/apache/blur/thrift/ThriftBlurShardServer.java
> > > >> >
> > > >>
> > >
> >
> b/blur-core/src/main/java/org/apache/blur/thrift/ThriftBlurShardServer.java
> > > >> > > index aeead2f..9d634ab 100644
> > > >> > > ---
> > > >> >
> > > >>
> > >
> >
> a/blur-core/src/main/java/org/apache/blur/thrift/ThriftBlurShardServer.java
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-core/src/main/java/org/apache/blur/thrift/ThriftBlurShardServer.java
> > > >> > > @@ -16,20 +16,25 @@ package org.apache.blur.thrift;
> > > >> > >   * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > >   * limitations under the License.
> > > >> > >   */
> > > >> > > -import static org.apache.blur.utils.BlurConstants.*;
> > > >> > > +import static org.apache.blur.utils.BlurConstants.BLUR_CLUSTER;
> > > >> > >  import static
> > > org.apache.blur.utils.BlurConstants.BLUR_CLUSTER_NAME;
> > > >> > >  import static
> > > >> > org.apache.blur.utils.BlurConstants.BLUR_CONTROLLER_BIND_PORT;
> > > >> > >  import static
> > > >> > org.apache.blur.utils.BlurConstants.BLUR_GUI_CONTROLLER_PORT;
> > > >> > >  import static
> > > >> org.apache.blur.utils.BlurConstants.BLUR_GUI_SHARD_PORT;
> > > >> > >  import static
> > > >> >
> > > >>
> > >
> >
> org.apache.blur.utils.BlurConstants.BLUR_INDEXMANAGER_SEARCH_THREAD_COUNT;
> > > >> > >  import static
> > > >> org.apache.blur.utils.BlurConstants.BLUR_MAX_CLAUSE_COUNT;
> > > >> > > +import static
> > > >> > org.apache.blur.utils.BlurConstants.BLUR_MAX_HEAP_PER_ROW_FETCH;
> > > >> > > +import static
> > > >> >
> > > >>
> > >
> >
> org.apache.blur.utils.BlurConstants.BLUR_MAX_RECORDS_PER_ROW_FETCH_REQUEST;
> > > >> > >  import static
> > > >> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_BIND_ADDRESS;
> > > >> > >  import static
> > > >> org.apache.blur.utils.BlurConstants.BLUR_SHARD_BIND_PORT;
> > > >> > >  import static
> > > >> >
> > > >>
> > >
> >
> org.apache.blur.utils.BlurConstants.BLUR_SHARD_BLOCKCACHE_DIRECT_MEMORY_ALLOCATION;
> > > >> > >  import static
> > > >> >
> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_BLOCKCACHE_SLAB_COUNT;
> > > >> > > +import static
> > > >> >
> > >
> org.apache.blur.utils.BlurConstants.BLUR_SHARD_EXPERIMENTAL_BLOCK_CACHE;
> > > >> > > +import static
> > > >> org.apache.blur.utils.BlurConstants.BLUR_SHARD_FETCHCOUNT;
> > > >> > >  import static
> > > >> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_FILTER_CACHE_CLASS;
> > > >> > >  import static
> > > >> org.apache.blur.utils.BlurConstants.BLUR_SHARD_HOSTNAME;
> > > >> > >  import static
> > > >> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_INDEX_WARMUP_CLASS;
> > > >> > > +import static
> > > >> >
> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_INDEX_WARMUP_THROTTLE;
> > > >> > >  import static
> > > >> >
> > > >>
> > >
> >
> org.apache.blur.utils.BlurConstants.BLUR_SHARD_INTERNAL_SEARCH_THREAD_COUNT;
> > > >> > >  import static
> > > >> >
> org.apache.blur.utils.BlurConstants.BLUR_SHARD_OPENER_THREAD_COUNT;
> > > >> > >  import static
> > > >> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_SAFEMODEDELAY;
> > > >> > > @@ -40,8 +45,6 @@ import static
> > > >> > org.apache.blur.utils.BlurConstants.BLUR_ZOOKEEPER_TIMEOUT;
> > > >> > >  import static
> > > >> >
> org.apache.blur.utils.BlurConstants.BLUR_ZOOKEEPER_TIMEOUT_DEFAULT;
> > > >> > >  import static org.apache.blur.utils.BlurUtil.quietClose;
> > > >> > >
> > > >> > > -import java.lang.management.ManagementFactory;
> > > >> > > -import java.util.List;
> > > >> > >  import java.util.concurrent.TimeUnit;
> > > >> > >
> > > >> > >  import org.apache.blur.BlurConfiguration;
> > > >> > > @@ -65,6 +68,9 @@ import
> > > >> > org.apache.blur.manager.writer.BlurIndexRefresher;
> > > >> > >  import org.apache.blur.metrics.JSONReporter;
> > > >> > >  import org.apache.blur.metrics.ReporterSetup;
> > > >> > >  import org.apache.blur.server.ShardServerEventHandler;
> > > >> > > +import org.apache.blur.store.BlockCacheDirectoryFactory;
> > > >> > > +import org.apache.blur.store.BlockCacheDirectoryFactoryV1;
> > > >> > > +import org.apache.blur.store.BlockCacheDirectoryFactoryV2;
> > > >> > >  import org.apache.blur.store.blockcache.BlockCache;
> > > >> > >  import org.apache.blur.store.blockcache.BlockDirectory;
> > > >> > >  import org.apache.blur.store.blockcache.BlockDirectoryCache;
> > > >> > > @@ -83,10 +89,13 @@ import org.apache.zookeeper.ZooKeeper;
> > > >> > >  import org.mortbay.jetty.servlet.ServletHolder;
> > > >> > >  import org.mortbay.jetty.webapp.WebAppContext;
> > > >> > >
> > > >> > > +import sun.misc.VM;
> > > >> > > +
> > > >> > >  public class ThriftBlurShardServer extends ThriftServer {
> > > >> > >
> > > >> > >    private static final Log LOG =
> > > >> > LogFactory.getLog(ThriftBlurShardServer.class);
> > > >> > >    private static final boolean enableJsonReporter = false;
> > > >> > > +  private static final long _64MB = 0;
> > > >> > >
> > > >> > >    public static void main(String[] args) throws Exception {
> > > >> > >      int serverIndex = getServerIndex(args);
> > > >> > > @@ -97,21 +106,13 @@ public class ThriftBlurShardServer extends
> > > >> > ThriftServer {
> > > >> > >      ReporterSetup.setupReporters(configuration);
> > > >> > >      MemoryReporter.enable();
> > > >> > >      setupJvmMetrics();
> > > >> > > -    //make this configurable
> > > >> > > +    // make this configurable
> > > >> > >      GCWatcher.init(0.75);
> > > >> > >      ThriftServer server = createServer(serverIndex,
> > configuration);
> > > >> > >      server.start();
> > > >> > >    }
> > > >> > >
> > > >> > >    public static ThriftServer createServer(int serverIndex,
> > > >> > BlurConfiguration configuration) throws Exception {
> > > >> > > -    // setup block cache
> > > >> > > -    // 134,217,728 is the slab size, therefore there are 16,384
> > > >> blocks
> > > >> > > -    // in a slab when using a block size of 8,192
> > > >> > > -    int numberOfBlocksPerSlab = 16384;
> > > >> > > -    int blockSize = BlockDirectory.BLOCK_SIZE;
> > > >> > > -    int slabCount =
> > > >> > configuration.getInt(BLUR_SHARD_BLOCKCACHE_SLAB_COUNT, -1);
> > > >> > > -    slabCount = getSlabCount(slabCount, numberOfBlocksPerSlab,
> > > >> > blockSize);
> > > >> > > -    Cache cache;
> > > >> > >      Configuration config = new Configuration();
> > > >> > >
> > > >> > >      String bindAddress =
> > > configuration.get(BLUR_SHARD_BIND_ADDRESS);
> > > >> > > @@ -132,34 +133,50 @@ public class ThriftBlurShardServer extends
> > > >> > ThriftServer {
> > > >> > >        httpServer = null;
> > > >> > >      }
> > > >> > >
> > > >> > > -    if (slabCount >= 1) {
> > > >> > > -      BlockCache blockCache;
> > > >> > > -      boolean directAllocation =
> > > >> >
> > >
> configuration.getBoolean(BLUR_SHARD_BLOCKCACHE_DIRECT_MEMORY_ALLOCATION,
> > > >> > true);
> > > >> > > -
> > > >> > > -      int slabSize = numberOfBlocksPerSlab * blockSize;
> > > >> > > -      LOG.info("Number of slabs of block cache [{0}] with
> direct
> > > >> memory
> > > >> > allocation set to [{1}]", slabCount,
> > > >> > > -          directAllocation);
> > > >> > > -      LOG.info("Block cache target memory usage, slab size of
> > [{0}]
> > > >> > will allocate [{1}] slabs and use ~[{2}] bytes",
> > > >> > > -          slabSize, slabCount, ((long) slabCount * (long)
> > > slabSize));
> > > >> > > -
> > > >> > > -      int _1024Size =
> > > >> > configuration.getInt("blur.shard.buffercache.1024", 8192);
> > > >> > > -      int _8192Size =
> > > >> > configuration.getInt("blur.shard.buffercache.8192", 8192);
> > > >> > > -      BufferStore.init(_1024Size, _8192Size);
> > > >> > > +    BlockCacheDirectoryFactory blockCacheDirectoryFactory;
> > > >> > > +    boolean experimentalBlockCache =
> > > >> > configuration.getBoolean(BLUR_SHARD_EXPERIMENTAL_BLOCK_CACHE,
> > false);
> > > >> > > +    if (!experimentalBlockCache) {
> > > >> > > +      // setup block cache
> > > >> > > +      // 134,217,728 is the slab size, therefore there are
> 16,384
> > > >> blocks
> > > >> > > +      // in a slab when using a block size of 8,192
> > > >> > > +      int numberOfBlocksPerSlab = 16384;
> > > >> > > +      int blockSize = BlockDirectory.BLOCK_SIZE;
> > > >> > > +      int slabCount =
> > > >> > configuration.getInt(BLUR_SHARD_BLOCKCACHE_SLAB_COUNT, -1);
> > > >> > > +      slabCount = getSlabCount(slabCount,
> numberOfBlocksPerSlab,
> > > >> > blockSize);
> > > >> > > +      Cache cache;
> > > >> > > +      if (slabCount >= 1) {
> > > >> > > +        BlockCache blockCache;
> > > >> > > +        boolean directAllocation =
> > > >> >
> > >
> configuration.getBoolean(BLUR_SHARD_BLOCKCACHE_DIRECT_MEMORY_ALLOCATION,
> > > >> > true);
> > > >> > > +
> > > >> > > +        int slabSize = numberOfBlocksPerSlab * blockSize;
> > > >> > > +        LOG.info("Number of slabs of block cache [{0}] with
> > direct
> > > >> > memory allocation set to [{1}]", slabCount,
> > > >> > > +            directAllocation);
> > > >> > > +        LOG.info("Block cache target memory usage, slab size of
> > > [{0}]
> > > >> > will allocate [{1}] slabs and use ~[{2}] bytes",
> > > >> > > +            slabSize, slabCount, ((long) slabCount * (long)
> > > >> slabSize));
> > > >> > > +
> > > >> > > +        int _1024Size =
> > > >> > configuration.getInt("blur.shard.buffercache.1024", 8192);
> > > >> > > +        int _8192Size =
> > > >> > configuration.getInt("blur.shard.buffercache.8192", 8192);
> > > >> > > +        BufferStore.init(_1024Size, _8192Size);
> > > >> > >
> > > >> > > -      try {
> > > >> > > -        long totalMemory = (long) slabCount * (long)
> > > >> > numberOfBlocksPerSlab * (long) blockSize;
> > > >> > > -        blockCache = new BlockCache(directAllocation,
> > totalMemory,
> > > >> > slabSize);
> > > >> > > -      } catch (OutOfMemoryError e) {
> > > >> > > -        if ("Direct buffer memory".equals(e.getMessage())) {
> > > >> > > -          System.err
> > > >> > > -              .println("The max direct memory is too low.
>  Either
> > > >> > increase by setting (-XX:MaxDirectMemorySize=<size>g
> > > >> -XX:+UseLargePages) or
> > > >> > disable direct allocation by
> > > >> > (blur.shard.blockcache.direct.memory.allocation=false) in
> > > >> > blur-site.properties");
> > > >> > > -          System.exit(1);
> > > >> > > +        try {
> > > >> > > +          long totalMemory = (long) slabCount * (long)
> > > >> > numberOfBlocksPerSlab * (long) blockSize;
> > > >> > > +          blockCache = new BlockCache(directAllocation,
> > > totalMemory,
> > > >> > slabSize);
> > > >> > > +        } catch (OutOfMemoryError e) {
> > > >> > > +          if ("Direct buffer memory".equals(e.getMessage())) {
> > > >> > > +            System.err
> > > >> > > +                .println("The max direct memory is too low.
> >  Either
> > > >> > increase by setting (-XX:MaxDirectMemorySize=<size>g
> > > >> -XX:+UseLargePages) or
> > > >> > disable direct allocation by
> > > >> > (blur.shard.blockcache.direct.memory.allocation=false) in
> > > >> > blur-site.properties");
> > > >> > > +            System.exit(1);
> > > >> > > +          }
> > > >> > > +          throw e;
> > > >> > >          }
> > > >> > > -        throw e;
> > > >> > > +        cache = new BlockDirectoryCache(blockCache);
> > > >> > > +      } else {
> > > >> > > +        cache = BlockDirectory.NO_CACHE;
> > > >> > >        }
> > > >> > > -      cache = new BlockDirectoryCache(blockCache);
> > > >> > > +      blockCacheDirectoryFactory = new
> > > >> > BlockCacheDirectoryFactoryV1(cache);
> > > >> > >      } else {
> > > >> > > -      cache = BlockDirectory.NO_CACHE;
> > > >> > > +      long totalNumberOfBytes = VM.maxDirectMemory() - _64MB;
> > > >> > > +      blockCacheDirectoryFactory = new
> > > >> > BlockCacheDirectoryFactoryV2(configuration, totalNumberOfBytes);
> > > >> > >      }
> > > >> > >
> > > >> > >      LOG.info("Shard Server using index [{0}] bind address
> [{1}]",
> > > >> > serverIndex, bindAddress + ":" + bindPort);
> > > >> > > @@ -185,7 +202,7 @@ public class ThriftBlurShardServer extends
> > > >> > ThriftServer {
> > > >> > >      BlurIndexWarmup indexWarmup =
> getIndexWarmup(configuration);
> > > >> > >
> > > >> > >      final DistributedIndexServer indexServer = new
> > > >> > DistributedIndexServer();
> > > >> > > -    indexServer.setCache(cache);
> > > >> > > +
> > > >> >
> > >  indexServer.setBlockCacheDirectoryFactory(blockCacheDirectoryFactory);
> > > >> > >      indexServer.setClusterStatus(clusterStatus);
> > > >> > >
> >  indexServer.setClusterName(configuration.get(BLUR_CLUSTER_NAME,
> > > >> > BLUR_CLUSTER));
> > > >> > >      indexServer.setConfiguration(config);
> > > >> > > @@ -260,33 +277,15 @@ public class ThriftBlurShardServer extends
> > > >> > ThriftServer {
> > > >> > >    private static int getSlabCount(int slabCount, int
> > > >> > numberOfBlocksPerSlab, int blockSize) {
> > > >> > >      if (slabCount < 0) {
> > > >> > >        long slabSize = numberOfBlocksPerSlab * blockSize;
> > > >> > > -      List<String> inputArguments =
> > > >> > ManagementFactory.getRuntimeMXBean().getInputArguments();
> > > >> > > -      for (String arg : inputArguments) {
> > > >> > > -        if (arg.startsWith("-XX:MaxDirectMemorySize")) {
> > > >> > > -          long maxDirectMemorySize =
> getMaxDirectMemorySize(arg);
> > > >> > > -          maxDirectMemorySize -= 64 * 1024 * 1024;
> > > >> > > -          return (int) (maxDirectMemorySize / slabSize);
> > > >> > > -        }
> > > >> > > +      long maxDirectMemorySize = VM.maxDirectMemory() - _64MB;
> > > >> > > +      if (maxDirectMemorySize < slabSize) {
> > > >> > > +        throw new RuntimeException("Auto slab setup cannot
> > happen,
> > > >> JVM
> > > >> > option -XX:MaxDirectMemorySize not set.");
> > > >> > >        }
> > > >> > > -      throw new RuntimeException("Auto slab setup cannot
> happen,
> > > JVM
> > > >> > option -XX:MaxDirectMemorySize not set.");
> > > >> > > +      return (int) (maxDirectMemorySize / slabSize);
> > > >> > >      }
> > > >> > >      return slabCount;
> > > >> > >    }
> > > >> > >
> > > >> > > -  private static long getMaxDirectMemorySize(String arg) {
> > > >> > > -    int index = arg.lastIndexOf('=');
> > > >> > > -    return parseNumber(arg.substring(index +
> > > >> 1).toLowerCase().replace("
> > > >> > ", ""));
> > > >> > > -  }
> > > >> > > -
> > > >> > > -  private static long parseNumber(String number) {
> > > >> > > -    if (number.endsWith("m")) {
> > > >> > > -      return Long.parseLong(number.substring(0,
> number.length() -
> > > >> 1)) *
> > > >> > 1024 * 1024;
> > > >> > > -    } else if (number.endsWith("g")) {
> > > >> > > -      return Long.parseLong(number.substring(0,
> number.length() -
> > > >> 1)) *
> > > >> > 1024 * 1024 * 1024;
> > > >> > > -    }
> > > >> > > -    throw new RuntimeException("Cannot parse [" + number +
> "]");
> > > >> > > -  }
> > > >> > > -
> > > >> > >    private static BlurFilterCache
> getFilterCache(BlurConfiguration
> > > >> > configuration) {
> > > >> > >      String _blurFilterCacheClass =
> > > >> > configuration.get(BLUR_SHARD_FILTER_CACHE_CLASS);
> > > >> > >      if (_blurFilterCacheClass != null) {
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactory.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactory.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactory.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..a8d31ab
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactory.java
> > > >> > > @@ -0,0 +1,28 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> or
> > > more
> > > >> > > + * contributor license agreements.  See the NOTICE file
> > distributed
> > > >> with
> > > >> > > + * this work for additional information regarding copyright
> > > >> ownership.
> > > >> > > + * The ASF licenses this file to You under the Apache License,
> > > >> Version
> > > >> > 2.0
> > > >> > > + * (the "License"); you may not use this file except in
> > compliance
> > > >> with
> > > >> > > + * the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store;
> > > >> > > +
> > > >> > > +import java.io.IOException;
> > > >> > > +import java.util.Set;
> > > >> > > +
> > > >> > > +import org.apache.lucene.store.Directory;
> > > >> > > +
> > > >> > > +public interface BlockCacheDirectoryFactory {
> > > >> > > +
> > > >> > > +  Directory newDirectory(String name, Directory directory,
> > > >> Set<String>
> > > >> > blockCacheFileTypes) throws IOException;
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV1.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV1.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV1.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..182b494
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV1.java
> > > >> > > @@ -0,0 +1,39 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> or
> > > more
> > > >> > > + * contributor license agreements.  See the NOTICE file
> > distributed
> > > >> with
> > > >> > > + * this work for additional information regarding copyright
> > > >> ownership.
> > > >> > > + * The ASF licenses this file to You under the Apache License,
> > > >> Version
> > > >> > 2.0
> > > >> > > + * (the "License"); you may not use this file except in
> > compliance
> > > >> with
> > > >> > > + * the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store;
> > > >> > > +
> > > >> > > +import java.io.IOException;
> > > >> > > +import java.util.Set;
> > > >> > > +
> > > >> > > +import org.apache.blur.store.blockcache.BlockDirectory;
> > > >> > > +import org.apache.blur.store.blockcache.Cache;
> > > >> > > +import org.apache.lucene.store.Directory;
> > > >> > > +
> > > >> > > +public class BlockCacheDirectoryFactoryV1 implements
> > > >> > BlockCacheDirectoryFactory {
> > > >> > > +
> > > >> > > +  private final Cache _cache;
> > > >> > > +
> > > >> > > +  public BlockCacheDirectoryFactoryV1(Cache cache) {
> > > >> > > +    _cache = cache;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public Directory newDirectory(String name, Directory
> directory,
> > > >> > Set<String> blockCacheFileTypes) throws IOException {
> > > >> > > +    return new BlockDirectory(name, directory, _cache,
> > > >> > blockCacheFileTypes);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV2.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV2.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV2.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..0f20900
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV2.java
> > > >> > > @@ -0,0 +1,62 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> or
> > > more
> > > >> > > + * contributor license agreements.  See the NOTICE file
> > distributed
> > > >> with
> > > >> > > + * this work for additional information regarding copyright
> > > >> ownership.
> > > >> > > + * The ASF licenses this file to You under the Apache License,
> > > >> Version
> > > >> > 2.0
> > > >> > > + * (the "License"); you may not use this file except in
> > compliance
> > > >> with
> > > >> > > + * the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store;
> > > >> > > +
> > > >> > > +import java.io.IOException;
> > > >> > > +import java.util.Set;
> > > >> > > +
> > > >> > > +import org.apache.blur.BlurConfiguration;
> > > >> > > +import org.apache.blur.store.blockcache_v2.BaseCache;
> > > >> > > +import org.apache.blur.store.blockcache_v2.BaseCache.STORE;
> > > >> > > +import org.apache.blur.store.blockcache_v2.Cache;
> > > >> > > +import org.apache.blur.store.blockcache_v2.CacheDirectory;
> > > >> > > +import org.apache.blur.store.blockcache_v2.FileNameBlockSize;
> > > >> > > +import org.apache.blur.store.blockcache_v2.FileNameFilter;
> > > >> > > +import org.apache.lucene.store.Directory;
> > > >> > > +
> > > >> > > +public class BlockCacheDirectoryFactoryV2 implements
> > > >> > BlockCacheDirectoryFactory {
> > > >> > > +
> > > >> > > +  private Cache _cache;
> > > >> > > +
> > > >> > > +  public BlockCacheDirectoryFactoryV2(BlurConfiguration
> > > >> configuration,
> > > >> > long totalNumberOfBytes) {
> > > >> > > +    int fileBufferSize = 8192;
> > > >> > > +    FileNameBlockSize fileNameBlockSize = new
> > FileNameBlockSize() {
> > > >> > > +      @Override
> > > >> > > +      public int getBlockSize(String directoryName, String
> > > fileName)
> > > >> {
> > > >> > > +        return 8192;
> > > >> > > +      }
> > > >> > > +    };
> > > >> > > +    FileNameFilter readFilter = new FileNameFilter() {
> > > >> > > +      @Override
> > > >> > > +      public boolean accept(String directoryName, String
> > fileName)
> > > {
> > > >> > > +        if (fileName.endsWith(".fdt") ||
> > > fileName.endsWith(".fdx")) {
> > > >> > > +          return true;
> > > >> > > +        }
> > > >> > > +        return true;
> > > >> > > +      }
> > > >> > > +    };
> > > >> > > +    FileNameFilter writeFilter = readFilter;
> > > >> > > +    _cache = new BaseCache(totalNumberOfBytes, fileBufferSize,
> > > >> > fileNameBlockSize, readFilter, writeFilter,
> > > >> > > +        STORE.OFF_HEAP);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public Directory newDirectory(String name, Directory
> directory,
> > > >> > Set<String> blockCacheFileTypes) throws IOException {
> > > >> > > +    return new CacheDirectory(name, directory, _cache);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/BaseCache.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/BaseCache.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/BaseCache.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..1ec2796
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/BaseCache.java
> > > >> > > @@ -0,0 +1,210 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2;
> > > >> > > +
> > > >> > > +import java.io.IOException;
> > > >> > > +import java.util.Iterator;
> > > >> > > +import java.util.Map;
> > > >> > > +import java.util.Map.Entry;
> > > >> > > +import java.util.Set;
> > > >> > > +import java.util.concurrent.ConcurrentHashMap;
> > > >> > > +import java.util.concurrent.atomic.AtomicLong;
> > > >> > > +
> > > >> > > +import
> > > >> >
> org.apache.blur.store.blockcache_v2.cachevalue.ByteArrayCacheValue;
> > > >> > > +import
> > > >> org.apache.blur.store.blockcache_v2.cachevalue.UnsafeCacheValue;
> > > >> > > +import org.apache.lucene.store.IOContext;
> > > >> > > +
> > > >> > > +import
> > > >> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap;
> > > >> > > +import com.googlecode.concurrentlinkedhashmap.EvictionListener;
> > > >> > > +import com.googlecode.concurrentlinkedhashmap.Weigher;
> > > >> > > +
> > > >> > > +public class BaseCache implements Cache {
> > > >> > > +  public enum STORE {
> > > >> > > +    ON_HEAP, OFF_HEAP
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  class BaseCacheEvictionListener implements
> > > >> EvictionListener<CacheKey,
> > > >> > CacheValue> {
> > > >> > > +    @Override
> > > >> > > +    public void onEviction(CacheKey key, CacheValue value) {
> > > >> > > +      addToReleaseQueue(value);
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  class BaseCacheWeigher implements Weigher<CacheValue> {
> > > >> > > +    @Override
> > > >> > > +    public int weightOf(CacheValue value) {
> > > >> > > +      return value.size();
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private final ConcurrentLinkedHashMap<CacheKey, CacheValue>
> > > >> _cacheMap;
> > > >> > > +  private final int _fileBufferSize;
> > > >> > > +  private final FileNameFilter _readFilter;
> > > >> > > +  private final FileNameFilter _writeFilter;
> > > >> > > +  private final STORE _store;
> > > >> > > +  private final FileNameBlockSize _fileNameBlockSize;
> > > >> > > +  private final Map<FileIdKey, Long> _fileNameToId = new
> > > >> > ConcurrentHashMap<FileIdKey, Long>();
> > > >> > > +  private final AtomicLong _fileId = new AtomicLong();
> > > >> > > +
> > > >> > > +  public BaseCache(long totalNumberOfBytes, int fileBufferSize,
> > > >> > FileNameBlockSize fileNameBlockSize,
> > > >> > > +      FileNameFilter readFilter, FileNameFilter writeFilter,
> > STORE
> > > >> > store) {
> > > >> > > +    _cacheMap = new ConcurrentLinkedHashMap.Builder<CacheKey,
> > > >> > CacheValue>().weigher(new BaseCacheWeigher())
> > > >> > > +
>  .maximumWeightedCapacity(totalNumberOfBytes).listener(new
> > > >> > BaseCacheEvictionListener()).build();
> > > >> > > +    _fileBufferSize = fileBufferSize;
> > > >> > > +    _readFilter = readFilter;
> > > >> > > +    _writeFilter = writeFilter;
> > > >> > > +    _store = store;
> > > >> > > +    _fileNameBlockSize = fileNameBlockSize;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private void addToReleaseQueue(CacheValue value) {
> > > >> > > +    if (value != null) {
> > > >> > > +      if (value.refCount() == 0) {
> > > >> > > +        value.release();
> > > >> > > +        return;
> > > >> > > +      }
> > > >> > > +      // @TODO create clean up queue
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public CacheValue newInstance(CacheDirectory directory,
> String
> > > >> > fileName) {
> > > >> > > +    switch (_store) {
> > > >> > > +    case ON_HEAP:
> > > >> > > +      return new
> ByteArrayCacheValue(getCacheBlockSize(directory,
> > > >> > fileName));
> > > >> > > +    case OFF_HEAP:
> > > >> > > +      return new UnsafeCacheValue(getCacheBlockSize(directory,
> > > >> > fileName));
> > > >> > > +    default:
> > > >> > > +      throw new RuntimeException("Unknown type [" + _store +
> > "]");
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public long getFileId(CacheDirectory directory, String
> > fileName)
> > > >> > throws IOException {
> > > >> > > +    FileIdKey cachedFileName = getCacheFileName(directory,
> > > fileName);
> > > >> > > +    Long id = _fileNameToId.get(cachedFileName);
> > > >> > > +    if (id != null) {
> > > >> > > +      return id;
> > > >> > > +    }
> > > >> > > +    long newId = _fileId.incrementAndGet();
> > > >> > > +    _fileNameToId.put(cachedFileName, id);
> > > >> > > +    return newId;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void removeFile(CacheDirectory directory, String
> > fileName)
> > > >> > throws IOException {
> > > >> > > +    FileIdKey cachedFileName = getCacheFileName(directory,
> > > fileName);
> > > >> > > +    _fileNameToId.remove(cachedFileName);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private FileIdKey getCacheFileName(CacheDirectory directory,
> > > String
> > > >> > fileName) throws IOException {
> > > >> > > +    long fileModified = directory.getFileModified(fileName);
> > > >> > > +    return new FileIdKey(directory.getDirectoryName(),
> fileName,
> > > >> > fileModified);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public int getCacheBlockSize(CacheDirectory directory, String
> > > >> > fileName) {
> > > >> > > +    return
> > > >> > _fileNameBlockSize.getBlockSize(directory.getDirectoryName(),
> > > fileName);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public int getFileBufferSize(CacheDirectory directory, String
> > > >> > fileName) {
> > > >> > > +    return _fileBufferSize;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public boolean cacheFileForReading(CacheDirectory directory,
> > > String
> > > >> > fileName, IOContext context) {
> > > >> > > +    return _readFilter.accept(directory.getDirectoryName(),
> > > >> fileName);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public boolean cacheFileForWriting(CacheDirectory directory,
> > > String
> > > >> > fileName, IOContext context) {
> > > >> > > +    return _writeFilter.accept(directory.getDirectoryName(),
> > > >> fileName);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public CacheValue get(CacheKey key) {
> > > >> > > +    return _cacheMap.get(key);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void put(CacheKey key, CacheValue value) {
> > > >> > > +    addToReleaseQueue(_cacheMap.put(key, value));
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void releaseDirectory(String directoryName) {
> > > >> > > +    Set<Entry<FileIdKey, Long>> entrySet =
> > > _fileNameToId.entrySet();
> > > >> > > +    Iterator<Entry<FileIdKey, Long>> iterator =
> > > entrySet.iterator();
> > > >> > > +    while (iterator.hasNext()) {
> > > >> > > +      Entry<FileIdKey, Long> entry = iterator.next();
> > > >> > > +      FileIdKey fileIdKey = entry.getKey();
> > > >> > > +      if (fileIdKey._directoryName.equals(directoryName)) {
> > > >> > > +        iterator.remove();
> > > >> > > +      }
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  static class FileIdKey {
> > > >> > > +    final String _directoryName;
> > > >> > > +    final String _fileName;
> > > >> > > +    final long _lastModified;
> > > >> > > +
> > > >> > > +    FileIdKey(String directoryName, String fileName, long
> > > >> lastModified)
> > > >> > {
> > > >> > > +      _directoryName = directoryName;
> > > >> > > +      _fileName = fileName;
> > > >> > > +      _lastModified = lastModified;
> > > >> > > +    }
> > > >> > > +
> > > >> > > +    @Override
> > > >> > > +    public int hashCode() {
> > > >> > > +      final int prime = 31;
> > > >> > > +      int result = 1;
> > > >> > > +      result = prime * result + ((_directoryName == null) ? 0 :
> > > >> > _directoryName.hashCode());
> > > >> > > +      result = prime * result + ((_fileName == null) ? 0 :
> > > >> > _fileName.hashCode());
> > > >> > > +      result = prime * result + (int) (_lastModified ^
> > > (_lastModified
> > > >> > >>> 32));
> > > >> > > +      return result;
> > > >> > > +    }
> > > >> > > +
> > > >> > > +    @Override
> > > >> > > +    public boolean equals(Object obj) {
> > > >> > > +      if (this == obj)
> > > >> > > +        return true;
> > > >> > > +      if (obj == null)
> > > >> > > +        return false;
> > > >> > > +      if (getClass() != obj.getClass())
> > > >> > > +        return false;
> > > >> > > +      FileIdKey other = (FileIdKey) obj;
> > > >> > > +      if (_directoryName == null) {
> > > >> > > +        if (other._directoryName != null)
> > > >> > > +          return false;
> > > >> > > +      } else if (!_directoryName.equals(other._directoryName))
> > > >> > > +        return false;
> > > >> > > +      if (_fileName == null) {
> > > >> > > +        if (other._fileName != null)
> > > >> > > +          return false;
> > > >> > > +      } else if (!_fileName.equals(other._fileName))
> > > >> > > +        return false;
> > > >> > > +      if (_lastModified != other._lastModified)
> > > >> > > +        return false;
> > > >> > > +      return true;
> > > >> > > +    }
> > > >> > > +
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/Cache.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/Cache.java
> > > >> >
> > > >>
> > >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/Cache.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..66e60f7
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/Cache.java
> > > >> > > @@ -0,0 +1,138 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2;
> > > >> > > +
> > > >> > > +import java.io.IOException;
> > > >> > > +
> > > >> > > +import org.apache.lucene.store.IOContext;
> > > >> > > +
> > > >> > > +public interface Cache {
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Creates a new instance of CacheValue, the cache capacity
> > > should
> > > >> be
> > > >> > used for
> > > >> > > +   * the given file.
> > > >> > > +   *
> > > >> > > +   * @param directory
> > > >> > > +   *          the directory.
> > > >> > > +   * @param fileName
> > > >> > > +   *          the file name.
> > > >> > > +   * @return the new CacheValue instance.
> > > >> > > +   */
> > > >> > > +  CacheValue newInstance(CacheDirectory directory, String
> > > fileName);
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Gets unique id for the given file. This is assumed to be
> > > unique
> > > >> > even if the
> > > >> > > +   * file is deleted and recreated.
> > > >> > > +   *
> > > >> > > +   * @param directory
> > > >> > > +   *          the directory.
> > > >> > > +   * @param fileName
> > > >> > > +   *          the file name.
> > > >> > > +   * @return the file id.
> > > >> > > +   * @throws IOException
> > > >> > > +   */
> > > >> > > +  long getFileId(CacheDirectory directory, String fileName)
> > throws
> > > >> > IOException;
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Get capacity of each cache entry for the given file.
> > > >> > > +   *
> > > >> > > +   * @param directory
> > > >> > > +   *          the directory.
> > > >> > > +   * @param fileName
> > > >> > > +   *          the file name.
> > > >> > > +   * @return the capacity.
> > > >> > > +   */
> > > >> > > +  int getCacheBlockSize(CacheDirectory directory, String
> > fileName);
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Gets buffer size of the buffer used while interacting with
> > the
> > > >> > underlying
> > > >> > > +   * directory.
> > > >> > > +   *
> > > >> > > +   * @param directory
> > > >> > > +   *          the directory.
> > > >> > > +   * @param fileName
> > > >> > > +   *          the file name.
> > > >> > > +   * @return the buffer size.
> > > >> > > +   */
> > > >> > > +  int getFileBufferSize(CacheDirectory directory, String
> > fileName);
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Checks whether file should be cached or not during
> reading.
> > > >> > > +   *
> > > >> > > +   * @param directory
> > > >> > > +   *          the directory.
> > > >> > > +   * @param fileName
> > > >> > > +   *          the file name.
> > > >> > > +   * @param context
> > > >> > > +   *          the IOContext from Lucene.
> > > >> > > +   * @return boolean.
> > > >> > > +   */
> > > >> > > +  boolean cacheFileForReading(CacheDirectory directory, String
> > > >> > fileName, IOContext context);
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Checks whether file should be cached or not during
> writing.
> > > >> > > +   *
> > > >> > > +   * @param directory
> > > >> > > +   *          the directory.
> > > >> > > +   * @param fileName
> > > >> > > +   *          the file name.
> > > >> > > +   * @param context
> > > >> > > +   *          the IOContext from Lucene.
> > > >> > > +   * @return boolean.
> > > >> > > +   */
> > > >> > > +  boolean cacheFileForWriting(CacheDirectory directory, String
> > > >> > fileName, IOContext context);
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Gets the cache value for the given key. Null if missing.
> > > >> > > +   *
> > > >> > > +   * @param key
> > > >> > > +   *          the key.
> > > >> > > +   * @return the cache value or null.
> > > >> > > +   */
> > > >> > > +  CacheValue get(CacheKey key);
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Puts the cache entry into the cache.
> > > >> > > +   *
> > > >> > > +   * @param key
> > > >> > > +   *          the key.
> > > >> > > +   * @param value
> > > >> > > +   *          the value.
> > > >> > > +   */
> > > >> > > +  void put(CacheKey key, CacheValue value);
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Removes the file from the cache.
> > > >> > > +   *
> > > >> > > +   * @param directory
> > > >> > > +   *          the directory.
> > > >> > > +   * @param fileName
> > > >> > > +   *          the file name.
> > > >> > > +   * @throws IOException
> > > >> > > +   */
> > > >> > > +  void removeFile(CacheDirectory directory, String fileName)
> > throws
> > > >> > IOException;
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * This is called when the CacheDirectory is finalized.
> > > >> > > +   *
> > > >> > > +   * @param directoryName
> > > >> > > +   *          the directory name.
> > > >> > > +   */
> > > >> > > +  void releaseDirectory(String directoryName);
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheDirectory.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheDirectory.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheDirectory.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..73cc147
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheDirectory.java
> > > >> > > @@ -0,0 +1,147 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2;
> > > >> > > +
> > > >> > > +import java.io.IOException;
> > > >> > > +import java.util.Collection;
> > > >> > > +
> > > >> > > +import org.apache.blur.store.blockcache.LastModified;
> > > >> > > +import org.apache.blur.store.hdfs.DirectoryDecorator;
> > > >> > > +import org.apache.lucene.store.Directory;
> > > >> > > +import org.apache.lucene.store.IOContext;
> > > >> > > +import org.apache.lucene.store.IndexInput;
> > > >> > > +import org.apache.lucene.store.IndexOutput;
> > > >> > > +import org.apache.lucene.store.Lock;
> > > >> > > +import org.apache.lucene.store.LockFactory;
> > > >> > > +
> > > >> > > +public class CacheDirectory extends Directory implements
> > > >> > DirectoryDecorator, LastModified {
> > > >> > > +
> > > >> > > +  private final Directory _internal;
> > > >> > > +  private final String _directoryName;
> > > >> > > +  private final Cache _cache;
> > > >> > > +
> > > >> > > +  public CacheDirectory(String directoryName, Directory
> > directory,
> > > >> > Cache cache) {
> > > >> > > +    if (!(directory instanceof LastModified)) {
> > > >> > > +      throw new RuntimeException("Directory [" + directory + "]
> > > does
> > > >> > not implement '" + LastModified.class.toString()
> > > >> > > +          + "'");
> > > >> > > +    }
> > > >> > > +    _directoryName = notNull(directoryName);
> > > >> > > +    _internal = notNull(directory);
> > > >> > > +    _cache = notNull(cache);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  protected void finalize() throws Throwable {
> > > >> > > +    _cache.releaseDirectory(getDirectoryName());
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public IndexInput openInput(String name, IOContext context)
> > > throws
> > > >> > IOException {
> > > >> > > +    IndexInput indexInput = _internal.openInput(name, context);
> > > >> > > +    if (_cache.cacheFileForReading(this, name, context)) {
> > > >> > > +      return new CacheIndexInput(this, name, indexInput,
> _cache);
> > > >> > > +    }
> > > >> > > +    return indexInput;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public IndexOutput createOutput(String name, IOContext
> context)
> > > >> > throws IOException {
> > > >> > > +    IndexOutput indexOutput = _internal.createOutput(name,
> > > context);
> > > >> > > +    if (_cache.cacheFileForWriting(this, name, context)) {
> > > >> > > +      return new CacheIndexOutput(this, name, indexOutput,
> > _cache);
> > > >> > > +    }
> > > >> > > +    return indexOutput;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public void deleteFile(String name) throws IOException {
> > > >> > > +    _cache.removeFile(this, name);
> > > >> > > +    _internal.deleteFile(name);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public String[] listAll() throws IOException {
> > > >> > > +    return _internal.listAll();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public boolean fileExists(String name) throws IOException {
> > > >> > > +    return _internal.fileExists(name);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public long fileLength(String name) throws IOException {
> > > >> > > +    return _internal.fileLength(name);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public void sync(Collection<String> names) throws
> IOException {
> > > >> > > +    _internal.sync(names);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public Lock makeLock(String name) {
> > > >> > > +    return _internal.makeLock(name);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public void clearLock(String name) throws IOException {
> > > >> > > +    _internal.clearLock(name);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public void close() throws IOException {
> > > >> > > +    _internal.close();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public void setLockFactory(LockFactory lockFactory) throws
> > > >> > IOException {
> > > >> > > +    _internal.setLockFactory(lockFactory);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public LockFactory getLockFactory() {
> > > >> > > +    return _internal.getLockFactory();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public String getLockID() {
> > > >> > > +    return _internal.getLockID();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public String toString() {
> > > >> > > +    return _internal.toString();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public void copy(Directory to, String src, String dest,
> > IOContext
> > > >> > context) throws IOException {
> > > >> > > +    _internal.copy(to, src, dest, context);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public IndexInputSlicer createSlicer(String name, IOContext
> > > >> context)
> > > >> > throws IOException {
> > > >> > > +    return _internal.createSlicer(name, context);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public String getDirectoryName() {
> > > >> > > +    return _directoryName;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public long getFileModified(String name) throws IOException {
> > > >> > > +    return ((LastModified) _internal).getFileModified(name);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public Directory getOriginalDirectory() {
> > > >> > > +    return _internal;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private static <T> T notNull(T t) {
> > > >> > > +    if (t == null) {
> > > >> > > +      throw new IllegalArgumentException("Cannot be null");
> > > >> > > +    }
> > > >> > > +    return t;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > > \ No newline at end of file
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexInput.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexInput.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexInput.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..5279292
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexInput.java
> > > >> > > @@ -0,0 +1,171 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2;
> > > >> > > +
> > > >> > > +import java.io.IOException;
> > > >> > > +
> > > >> > > +import org.apache.blur.store.buffer.BufferStore;
> > > >> > > +import org.apache.lucene.store.IndexInput;
> > > >> > > +
> > > >> > > +public class CacheIndexInput extends IndexInput {
> > > >> > > +
> > > >> > > +  private final long _fileLength;
> > > >> > > +  private final long _fileId;
> > > >> > > +  private final int _cacheBlockSize;
> > > >> > > +  private final int _bufferSize;
> > > >> > > +  private final CacheDirectory _directory;
> > > >> > > +  private final String _fileName;
> > > >> > > +  private final IndexInput _indexInput;
> > > >> > > +  private final Cache _cache;
> > > >> > > +  private final CacheKey _key = new CacheKey();
> > > >> > > +
> > > >> > > +  private CacheValue _cacheValue;
> > > >> > > +  private long _position;
> > > >> > > +  private int _blockPosition;
> > > >> > > +
> > > >> > > +  public CacheIndexInput(CacheDirectory directory, String
> > fileName,
> > > >> > IndexInput indexInput, Cache cache) throws IOException {
> > > >> > > +    super(fileName);
> > > >> > > +    _directory = directory;
> > > >> > > +    _fileName = fileName;
> > > >> > > +    _indexInput = indexInput;
> > > >> > > +    _fileLength = indexInput.length();
> > > >> > > +    _cache = cache;
> > > >> > > +
> > > >> > > +    _fileId = _cache.getFileId(_directory, _fileName);
> > > >> > > +    _cacheBlockSize = _cache.getCacheBlockSize(_directory,
> > > >> _fileName);
> > > >> > > +    _bufferSize = _cache.getFileBufferSize(_directory,
> > _fileName);
> > > >> > > +    _key.setFileId(_fileId);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public byte readByte() throws IOException {
> > > >> > > +    tryToFill();
> > > >> > > +    byte b = _cacheValue.read(_blockPosition);
> > > >> > > +    _position++;
> > > >> > > +    _blockPosition++;
> > > >> > > +    return b;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void readBytes(byte[] b, int offset, int len) throws
> > > >> > IOException {
> > > >> > > +    while (len > 0) {
> > > >> > > +      tryToFill();
> > > >> > > +      int remaining = remaining();
> > > >> > > +      int length = Math.min(len, remaining);
> > > >> > > +      _cacheValue.read(_blockPosition, b, offset, length);
> > > >> > > +      offset += length;
> > > >> > > +      len -= length;
> > > >> > > +      _position += length;
> > > >> > > +      _blockPosition += length;
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private int remaining() {
> > > >> > > +    return _cacheValue.length() - _blockPosition;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private void tryToFill() throws IOException {
> > > >> > > +    if (_cacheValue == null) {
> > > >> > > +      fill();
> > > >> > > +    } else if (remaining() == 0) {
> > > >> > > +      releaseCache();
> > > >> > > +      fill();
> > > >> > > +    } else {
> > > >> > > +      return;
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private void releaseCache() {
> > > >> > > +    if (_cacheValue != null) {
> > > >> > > +      _cacheValue.decRef();
> > > >> > > +      _cacheValue = null;
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private void fill() throws IOException {
> > > >> > > +    _key.setBlockId(getBlockId());
> > > >> > > +    _cacheValue = _cache.get(_key);
> > > >> > > +    if (_cacheValue == null) {
> > > >> > > +      _cacheValue = _cache.newInstance(_directory, _fileName);
> > > >> > > +      long filePosition = getFilePosition();
> > > >> > > +      _indexInput.seek(filePosition);
> > > >> > > +      byte[] buffer = BufferStore.takeBuffer(_bufferSize);
> > > >> > > +      int len = (int) Math.min(_cacheBlockSize, _fileLength -
> > > >> > filePosition);
> > > >> > > +      int cachePosition = 0;
> > > >> > > +      while (len > 0) {
> > > >> > > +        int length = Math.min(_bufferSize, len);
> > > >> > > +        _indexInput.readBytes(buffer, 0, length);
> > > >> > > +        _cacheValue.write(cachePosition, buffer, 0, length);
> > > >> > > +        len -= length;
> > > >> > > +        cachePosition += length;
> > > >> > > +      }
> > > >> > > +    }
> > > >> > > +    _cache.put(_key.clone(), _cacheValue);
> > > >> > > +    _cacheValue.incRef();
> > > >> > > +    _blockPosition = getBlockPosition();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private int getBlockPosition() {
> > > >> > > +    return (int) (_position % _cacheBlockSize);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private long getFilePosition() {
> > > >> > > +    // make this a mask...?
> > > >> > > +    return getBlockId() * _cacheBlockSize;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private long getBlockId(long pos) {
> > > >> > > +    return pos / _cacheBlockSize;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private long getBlockId() {
> > > >> > > +    return _position / _cacheBlockSize;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void close() throws IOException {
> > > >> > > +    _indexInput.close();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public long getFilePointer() {
> > > >> > > +    return _position;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void seek(long pos) throws IOException {
> > > >> > > +    if (_position >= _fileLength) {
> > > >> > > +      throw new IOException("Can not seek past end of file [" +
> > > pos +
> > > >> > "] filelength [" + _fileLength + "]");
> > > >> > > +    }
> > > >> > > +    long oldBlockId = getBlockId();
> > > >> > > +    _position = pos;
> > > >> > > +    long newBlockId = getBlockId(pos);
> > > >> > > +    if (newBlockId == oldBlockId) {
> > > >> > > +      // need to set new block position
> > > >> > > +      _blockPosition = getBlockPosition();
> > > >> > > +    } else {
> > > >> > > +      releaseCache();
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public long length() {
> > > >> > > +    return _fileLength;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexOutput.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexOutput.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexOutput.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..dbb346c
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexOutput.java
> > > >> > > @@ -0,0 +1,130 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2;
> > > >> > > +
> > > >> > > +import java.io.IOException;
> > > >> > > +
> > > >> > > +import org.apache.blur.store.buffer.BufferStore;
> > > >> > > +import org.apache.lucene.store.IndexOutput;
> > > >> > > +
> > > >> > > +public class CacheIndexOutput extends IndexOutput {
> > > >> > > +
> > > >> > > +  private final IndexOutput _indexOutput;
> > > >> > > +  private final int _fileBufferSize;
> > > >> > > +  private final Cache _cache;
> > > >> > > +  private final String _fileName;
> > > >> > > +  private final CacheDirectory _directory;
> > > >> > > +  private final long _fileId;
> > > >> > > +
> > > >> > > +  private long _position;
> > > >> > > +  private byte[] _buffer;
> > > >> > > +  private int _bufferPosition;
> > > >> > > +  private int _lastWrittenPosition;
> > > >> > > +  private int _cacheBlockSize;
> > > >> > > +  private CacheValue _cacheValue;
> > > >> > > +
> > > >> > > +  public CacheIndexOutput(CacheDirectory directory, String
> > > fileName,
> > > >> > IndexOutput indexOutput, Cache cache) throws IOException {
> > > >> > > +    _cache = cache;
> > > >> > > +    _directory = directory;
> > > >> > > +    _fileName = fileName;
> > > >> > > +    _fileBufferSize = _cache.getFileBufferSize(_directory,
> > > >> _fileName);
> > > >> > > +    _cacheBlockSize = _cache.getCacheBlockSize(_directory,
> > > >> _fileName);
> > > >> > > +    _fileId = _cache.getFileId(_directory, _fileName);
> > > >> > > +    _indexOutput = indexOutput;
> > > >> > > +    _buffer = BufferStore.takeBuffer(_cacheBlockSize);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void writeByte(byte b) throws IOException {
> > > >> > > +    tryToFlush();
> > > >> > > +    _buffer[_bufferPosition] = b;
> > > >> > > +    _bufferPosition++;
> > > >> > > +    _position++;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void writeBytes(byte[] b, int offset, int len) throws
> > > >> > IOException {
> > > >> > > +    while (len > 0) {
> > > >> > > +      tryToFlush();
> > > >> > > +      int remaining = remaining();
> > > >> > > +      int length = Math.min(len, remaining);
> > > >> > > +      System.arraycopy(b, offset, _buffer, _bufferPosition,
> > > length);
> > > >> > > +      _bufferPosition += length;
> > > >> > > +      _position += length;
> > > >> > > +      len -= length;
> > > >> > > +      offset += length;
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private int remaining() {
> > > >> > > +    return _fileBufferSize - _bufferPosition;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private void tryToFlush() throws IOException {
> > > >> > > +    if (remaining() == 0) {
> > > >> > > +      flushInternal();
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private void flushInternal() throws IOException {
> > > >> > > +    if (_cacheValue == null) {
> > > >> > > +      _cacheValue = _cache.newInstance(_directory, _fileName);
> > > >> > > +    }
> > > >> > > +    int length = _bufferPosition - _lastWrittenPosition;
> > > >> > > +    _indexOutput.writeBytes(_buffer, _lastWrittenPosition,
> > length);
> > > >> > > +    _cacheValue.write(_bufferPosition, _buffer,
> > > _lastWrittenPosition,
> > > >> > length);
> > > >> > > +    if (_bufferPosition == _fileBufferSize) {
> > > >> > > +      _cache.put(new CacheKey(_fileId, getBlockId()),
> > _cacheValue);
> > > >> > > +      _bufferPosition = 0;
> > > >> > > +      _cacheValue = null;
> > > >> > > +    }
> > > >> > > +    _lastWrittenPosition = _bufferPosition;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private long getBlockId() {
> > > >> > > +    return _position / _cacheBlockSize;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void seek(long pos) throws IOException {
> > > >> > > +    throw new IOException("Seek is not supported.");
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void close() throws IOException {
> > > >> > > +    flush();
> > > >> > > +    _indexOutput.close();
> > > >> > > +    BufferStore.putBuffer(_buffer);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void flush() throws IOException {
> > > >> > > +    flushInternal();
> > > >> > > +    _indexOutput.flush();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public long getFilePointer() {
> > > >> > > +    return _position;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public long length() throws IOException {
> > > >> > > +    return getFilePointer();
> > > >> > > +  }
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheKey.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheKey.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheKey.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..340e488
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheKey.java
> > > >> > > @@ -0,0 +1,89 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2;
> > > >> > > +
> > > >> > > +public class CacheKey implements Cloneable {
> > > >> > > +
> > > >> > > +  private long _fileId;
> > > >> > > +  private long _blockId;
> > > >> > > +
> > > >> > > +  public CacheKey() {
> > > >> > > +
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public CacheKey(long fileId, long blockId) {
> > > >> > > +    _fileId = fileId;
> > > >> > > +    _blockId = blockId;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public long getFileId() {
> > > >> > > +    return _fileId;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public void setFileId(long fileId) {
> > > >> > > +    _fileId = fileId;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public long getBlockId() {
> > > >> > > +    return _blockId;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public void setBlockId(long blockId) {
> > > >> > > +    _blockId = blockId;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public CacheKey clone() {
> > > >> > > +    try {
> > > >> > > +      return (CacheKey) super.clone();
> > > >> > > +    } catch (CloneNotSupportedException e) {
> > > >> > > +      throw new RuntimeException(e);
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public int hashCode() {
> > > >> > > +    final int prime = 31;
> > > >> > > +    int result = 1;
> > > >> > > +    result = prime * result + (int) (_blockId ^ (_blockId >>>
> > 32));
> > > >> > > +    result = prime * result + (int) (_fileId ^ (_fileId >>>
> 32));
> > > >> > > +    return result;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public boolean equals(Object obj) {
> > > >> > > +    if (this == obj)
> > > >> > > +      return true;
> > > >> > > +    if (obj == null)
> > > >> > > +      return false;
> > > >> > > +    if (getClass() != obj.getClass())
> > > >> > > +      return false;
> > > >> > > +    CacheKey other = (CacheKey) obj;
> > > >> > > +    if (_blockId != other._blockId)
> > > >> > > +      return false;
> > > >> > > +    if (_fileId != other._fileId)
> > > >> > > +      return false;
> > > >> > > +    return true;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public String toString() {
> > > >> > > +    return "CacheKey [_fileId=" + _fileId + ", _blockId=" +
> > > _blockId
> > > >> +
> > > >> > "]";
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheValue.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheValue.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheValue.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..0f7b976
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheValue.java
> > > >> > > @@ -0,0 +1,93 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2;
> > > >> > > +
> > > >> > > +public interface CacheValue {
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * The actual size of the the underlying resource.
> > > >> > > +   *
> > > >> > > +   * @return the size.
> > > >> > > +   */
> > > >> > > +  int size();
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * The length of the data in this block.
> > > >> > > +   *
> > > >> > > +   * @return the length.
> > > >> > > +   */
> > > >> > > +  int length();
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Writes data out to a given position in this block.
> > > >> > > +   *
> > > >> > > +   * @param position
> > > >> > > +   *          the position.
> > > >> > > +   * @param buf
> > > >> > > +   *          the buffer.
> > > >> > > +   * @param offset
> > > >> > > +   *          the offset in the buffer.
> > > >> > > +   * @param length
> > > >> > > +   *          the length of bytes to write.
> > > >> > > +   */
> > > >> > > +  void write(int position, byte[] buf, int offset, int length);
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Reads data into the buffer given the position.
> > > >> > > +   *
> > > >> > > +   * @param position
> > > >> > > +   *          the position to read.
> > > >> > > +   * @param buf
> > > >> > > +   *          the buffer to read into.
> > > >> > > +   * @param offset
> > > >> > > +   *          the offset within the buffer.
> > > >> > > +   * @param length
> > > >> > > +   *          the length of data to read.
> > > >> > > +   */
> > > >> > > +  void read(int position, byte[] buf, int offset, int length);
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Reads a byte from the given position.
> > > >> > > +   *
> > > >> > > +   * @param position
> > > >> > > +   *          the position.
> > > >> > > +   * @return the byte.
> > > >> > > +   */
> > > >> > > +  byte read(int position);
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Increments the reference.
> > > >> > > +   */
> > > >> > > +  void incRef();
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Decrements the reference.
> > > >> > > +   */
> > > >> > > +  void decRef();
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Gets the reference count.
> > > >> > > +   */
> > > >> > > +  long refCount();
> > > >> > > +
> > > >> > > +  /**
> > > >> > > +   * Releases any underlying resources.
> > > >> > > +   */
> > > >> > > +  void release();
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameBlockSize.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameBlockSize.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameBlockSize.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..8890459
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameBlockSize.java
> > > >> > > @@ -0,0 +1,24 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2;
> > > >> > > +
> > > >> > > +public interface FileNameBlockSize {
> > > >> > > +
> > > >> > > +  int getBlockSize(String directoryName, String fileName);
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameFilter.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameFilter.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameFilter.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..03a969c
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameFilter.java
> > > >> > > @@ -0,0 +1,24 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2;
> > > >> > > +
> > > >> > > +public interface FileNameFilter {
> > > >> > > +
> > > >> > > +  boolean accept(String directoryName, String fileName);
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/BaseCacheValue.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/BaseCacheValue.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/BaseCacheValue.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..a5e10a3
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/BaseCacheValue.java
> > > >> > > @@ -0,0 +1,83 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2.cachevalue;
> > > >> > > +
> > > >> > > +import java.util.concurrent.atomic.AtomicLong;
> > > >> > > +
> > > >> > > +import org.apache.blur.store.blockcache_v2.CacheValue;
> > > >> > > +
> > > >> > > +@SuppressWarnings("serial")
> > > >> > > +public abstract class BaseCacheValue extends AtomicLong
> > implements
> > > >> > CacheValue {
> > > >> > > +
> > > >> > > +  private final int _length;
> > > >> > > +
> > > >> > > +  public BaseCacheValue(int length) {
> > > >> > > +    _length = length;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public final int length() {
> > > >> > > +    return _length;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void write(int position, byte[] buf, int offset, int
> > > >> length) {
> > > >> > > +    if (position + length > _length) {
> > > >> > > +      throw new ArrayIndexOutOfBoundsException(position +
> > length);
> > > >> > > +    }
> > > >> > > +    writeInternal(position, buf, offset, length);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void read(int position, byte[] buf, int offset, int
> > > length)
> > > >> {
> > > >> > > +    if (position + length > _length) {
> > > >> > > +      throw new ArrayIndexOutOfBoundsException(position +
> > length);
> > > >> > > +    }
> > > >> > > +    readInternal(position, buf, offset, length);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public byte read(int position) {
> > > >> > > +    if (position >= _length) {
> > > >> > > +      throw new ArrayIndexOutOfBoundsException(position);
> > > >> > > +    }
> > > >> > > +    return readInternal(position);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  protected abstract void writeInternal(int position, byte[]
> buf,
> > > int
> > > >> > offset, int length);
> > > >> > > +
> > > >> > > +  protected abstract byte readInternal(int position);
> > > >> > > +
> > > >> > > +  protected abstract void readInternal(int position, byte[]
> buf,
> > > int
> > > >> > offset, int length);
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public final void incRef() {
> > > >> > > +    incrementAndGet();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public final void decRef() {
> > > >> > > +    decrementAndGet();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public final long refCount() {
> > > >> > > +    return get();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValue.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValue.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValue.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..c4d84bf
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValue.java
> > > >> > > @@ -0,0 +1,55 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2.cachevalue;
> > > >> > > +
> > > >> > > +@SuppressWarnings("serial")
> > > >> > > +public class ByteArrayCacheValue extends BaseCacheValue {
> > > >> > > +
> > > >> > > +  private final byte[] _buffer;
> > > >> > > +
> > > >> > > +  public ByteArrayCacheValue(int length) {
> > > >> > > +    super(length);
> > > >> > > +    _buffer = new byte[length];
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  protected void writeInternal(int position, byte[] buf, int
> > > offset,
> > > >> > int length) {
> > > >> > > +    System.arraycopy(buf, offset, _buffer, position, length);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  protected void readInternal(int position, byte[] buf, int
> > offset,
> > > >> int
> > > >> > length) {
> > > >> > > +    System.arraycopy(_buffer, position, buf, offset, length);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  protected byte readInternal(int position) {
> > > >> > > +    return _buffer[position];
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void release() {
> > > >> > > +    // no op
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public int size() {
> > > >> > > +    return length();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValue.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValue.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValue.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..a918250
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValue.java
> > > >> > > @@ -0,0 +1,99 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2.cachevalue;
> > > >> > > +
> > > >> > > +import java.lang.reflect.Field;
> > > >> > > +
> > > >> > > +import sun.misc.Unsafe;
> > > >> > > +
> > > >> > > +@SuppressWarnings("serial")
> > > >> > > +public class UnsafeCacheValue extends BaseCacheValue {
> > > >> > > +
> > > >> > > +  private static final int MINIMUM_SIZE = 1024;
> > > >> > > +
> > > >> > > +  private static final Unsafe _unsafe;
> > > >> > > +
> > > >> > > +  static {
> > > >> > > +    try {
> > > >> > > +      Class<?> clazz = Class.forName("java.nio.Bits");
> > > >> > > +      Field field = clazz.getDeclaredField("unsafe");
> > > >> > > +      field.setAccessible(true);
> > > >> > > +      _unsafe = (Unsafe) field.get(null);
> > > >> > > +    } catch (Exception e) {
> > > >> > > +      throw new RuntimeException(e);
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private static final int BYTE_ARRAY_BASE_OFFSET =
> > > >> > _unsafe.arrayBaseOffset(byte[].class);
> > > >> > > +
> > > >> > > +  private static void copyFromArray(byte[] src, int srcOffset,
> > int
> > > >> > length, long destAddress) {
> > > >> > > +    long offset = BYTE_ARRAY_BASE_OFFSET + srcOffset;
> > > >> > > +    _unsafe.copyMemory(src, offset, null, destAddress, length);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private static void copyToArray(long srcAddress, byte[] dst,
> > int
> > > >> > dstOffset, int length) {
> > > >> > > +    long offset = BYTE_ARRAY_BASE_OFFSET + dstOffset;
> > > >> > > +    _unsafe.copyMemory(null, srcAddress, dst, offset, length);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private final long _address;
> > > >> > > +  private final int _capacity;
> > > >> > > +
> > > >> > > +  public UnsafeCacheValue(int length) {
> > > >> > > +    super(length);
> > > >> > > +    _capacity = getCapacity(length);
> > > >> > > +    _address = _unsafe.allocateMemory(_capacity);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private int getCapacity(int length) {
> > > >> > > +    if (length < MINIMUM_SIZE) {
> > > >> > > +      return MINIMUM_SIZE;
> > > >> > > +    }
> > > >> > > +    return length;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  protected void writeInternal(int position, byte[] buf, int
> > > offset,
> > > >> > int length) {
> > > >> > > +    copyFromArray(buf, offset, length,
> resolveAddress(position));
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  protected void readInternal(int position, byte[] buf, int
> > offset,
> > > >> int
> > > >> > length) {
> > > >> > > +    copyToArray(resolveAddress(position), buf, offset, length);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  protected byte readInternal(int position) {
> > > >> > > +    return _unsafe.getByte(resolveAddress(position));
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  private long resolveAddress(int position) {
> > > >> > > +    return _address + position;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public void release() {
> > > >> > > +    _unsafe.freeMemory(_address);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Override
> > > >> > > +  public int size() {
> > > >> > > +    return _capacity;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexInputTest.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexInputTest.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexInputTest.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..6e897bc
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexInputTest.java
> > > >> > > @@ -0,0 +1,206 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2;
> > > >> > > +
> > > >> > > +import static org.junit.Assert.assertArrayEquals;
> > > >> > > +import static org.junit.Assert.assertEquals;
> > > >> > > +import static org.junit.Assert.fail;
> > > >> > > +
> > > >> > > +import java.io.IOException;
> > > >> > > +import java.util.Random;
> > > >> > > +
> > > >> > > +import
> > > >> >
> org.apache.blur.store.blockcache_v2.cachevalue.ByteArrayCacheValue;
> > > >> > > +import org.apache.blur.store.buffer.BufferStore;
> > > >> > > +import org.apache.lucene.store.IOContext;
> > > >> > > +import org.apache.lucene.store.IndexInput;
> > > >> > > +import org.apache.lucene.store.IndexOutput;
> > > >> > > +import org.apache.lucene.store.RAMDirectory;
> > > >> > > +import org.junit.Before;
> > > >> > > +import org.junit.Test;
> > > >> > > +
> > > >> > > +import
> > > >> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap;
> > > >> > > +import com.googlecode.concurrentlinkedhashmap.EvictionListener;
> > > >> > > +import com.googlecode.concurrentlinkedhashmap.Weigher;
> > > >> > > +
> > > >> > > +public class CacheIndexInputTest {
> > > >> > > +
> > > >> > > +  private long seed;
> > > >> > > +
> > > >> > > +  private final int sampleSize = 10000;
> > > >> > > +  private final int maxBufSize = 10000;
> > > >> > > +  private final int maxOffset = 1000;
> > > >> > > +
> > > >> > > +  @Before
> > > >> > > +  public void setup() {
> > > >> > > +    BufferStore.init(128, 128);
> > > >> > > +    seed = new Random().nextLong();
> > > >> > > +    System.out.println("Using seed [" + seed + "]");
> > > >> > > +    // seed = -265282183286396219l;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test1() throws IOException {
> > > >> > > +    RAMDirectory directory = new RAMDirectory();
> > > >> > > +
> > > >> > > +    String name = "test";
> > > >> > > +
> > > >> > > +    IndexOutput output = directory.createOutput(name,
> > > >> > IOContext.DEFAULT);
> > > >> > > +    byte[] bs = "hello world".getBytes();
> > > >> > > +    output.writeBytes(bs, bs.length);
> > > >> > > +    output.close();
> > > >> > > +
> > > >> > > +    IndexInput input = directory.openInput(name,
> > > IOContext.DEFAULT);
> > > >> > > +    Cache cache = getCache();
> > > >> > > +    CacheIndexInput cacheInput = new CacheIndexInput(null,
> name,
> > > >> input,
> > > >> > cache);
> > > >> > > +    byte[] buf = new byte[bs.length];
> > > >> > > +    cacheInput.readBytes(buf, 0, buf.length);
> > > >> > > +    cacheInput.close();
> > > >> > > +
> > > >> > > +    assertArrayEquals(bs, buf);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test2() throws IOException {
> > > >> > > +    Cache cache = getCache();
> > > >> > > +    RAMDirectory directory = new RAMDirectory();
> > > >> > > +    Random random = new Random(seed);
> > > >> > > +
> > > >> > > +    String name = "test2";
> > > >> > > +    long size = (10 * 1024 * 1024) + 13;
> > > >> > > +
> > > >> > > +    IndexOutput output = directory.createOutput(name,
> > > >> > IOContext.DEFAULT);
> > > >> > > +    writeRandomData(size, random, output);
> > > >> > > +    output.close();
> > > >> > > +
> > > >> > > +    IndexInput input = directory.openInput(name,
> > > IOContext.DEFAULT);
> > > >> > > +    IndexInput testInput = new CacheIndexInput(null, name,
> > > >> > input.clone(), cache);
> > > >> > > +    readRandomData(input, testInput, random, sampleSize,
> > > maxBufSize,
> > > >> > maxOffset);
> > > >> > > +    testInput.close();
> > > >> > > +    input.close();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public static void readRandomData(IndexInput baseInput,
> > > IndexInput
> > > >> > testInput, Random random, int sampleSize,
> > > >> > > +      int maxBufSize, int maxOffset) throws IOException {
> > > >> > > +    assertEquals(baseInput.length(), testInput.length());
> > > >> > > +    int fileLength = (int) baseInput.length();
> > > >> > > +    for (int i = 0; i < sampleSize; i++) {
> > > >> > > +      int position = random.nextInt(fileLength - maxBufSize);
> > > >> > > +      int bufSize = random.nextInt(maxBufSize - maxOffset) + 1;
> > > >> > > +      byte[] buf1 = new byte[bufSize];
> > > >> > > +      byte[] buf2 = new byte[bufSize];
> > > >> > > +
> > > >> > > +      int offset = random.nextInt(Math.min(maxOffset,
> bufSize));
> > > >> > > +      int len = Math.min(random.nextInt(bufSize - offset),
> > > >> fileLength -
> > > >> > position);
> > > >> > > +
> > > >> > > +      baseInput.seek(position);
> > > >> > > +      baseInput.readBytes(buf1, offset, len);
> > > >> > > +      testInput.seek(position);
> > > >> > > +      testInput.readBytes(buf2, offset, len);
> > > >> > > +      assertArrayEquals("Read [" + i + "] The position is [" +
> > > >> position
> > > >> > + "] and bufSize [" + bufSize + "]", buf1, buf2);
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  public static void writeRandomData(long size, Random random,
> > > >> > IndexOutput... outputs) throws IOException {
> > > >> > > +    byte[] buf = new byte[1024];
> > > >> > > +    for (long l = 0; l < size; l += buf.length) {
> > > >> > > +      random.nextBytes(buf);
> > > >> > > +      int length = (int) Math.min(buf.length, size - l);
> > > >> > > +      for (IndexOutput output : outputs) {
> > > >> > > +        output.writeBytes(buf, length);
> > > >> > > +      }
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +
> > > >> > > +  public static Cache getCache() {
> > > >> > > +    EvictionListener<CacheKey, CacheValue> listener = new
> > > >> > EvictionListener<CacheKey, CacheValue>() {
> > > >> > > +      @Override
> > > >> > > +      public void onEviction(CacheKey key, CacheValue value) {
> > > >> > > +        if (value.refCount() == 0) {
> > > >> > > +          value.release();
> > > >> > > +        } else {
> > > >> > > +          // doing something else...
> > > >> > > +          fail();
> > > >> > > +        }
> > > >> > > +      }
> > > >> > > +    };
> > > >> > > +    Weigher<CacheValue> weigher = new Weigher<CacheValue>() {
> > > >> > > +      @Override
> > > >> > > +      public int weightOf(CacheValue value) {
> > > >> > > +        return value.length();
> > > >> > > +      }
> > > >> > > +    };
> > > >> > > +    long maximumWeightedCapacity = 1 * 1024 * 1024;
> > > >> > > +    final ConcurrentLinkedHashMap<CacheKey, CacheValue> cache =
> > new
> > > >> > ConcurrentLinkedHashMap.Builder<CacheKey, CacheValue>()
> > > >> > > +
> > > >> >
> > > >>
> > >
> >
>  .weigher(weigher).maximumWeightedCapacity(maximumWeightedCapacity).listener(listener).build();
> > > >> > > +    Cache cacheFactory = new Cache() {
> > > >> > > +
> > > >> > > +      @Override
> > > >> > > +      public CacheValue newInstance(CacheDirectory directory,
> > > String
> > > >> > fileName) {
> > > >> > > +        return new
> > ByteArrayCacheValue(getCacheBlockSize(directory,
> > > >> > fileName));
> > > >> > > +      }
> > > >> > > +
> > > >> > > +      @Override
> > > >> > > +      public long getFileId(CacheDirectory directory, String
> > > >> fileName) {
> > > >> > > +        return fileName.hashCode();
> > > >> > > +      }
> > > >> > > +
> > > >> > > +      @Override
> > > >> > > +      public int getFileBufferSize(CacheDirectory directory,
> > String
> > > >> > fileName) {
> > > >> > > +        return 1024;
> > > >> > > +      }
> > > >> > > +
> > > >> > > +      @Override
> > > >> > > +      public int getCacheBlockSize(CacheDirectory directory,
> > String
> > > >> > fileName) {
> > > >> > > +        return 8192;
> > > >> > > +      }
> > > >> > > +
> > > >> > > +      @Override
> > > >> > > +      public boolean cacheFileForReading(CacheDirectory
> > directory,
> > > >> > String name, IOContext context) {
> > > >> > > +        return true;
> > > >> > > +      }
> > > >> > > +
> > > >> > > +      @Override
> > > >> > > +      public boolean cacheFileForWriting(CacheDirectory
> > directory,
> > > >> > String name, IOContext context) {
> > > >> > > +        return true;
> > > >> > > +      }
> > > >> > > +
> > > >> > > +      @Override
> > > >> > > +      public CacheValue get(CacheKey key) {
> > > >> > > +        return cache.get(key);
> > > >> > > +      }
> > > >> > > +
> > > >> > > +      @Override
> > > >> > > +      public void put(CacheKey key, CacheValue value) {
> > > >> > > +        cache.put(key, value);
> > > >> > > +      }
> > > >> > > +
> > > >> > > +      @Override
> > > >> > > +      public void removeFile(CacheDirectory directory, String
> > > >> fileName)
> > > >> > throws IOException {
> > > >> > > +
> > > >> > > +      }
> > > >> > > +
> > > >> > > +      @Override
> > > >> > > +      public void releaseDirectory(String directoryName) {
> > > >> > > +
> > > >> > > +      }
> > > >> > > +    };
> > > >> > > +    return cacheFactory;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexOutputTest.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexOutputTest.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexOutputTest.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..2dc8b5b
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexOutputTest.java
> > > >> > > @@ -0,0 +1,101 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2;
> > > >> > > +
> > > >> > > +import static org.junit.Assert.assertArrayEquals;
> > > >> > > +import static org.junit.Assert.assertEquals;
> > > >> > > +
> > > >> > > +import java.io.IOException;
> > > >> > > +import java.util.Random;
> > > >> > > +
> > > >> > > +import org.apache.blur.store.buffer.BufferStore;
> > > >> > > +import org.apache.lucene.store.IOContext;
> > > >> > > +import org.apache.lucene.store.IndexInput;
> > > >> > > +import org.apache.lucene.store.IndexOutput;
> > > >> > > +import org.apache.lucene.store.RAMDirectory;
> > > >> > > +import org.junit.Before;
> > > >> > > +import org.junit.Test;
> > > >> > > +
> > > >> > > +public class CacheIndexOutputTest {
> > > >> > > +
> > > >> > > +  private long seed;
> > > >> > > +
> > > >> > > +  private final int sampleSize = 10000;
> > > >> > > +  private final int maxBufSize = 10000;
> > > >> > > +  private final int maxOffset = 1000;
> > > >> > > +
> > > >> > > +  @Before
> > > >> > > +  public void setup() {
> > > >> > > +    BufferStore.init(128, 128);
> > > >> > > +    seed = new Random().nextLong();
> > > >> > > +    System.out.println("Using seed [" + seed + "]");
> > > >> > > +    // seed = -265282183286396219l;
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test1() throws IOException {
> > > >> > > +    Random random = new Random(seed);
> > > >> > > +    RAMDirectory directory = new RAMDirectory();
> > > >> > > +    IndexOutput output = directory.createOutput("test",
> > > >> > IOContext.DEFAULT);
> > > >> > > +
> > > >> > > +    Cache cache = CacheIndexInputTest.getCache();
> > > >> > > +    CacheIndexOutput indexOutput = new CacheIndexOutput(null,
> > > "test",
> > > >> > output, cache);
> > > >> > > +    indexOutput.writeByte((byte) 1);
> > > >> > > +    indexOutput.writeByte((byte) 2);
> > > >> > > +    byte[] b = new byte[16000];
> > > >> > > +    random.nextBytes(b);
> > > >> > > +    indexOutput.writeBytes(b, 16000);
> > > >> > > +    indexOutput.close();
> > > >> > > +
> > > >> > > +    IndexInput input = directory.openInput("test",
> > > >> IOContext.DEFAULT);
> > > >> > > +    assertEquals(16002, input.length());
> > > >> > > +    assertEquals(1, input.readByte());
> > > >> > > +    assertEquals(2, input.readByte());
> > > >> > > +
> > > >> > > +    byte[] buf = new byte[16000];
> > > >> > > +    input.readBytes(buf, 0, 16000);
> > > >> > > +    input.close();
> > > >> > > +    assertArrayEquals(b, buf);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test2() throws IOException {
> > > >> > > +    Cache cache = CacheIndexInputTest.getCache();
> > > >> > > +    RAMDirectory directory = new RAMDirectory();
> > > >> > > +    RAMDirectory directory2 = new RAMDirectory();
> > > >> > > +
> > > >> > > +    Random random = new Random(seed);
> > > >> > > +
> > > >> > > +    String name = "test2";
> > > >> > > +    long size = (10 * 1024 * 1024) + 13;
> > > >> > > +
> > > >> > > +    IndexOutput output = directory.createOutput(name,
> > > >> > IOContext.DEFAULT);
> > > >> > > +    IndexOutput output2 = directory2.createOutput(name,
> > > >> > IOContext.DEFAULT);
> > > >> > > +    CacheIndexOutput cacheIndexOutput = new
> > CacheIndexOutput(null,
> > > >> > name, output2, cache);
> > > >> > > +    CacheIndexInputTest.writeRandomData(size, random, output,
> > > >> > cacheIndexOutput);
> > > >> > > +    output.close();
> > > >> > > +    cacheIndexOutput.close();
> > > >> > > +
> > > >> > > +    IndexInput input = directory.openInput(name,
> > > IOContext.DEFAULT);
> > > >> > > +    IndexInput testInput = directory2.openInput(name,
> > > >> > IOContext.DEFAULT);
> > > >> > > +    CacheIndexInputTest.readRandomData(input, testInput,
> random,
> > > >> > sampleSize, maxBufSize, maxOffset);
> > > >> > > +    testInput.close();
> > > >> > > +    input.close();
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValueTest.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValueTest.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValueTest.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..8d33ece
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValueTest.java
> > > >> > > @@ -0,0 +1,68 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2.cachevalue;
> > > >> > > +
> > > >> > > +import static org.junit.Assert.*;
> > > >> > > +
> > > >> > > +import org.junit.Test;
> > > >> > > +
> > > >> > > +public class ByteArrayCacheValueTest {
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test1() {
> > > >> > > +    ByteArrayCacheValue value = new ByteArrayCacheValue(10);
> > > >> > > +    byte[] buf = "hello world".getBytes();
> > > >> > > +    value.write(0, buf, 0, 10);
> > > >> > > +    byte[] buf2 = new byte[10];
> > > >> > > +    value.read(0, buf2, 0, 10);
> > > >> > > +    assertArrayEquals("hello worl".getBytes(), buf2);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test2() {
> > > >> > > +    ByteArrayCacheValue value = new ByteArrayCacheValue(10);
> > > >> > > +    byte[] buf = "hello world".getBytes();
> > > >> > > +    try {
> > > >> > > +      value.write(0, buf, 0, 11);
> > > >> > > +      fail();
> > > >> > > +    } catch (ArrayIndexOutOfBoundsException e) {
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test3() {
> > > >> > > +    ByteArrayCacheValue value = new ByteArrayCacheValue(10);
> > > >> > > +    byte[] buf = "hello world".getBytes();
> > > >> > > +    try {
> > > >> > > +      value.write(8, buf, 0, 3);
> > > >> > > +      fail();
> > > >> > > +    } catch (ArrayIndexOutOfBoundsException e) {
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test4() {
> > > >> > > +    ByteArrayCacheValue value = new ByteArrayCacheValue(10);
> > > >> > > +    byte[] buf = "hello world".getBytes();
> > > >> > > +    value.write(8, buf, 0, 2);
> > > >> > > +
> > > >> > > +    assertEquals('h', value.read(8));
> > > >> > > +    assertEquals('e', value.read(9));
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValueTest.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> >
> > > >>
> > >
> >
> a/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValueTest.java
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValueTest.java
> > > >> > > new file mode 100644
> > > >> > > index 0000000..37f5926
> > > >> > > --- /dev/null
> > > >> > > +++
> > > >> >
> > > >>
> > >
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValueTest.java
> > > >> > > @@ -0,0 +1,68 @@
> > > >> > > +/**
> > > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > > >> > > + * or more contributor license agreements.  See the NOTICE file
> > > >> > > + * distributed with this work for additional information
> > > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > > >> > > + * to you under the Apache License, Version 2.0 (the
> > > >> > > + * "License"); you may not use this file except in compliance
> > > >> > > + * with the License.  You may obtain a copy of the License at
> > > >> > > + *
> > > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > > >> > > + *
> > > >> > > + * Unless required by applicable law or agreed to in writing,
> > > >> software
> > > >> > > + * distributed under the License is distributed on an "AS IS"
> > > BASIS,
> > > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> > or
> > > >> > implied.
> > > >> > > + * See the License for the specific language governing
> > permissions
> > > >> and
> > > >> > > + * limitations under the License.
> > > >> > > + */
> > > >> > > +package org.apache.blur.store.blockcache_v2.cachevalue;
> > > >> > > +
> > > >> > > +import static org.junit.Assert.*;
> > > >> > > +
> > > >> > > +import org.junit.Test;
> > > >> > > +
> > > >> > > +public class UnsafeCacheValueTest {
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test1() {
> > > >> > > +    UnsafeCacheValue value = new UnsafeCacheValue(10);
> > > >> > > +    byte[] buf = "hello world".getBytes();
> > > >> > > +    value.write(0, buf, 0, 10);
> > > >> > > +    byte[] buf2 = new byte[10];
> > > >> > > +    value.read(0, buf2, 0, 10);
> > > >> > > +    assertArrayEquals("hello worl".getBytes(), buf2);
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test2() {
> > > >> > > +    UnsafeCacheValue value = new UnsafeCacheValue(10);
> > > >> > > +    byte[] buf = "hello world".getBytes();
> > > >> > > +    try {
> > > >> > > +      value.write(0, buf, 0, 11);
> > > >> > > +      fail();
> > > >> > > +    } catch (ArrayIndexOutOfBoundsException e) {
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test3() {
> > > >> > > +    UnsafeCacheValue value = new UnsafeCacheValue(10);
> > > >> > > +    byte[] buf = "hello world".getBytes();
> > > >> > > +    try {
> > > >> > > +      value.write(8, buf, 0, 3);
> > > >> > > +      fail();
> > > >> > > +    } catch (ArrayIndexOutOfBoundsException e) {
> > > >> > > +    }
> > > >> > > +  }
> > > >> > > +
> > > >> > > +  @Test
> > > >> > > +  public void test4() {
> > > >> > > +    UnsafeCacheValue value = new UnsafeCacheValue(10);
> > > >> > > +    byte[] buf = "hello world".getBytes();
> > > >> > > +    value.write(8, buf, 0, 2);
> > > >> > > +
> > > >> > > +    assertEquals('h', value.read(8));
> > > >> > > +    assertEquals('e', value.read(9));
> > > >> > > +  }
> > > >> > > +
> > > >> > > +}
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-util/src/main/java/org/apache/blur/utils/BlurConstants.java
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> > > >> > a/blur-util/src/main/java/org/apache/blur/utils/BlurConstants.java
> > > >> > b/blur-util/src/main/java/org/apache/blur/utils/BlurConstants.java
> > > >> > > index 60368f8..fbe86d0 100644
> > > >> > > ---
> > > a/blur-util/src/main/java/org/apache/blur/utils/BlurConstants.java
> > > >> > > +++
> > > b/blur-util/src/main/java/org/apache/blur/utils/BlurConstants.java
> > > >> > > @@ -44,6 +44,7 @@ public class BlurConstants {
> > > >> > >    public static final String BLUR_SHARD_BIND_ADDRESS =
> > > >> > "blur.shard.bind.address";
> > > >> > >    public static final String
> > > >> > BLUR_SHARD_BLOCKCACHE_DIRECT_MEMORY_ALLOCATION =
> > > >> > "blur.shard.blockcache.direct.memory.allocation";
> > > >> > >    public static final String BLUR_SHARD_BLOCKCACHE_SLAB_COUNT =
> > > >> > "blur.shard.blockcache.slab.count";
> > > >> > > +  public static final String
> BLUR_SHARD_EXPERIMENTAL_BLOCK_CACHE
> > =
> > > >> > "blur.shard.experimental.block.cache";
> > > >> > >    public static final String BLUR_SHARD_SAFEMODEDELAY =
> > > >> > "blur.shard.safemodedelay";
> > > >> > >    public static final String BLUR_CONTROLLER_HOSTNAME =
> > > >> > "blur.controller.hostname";
> > > >> > >    public static final String BLUR_CONTROLLER_BIND_PORT =
> > > >> > "blur.controller.bind.port";
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-util/src/main/resources/blur-default.properties
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git
> a/blur-util/src/main/resources/blur-default.properties
> > > >> > b/blur-util/src/main/resources/blur-default.properties
> > > >> > > index d28f39d..4a1ef8b 100644
> > > >> > > --- a/blur-util/src/main/resources/blur-default.properties
> > > >> > > +++ b/blur-util/src/main/resources/blur-default.properties
> > > >> > > @@ -66,6 +66,9 @@ blur.shard.index.warmup.throttle=30000000
> > > >> > >  # By default the block cache using off heap memory
> > > >> > >  blur.shard.blockcache.direct.memory.allocation=true
> > > >> > >
> > > >> > > +# By default the experimental block cache is off
> > > >> > > +blur.shard.experimental.block.cache=false
> > > >> > > +
> > > >> > >  # The slabs in the blockcache are automatically configured by
> > > default
> > > >> > (-1) otherwise 1 slab equals 128MB
> > > >> > >  # The auto config is detected through the
> MaxDirectoryMemorySize
> > > >> > provided to the JVM
> > > >> > >  blur.shard.blockcache.slab.count=-1
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/docs/cluster-setup.html
> > > >> > >
> > > ----------------------------------------------------------------------
> > > >> > > diff --git a/docs/cluster-setup.html b/docs/cluster-setup.html
> > > >> > > index d6ba58b..e63e4f5 100644
> > > >> > > --- a/docs/cluster-setup.html
> > > >> > > +++ b/docs/cluster-setup.html
> > > >> > > @@ -284,6 +284,9 @@ blur.shard.index.warmup.throttle=30000000
> > > >> > >  # By default the block cache using off heap memory
> > > >> > >  blur.shard.blockcache.direct.memory.allocation=true
> > > >> > >
> > > >> > > +# By default the experimental block cache is off
> > > >> > > +blur.shard.experimental.block.cache=false
> > > >> > > +
> > > >> > >  # The slabs in the blockcache are automatically configured by
> > > >> > >  # default (-1) otherwise 1 slab equals 128MB. The auto config
> > > >> > >  # is detected through the MaxDirectoryMemorySize provided to
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message