incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <amccu...@gmail.com>
Subject Re: git commit: Adding first cut at the block cache version 2. Can be enabling in the blur-site via "blur.shard.experimental.block.cache" property.
Date Tue, 03 Sep 2013 21:03:00 GMT
Rahul these are all great question.  I will attempt to answer them below.


On Tue, Sep 3, 2013 at 4:30 PM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Hi Aaron,
>
> Below are a few questions that I had while looking at the existing
> BlockCache functionality. While these may be irrelevant with the new cache
> implementation, some clarifications would still be helpful. Thank You.
>
> 1. In BlockCache we have an eviction listener registered but where are we
> actually removing elements from the cache? If our cache is full and it has
> stale data which has not been used for certain time and new data
> comes....we are not replacing it. Shouldn't we replace unused blocks from
> the cache?
>

Ok, this is a fairly complex implementation (hence the V2).  The map that
is used in the LRU cache is the ConcurrentLinkedHashMap (from google code)
and it actually calls the eviction handler.  The key in the LRU are
basically directory + file + block number and the value is simply a pointer
to a slab (the array of ByteBuffers) and a position in that slab where the
8K of cache data is stored.  Along with the key and value in the LRU there
are bitsets that mark what blocks are not in use and which ones are.  So
when an entry in the LRU is evicted the bitset is updated as being not in
use, then another put entry can make use of that empty slot.


> 2. What is the reason to duplicate a slab before returning it in the
> BlurCache.getSlab() method?
>

Thread safety, you have to duplicate the ByteBuffer to allow multiple
threads to read and write to the ByteBuffer at the same time.


>
> 3. Why is BlockLocks not extending OpenBitSet?
>

Thread safety again, OpenBitSet is not thread safe.


>
> 4. What is the purpose of using AtomicLongArray and wlen instead of a byte
> array in BlockLocks?...With AtomicLongArray if we have to modify a single
> bit we have update the whole word(64bits) containing the bit. (Note :
> assuming byte level operations are atomic)
>

Length, if you use an atomic byte array you can only address 2GB of space
if you use an atomic long array you can address 16GB (8 bytes per long).
Probably overkill but it also made the writing of that bitset easier
because I could just copy the logic from OpenBitSet.


>
> 5. What is the use of BlockCache._lockCounters? Is it just the
> cardinality(no of bits set) of BlockLocks?
>

Speed, calculating cardinality requires a sweep of all the memory in the
bitset.  Keeping the counter makes it faster.


>
> 6. Why are we cloning the BlockCacheKey (BlockCache.store()) before putting
> it into the _cache(Map) for new locations. The code seems to be releasing
> the old location which might be a result of a concurrent process adding the
> location after the current process has read the cache and found that it
> does not exist.
>

BlockCacheKey is cloned because it is likely reused in the calling logic
(can't trust anyone) and if the object is changed by the caller and not
cloned the hash map can get messed up.


>
> 7. What is the use of _threadLocalSlabs in BlockCache?
>

I did some testing and it seems like duplicate on the ByteBuffer is slower
than just using a ThreadLocal variable and reusing the duplicated
ByteBuffer over and over again.


Hope this helps.  Let me know if you have more questions.

Aaron


>
> - Rahul
>
>
> On Tue, Sep 3, 2013 at 1:17 PM, Aaron McCurry <amccurry@gmail.com> wrote:
>
> > Whoops, I meant to push this into a branch (Thanks Tim).  Sorry.  I can
> > move it into a branch tonight if everyone wants me to...  But the code is
> > disabled.
> >
> > Aaron
> >
> >
> > On Tue, Sep 3, 2013 at 4:13 PM, Aaron McCurry <amccurry@gmail.com>
> wrote:
> >
> > > I have added the code for a replacement block cache (V2).  It is
> > currently
> > > disabled and marked experimental, the original block cache is still in
> > > place and the default for  Blur.  I wanted to add the code in so that
> > when
> > > I test the upcoming RC candidate that I can try it out.  I apologize
> for
> > > adding it before posting about the work but my laptop was dieing (looks
> > > like the display is dead) and I was afraid of losing the work.  I plan
> on
> > > posting an explanation at some point tonight when I can get on my other
> > > computer.  Again, this is not effecting any of the current block cache.
> > >
> > > Aaron
> > >
> > >
> > > On Tue, Sep 3, 2013 at 4:05 PM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > >> Could you elaborate on what we are trying to achieve? I was looking at
> > the
> > >> existing functionality and have a few questions which I will post
> soon.
> > >>
> > >> - Rahul
> > >>
> > >>
> > >> On Tue, Sep 3, 2013 at 1:02 PM, Tim Williams <williamstw@gmail.com>
> > >> wrote:
> > >>
> > >> > in master? what's the plan for this?
> > >> >
> > >> > --tim
> > >> >
> > >> > On Tue, Sep 3, 2013 at 3:34 PM,  <amccurry@apache.org> wrote:
> > >> > > Updated Branches:
> > >> > >   refs/heads/master bc55d21fb -> 5ea9c30e2
> > >> > >
> > >> > >
> > >> > > Adding first cut at the block cache version 2.  Can be enabling in
> > the
> > >> > blur-site via "blur.shard.experimental.block.cache" property.
> > >> > >
> > >> > >
> > >> > > Project:
> http://git-wip-us.apache.org/repos/asf/incubator-blur/repo
> > >> > > Commit:
> > >> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/commit/5ea9c30e
> > >> > > Tree:
> > >> > http://git-wip-us.apache.org/repos/asf/incubator-blur/tree/5ea9c30e
> > >> > > Diff:
> > >> > http://git-wip-us.apache.org/repos/asf/incubator-blur/diff/5ea9c30e
> > >> > >
> > >> > > Branch: refs/heads/master
> > >> > > Commit: 5ea9c30e2172921be492bdb10ccd4a802e398a23
> > >> > > Parents: bc55d21
> > >> > > Author: Aaron McCurry <amccurry@gmail.com>
> > >> > > Authored: Tue Sep 3 15:33:07 2013 -0400
> > >> > > Committer: Aaron McCurry <amccurry@gmail.com>
> > >> > > Committed: Tue Sep 3 15:33:07 2013 -0400
> > >> > >
> > >> > >
> > ----------------------------------------------------------------------
> > >> > >  .../indexserver/DistributedIndexServer.java     |  15 +-
> > >> > >  .../blur/thrift/ThriftBlurShardServer.java      | 117 +++++------
> > >> > >  .../blur/store/BlockCacheDirectoryFactory.java  |  28 +++
> > >> > >  .../store/BlockCacheDirectoryFactoryV1.java     |  39 ++++
> > >> > >  .../store/BlockCacheDirectoryFactoryV2.java     |  62 ++++++
> > >> > >  .../blur/store/blockcache_v2/BaseCache.java     | 210
> > >> > +++++++++++++++++++
> > >> > >  .../apache/blur/store/blockcache_v2/Cache.java  | 138
> ++++++++++++
> > >> > >  .../store/blockcache_v2/CacheDirectory.java     | 147
> +++++++++++++
> > >> > >  .../store/blockcache_v2/CacheIndexInput.java    | 171
> > +++++++++++++++
> > >> > >  .../store/blockcache_v2/CacheIndexOutput.java   | 130
> ++++++++++++
> > >> > >  .../blur/store/blockcache_v2/CacheKey.java      |  89 ++++++++
> > >> > >  .../blur/store/blockcache_v2/CacheValue.java    |  93 ++++++++
> > >> > >  .../store/blockcache_v2/FileNameBlockSize.java  |  24 +++
> > >> > >  .../store/blockcache_v2/FileNameFilter.java     |  24 +++
> > >> > >  .../cachevalue/BaseCacheValue.java              |  83 ++++++++
> > >> > >  .../cachevalue/ByteArrayCacheValue.java         |  55 +++++
> > >> > >  .../cachevalue/UnsafeCacheValue.java            |  99 +++++++++
> > >> > >  .../blockcache_v2/CacheIndexInputTest.java      | 206
> > >> ++++++++++++++++++
> > >> > >  .../blockcache_v2/CacheIndexOutputTest.java     | 101 +++++++++
> > >> > >  .../cachevalue/ByteArrayCacheValueTest.java     |  68 ++++++
> > >> > >  .../cachevalue/UnsafeCacheValueTest.java        |  68 ++++++
> > >> > >  .../org/apache/blur/utils/BlurConstants.java    |   1 +
> > >> > >  .../src/main/resources/blur-default.properties  |   3 +
> > >> > >  docs/cluster-setup.html                         |   3 +
> > >> > >  24 files changed, 1907 insertions(+), 67 deletions(-)
> > >> > >
> > ----------------------------------------------------------------------
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-core/src/main/java/org/apache/blur/manager/indexserver/DistributedIndexServer.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-core/src/main/java/org/apache/blur/manager/indexserver/DistributedIndexServer.java
> > >> >
> > >>
> >
> b/blur-core/src/main/java/org/apache/blur/manager/indexserver/DistributedIndexServer.java
> > >> > > index 6b7e942..ff3ce79 100644
> > >> > > ---
> > >> >
> > >>
> >
> a/blur-core/src/main/java/org/apache/blur/manager/indexserver/DistributedIndexServer.java
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-core/src/main/java/org/apache/blur/manager/indexserver/DistributedIndexServer.java
> > >> > > @@ -63,8 +63,7 @@ import org.apache.blur.metrics.AtomicLongGauge;
> > >> > >  import org.apache.blur.server.IndexSearcherClosable;
> > >> > >  import org.apache.blur.server.ShardContext;
> > >> > >  import org.apache.blur.server.TableContext;
> > >> > > -import org.apache.blur.store.blockcache.BlockDirectory;
> > >> > > -import org.apache.blur.store.blockcache.Cache;
> > >> > > +import org.apache.blur.store.BlockCacheDirectoryFactory;
> > >> > >  import org.apache.blur.store.hdfs.BlurLockFactory;
> > >> > >  import org.apache.blur.store.hdfs.HdfsDirectory;
> > >> > >  import org.apache.blur.thrift.generated.ShardState;
> > >> > > @@ -107,7 +106,7 @@ public class DistributedIndexServer extends
> > >> > AbstractIndexServer {
> > >> > >    private Configuration _configuration;
> > >> > >    private String _nodeName;
> > >> > >    private int _shardOpenerThreadCount;
> > >> > > -  private Cache _cache;
> > >> > > +  private BlockCacheDirectoryFactory _blockCacheDirectoryFactory;
> > >> > >    private ZooKeeper _zookeeper;
> > >> > >    private String _cluster;
> > >> > >
> > >> > > @@ -448,7 +447,7 @@ public class DistributedIndexServer extends
> > >> > AbstractIndexServer {
> > >> > >      boolean blockCacheEnabled =
> > >> > _clusterStatus.isBlockCacheEnabled(_cluster, table);
> > >> > >      if (blockCacheEnabled) {
> > >> > >        Set<String> blockCacheFileTypes =
> > >> > _clusterStatus.getBlockCacheFileTypes(_cluster, table);
> > >> > > -      dir = new BlockDirectory(table + "_" + shard, directory,
> > >> _cache,
> > >> > blockCacheFileTypes);
> > >> > > +      dir = _blockCacheDirectoryFactory.newDirectory(table + "_"
> +
> > >> > shard, directory, blockCacheFileTypes);
> > >> > >      } else {
> > >> > >        dir = directory;
> > >> > >      }
> > >> > > @@ -717,10 +716,6 @@ public class DistributedIndexServer extends
> > >> > AbstractIndexServer {
> > >> > >      _shardOpenerThreadCount = shardOpenerThreadCount;
> > >> > >    }
> > >> > >
> > >> > > -  public void setCache(Cache cache) {
> > >> > > -    _cache = cache;
> > >> > > -  }
> > >> > > -
> > >> > >    public void setZookeeper(ZooKeeper zookeeper) {
> > >> > >      _zookeeper = zookeeper;
> > >> > >    }
> > >> > > @@ -752,4 +747,8 @@ public class DistributedIndexServer extends
> > >> > AbstractIndexServer {
> > >> > >    public static AtomicLong getPauseWarmup() {
> > >> > >      return _pauseWarmup;
> > >> > >    }
> > >> > > +
> > >> > > +  public void
> > >> setBlockCacheDirectoryFactory(BlockCacheDirectoryFactory
> > >> > blockCacheDirectoryFactory) {
> > >> > > +    _blockCacheDirectoryFactory = blockCacheDirectoryFactory;
> > >> > > +  }
> > >> > >  }
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-core/src/main/java/org/apache/blur/thrift/ThriftBlurShardServer.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-core/src/main/java/org/apache/blur/thrift/ThriftBlurShardServer.java
> > >> >
> > >>
> >
> b/blur-core/src/main/java/org/apache/blur/thrift/ThriftBlurShardServer.java
> > >> > > index aeead2f..9d634ab 100644
> > >> > > ---
> > >> >
> > >>
> >
> a/blur-core/src/main/java/org/apache/blur/thrift/ThriftBlurShardServer.java
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-core/src/main/java/org/apache/blur/thrift/ThriftBlurShardServer.java
> > >> > > @@ -16,20 +16,25 @@ package org.apache.blur.thrift;
> > >> > >   * See the License for the specific language governing
> permissions
> > >> and
> > >> > >   * limitations under the License.
> > >> > >   */
> > >> > > -import static org.apache.blur.utils.BlurConstants.*;
> > >> > > +import static org.apache.blur.utils.BlurConstants.BLUR_CLUSTER;
> > >> > >  import static
> > org.apache.blur.utils.BlurConstants.BLUR_CLUSTER_NAME;
> > >> > >  import static
> > >> > org.apache.blur.utils.BlurConstants.BLUR_CONTROLLER_BIND_PORT;
> > >> > >  import static
> > >> > org.apache.blur.utils.BlurConstants.BLUR_GUI_CONTROLLER_PORT;
> > >> > >  import static
> > >> org.apache.blur.utils.BlurConstants.BLUR_GUI_SHARD_PORT;
> > >> > >  import static
> > >> >
> > >>
> >
> org.apache.blur.utils.BlurConstants.BLUR_INDEXMANAGER_SEARCH_THREAD_COUNT;
> > >> > >  import static
> > >> org.apache.blur.utils.BlurConstants.BLUR_MAX_CLAUSE_COUNT;
> > >> > > +import static
> > >> > org.apache.blur.utils.BlurConstants.BLUR_MAX_HEAP_PER_ROW_FETCH;
> > >> > > +import static
> > >> >
> > >>
> >
> org.apache.blur.utils.BlurConstants.BLUR_MAX_RECORDS_PER_ROW_FETCH_REQUEST;
> > >> > >  import static
> > >> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_BIND_ADDRESS;
> > >> > >  import static
> > >> org.apache.blur.utils.BlurConstants.BLUR_SHARD_BIND_PORT;
> > >> > >  import static
> > >> >
> > >>
> >
> org.apache.blur.utils.BlurConstants.BLUR_SHARD_BLOCKCACHE_DIRECT_MEMORY_ALLOCATION;
> > >> > >  import static
> > >> >
> org.apache.blur.utils.BlurConstants.BLUR_SHARD_BLOCKCACHE_SLAB_COUNT;
> > >> > > +import static
> > >> >
> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_EXPERIMENTAL_BLOCK_CACHE;
> > >> > > +import static
> > >> org.apache.blur.utils.BlurConstants.BLUR_SHARD_FETCHCOUNT;
> > >> > >  import static
> > >> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_FILTER_CACHE_CLASS;
> > >> > >  import static
> > >> org.apache.blur.utils.BlurConstants.BLUR_SHARD_HOSTNAME;
> > >> > >  import static
> > >> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_INDEX_WARMUP_CLASS;
> > >> > > +import static
> > >> >
> org.apache.blur.utils.BlurConstants.BLUR_SHARD_INDEX_WARMUP_THROTTLE;
> > >> > >  import static
> > >> >
> > >>
> >
> org.apache.blur.utils.BlurConstants.BLUR_SHARD_INTERNAL_SEARCH_THREAD_COUNT;
> > >> > >  import static
> > >> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_OPENER_THREAD_COUNT;
> > >> > >  import static
> > >> > org.apache.blur.utils.BlurConstants.BLUR_SHARD_SAFEMODEDELAY;
> > >> > > @@ -40,8 +45,6 @@ import static
> > >> > org.apache.blur.utils.BlurConstants.BLUR_ZOOKEEPER_TIMEOUT;
> > >> > >  import static
> > >> > org.apache.blur.utils.BlurConstants.BLUR_ZOOKEEPER_TIMEOUT_DEFAULT;
> > >> > >  import static org.apache.blur.utils.BlurUtil.quietClose;
> > >> > >
> > >> > > -import java.lang.management.ManagementFactory;
> > >> > > -import java.util.List;
> > >> > >  import java.util.concurrent.TimeUnit;
> > >> > >
> > >> > >  import org.apache.blur.BlurConfiguration;
> > >> > > @@ -65,6 +68,9 @@ import
> > >> > org.apache.blur.manager.writer.BlurIndexRefresher;
> > >> > >  import org.apache.blur.metrics.JSONReporter;
> > >> > >  import org.apache.blur.metrics.ReporterSetup;
> > >> > >  import org.apache.blur.server.ShardServerEventHandler;
> > >> > > +import org.apache.blur.store.BlockCacheDirectoryFactory;
> > >> > > +import org.apache.blur.store.BlockCacheDirectoryFactoryV1;
> > >> > > +import org.apache.blur.store.BlockCacheDirectoryFactoryV2;
> > >> > >  import org.apache.blur.store.blockcache.BlockCache;
> > >> > >  import org.apache.blur.store.blockcache.BlockDirectory;
> > >> > >  import org.apache.blur.store.blockcache.BlockDirectoryCache;
> > >> > > @@ -83,10 +89,13 @@ import org.apache.zookeeper.ZooKeeper;
> > >> > >  import org.mortbay.jetty.servlet.ServletHolder;
> > >> > >  import org.mortbay.jetty.webapp.WebAppContext;
> > >> > >
> > >> > > +import sun.misc.VM;
> > >> > > +
> > >> > >  public class ThriftBlurShardServer extends ThriftServer {
> > >> > >
> > >> > >    private static final Log LOG =
> > >> > LogFactory.getLog(ThriftBlurShardServer.class);
> > >> > >    private static final boolean enableJsonReporter = false;
> > >> > > +  private static final long _64MB = 0;
> > >> > >
> > >> > >    public static void main(String[] args) throws Exception {
> > >> > >      int serverIndex = getServerIndex(args);
> > >> > > @@ -97,21 +106,13 @@ public class ThriftBlurShardServer extends
> > >> > ThriftServer {
> > >> > >      ReporterSetup.setupReporters(configuration);
> > >> > >      MemoryReporter.enable();
> > >> > >      setupJvmMetrics();
> > >> > > -    //make this configurable
> > >> > > +    // make this configurable
> > >> > >      GCWatcher.init(0.75);
> > >> > >      ThriftServer server = createServer(serverIndex,
> configuration);
> > >> > >      server.start();
> > >> > >    }
> > >> > >
> > >> > >    public static ThriftServer createServer(int serverIndex,
> > >> > BlurConfiguration configuration) throws Exception {
> > >> > > -    // setup block cache
> > >> > > -    // 134,217,728 is the slab size, therefore there are 16,384
> > >> blocks
> > >> > > -    // in a slab when using a block size of 8,192
> > >> > > -    int numberOfBlocksPerSlab = 16384;
> > >> > > -    int blockSize = BlockDirectory.BLOCK_SIZE;
> > >> > > -    int slabCount =
> > >> > configuration.getInt(BLUR_SHARD_BLOCKCACHE_SLAB_COUNT, -1);
> > >> > > -    slabCount = getSlabCount(slabCount, numberOfBlocksPerSlab,
> > >> > blockSize);
> > >> > > -    Cache cache;
> > >> > >      Configuration config = new Configuration();
> > >> > >
> > >> > >      String bindAddress =
> > configuration.get(BLUR_SHARD_BIND_ADDRESS);
> > >> > > @@ -132,34 +133,50 @@ public class ThriftBlurShardServer extends
> > >> > ThriftServer {
> > >> > >        httpServer = null;
> > >> > >      }
> > >> > >
> > >> > > -    if (slabCount >= 1) {
> > >> > > -      BlockCache blockCache;
> > >> > > -      boolean directAllocation =
> > >> >
> > configuration.getBoolean(BLUR_SHARD_BLOCKCACHE_DIRECT_MEMORY_ALLOCATION,
> > >> > true);
> > >> > > -
> > >> > > -      int slabSize = numberOfBlocksPerSlab * blockSize;
> > >> > > -      LOG.info("Number of slabs of block cache [{0}] with direct
> > >> memory
> > >> > allocation set to [{1}]", slabCount,
> > >> > > -          directAllocation);
> > >> > > -      LOG.info("Block cache target memory usage, slab size of
> [{0}]
> > >> > will allocate [{1}] slabs and use ~[{2}] bytes",
> > >> > > -          slabSize, slabCount, ((long) slabCount * (long)
> > slabSize));
> > >> > > -
> > >> > > -      int _1024Size =
> > >> > configuration.getInt("blur.shard.buffercache.1024", 8192);
> > >> > > -      int _8192Size =
> > >> > configuration.getInt("blur.shard.buffercache.8192", 8192);
> > >> > > -      BufferStore.init(_1024Size, _8192Size);
> > >> > > +    BlockCacheDirectoryFactory blockCacheDirectoryFactory;
> > >> > > +    boolean experimentalBlockCache =
> > >> > configuration.getBoolean(BLUR_SHARD_EXPERIMENTAL_BLOCK_CACHE,
> false);
> > >> > > +    if (!experimentalBlockCache) {
> > >> > > +      // setup block cache
> > >> > > +      // 134,217,728 is the slab size, therefore there are 16,384
> > >> blocks
> > >> > > +      // in a slab when using a block size of 8,192
> > >> > > +      int numberOfBlocksPerSlab = 16384;
> > >> > > +      int blockSize = BlockDirectory.BLOCK_SIZE;
> > >> > > +      int slabCount =
> > >> > configuration.getInt(BLUR_SHARD_BLOCKCACHE_SLAB_COUNT, -1);
> > >> > > +      slabCount = getSlabCount(slabCount, numberOfBlocksPerSlab,
> > >> > blockSize);
> > >> > > +      Cache cache;
> > >> > > +      if (slabCount >= 1) {
> > >> > > +        BlockCache blockCache;
> > >> > > +        boolean directAllocation =
> > >> >
> > configuration.getBoolean(BLUR_SHARD_BLOCKCACHE_DIRECT_MEMORY_ALLOCATION,
> > >> > true);
> > >> > > +
> > >> > > +        int slabSize = numberOfBlocksPerSlab * blockSize;
> > >> > > +        LOG.info("Number of slabs of block cache [{0}] with
> direct
> > >> > memory allocation set to [{1}]", slabCount,
> > >> > > +            directAllocation);
> > >> > > +        LOG.info("Block cache target memory usage, slab size of
> > [{0}]
> > >> > will allocate [{1}] slabs and use ~[{2}] bytes",
> > >> > > +            slabSize, slabCount, ((long) slabCount * (long)
> > >> slabSize));
> > >> > > +
> > >> > > +        int _1024Size =
> > >> > configuration.getInt("blur.shard.buffercache.1024", 8192);
> > >> > > +        int _8192Size =
> > >> > configuration.getInt("blur.shard.buffercache.8192", 8192);
> > >> > > +        BufferStore.init(_1024Size, _8192Size);
> > >> > >
> > >> > > -      try {
> > >> > > -        long totalMemory = (long) slabCount * (long)
> > >> > numberOfBlocksPerSlab * (long) blockSize;
> > >> > > -        blockCache = new BlockCache(directAllocation,
> totalMemory,
> > >> > slabSize);
> > >> > > -      } catch (OutOfMemoryError e) {
> > >> > > -        if ("Direct buffer memory".equals(e.getMessage())) {
> > >> > > -          System.err
> > >> > > -              .println("The max direct memory is too low.  Either
> > >> > increase by setting (-XX:MaxDirectMemorySize=<size>g
> > >> -XX:+UseLargePages) or
> > >> > disable direct allocation by
> > >> > (blur.shard.blockcache.direct.memory.allocation=false) in
> > >> > blur-site.properties");
> > >> > > -          System.exit(1);
> > >> > > +        try {
> > >> > > +          long totalMemory = (long) slabCount * (long)
> > >> > numberOfBlocksPerSlab * (long) blockSize;
> > >> > > +          blockCache = new BlockCache(directAllocation,
> > totalMemory,
> > >> > slabSize);
> > >> > > +        } catch (OutOfMemoryError e) {
> > >> > > +          if ("Direct buffer memory".equals(e.getMessage())) {
> > >> > > +            System.err
> > >> > > +                .println("The max direct memory is too low.
>  Either
> > >> > increase by setting (-XX:MaxDirectMemorySize=<size>g
> > >> -XX:+UseLargePages) or
> > >> > disable direct allocation by
> > >> > (blur.shard.blockcache.direct.memory.allocation=false) in
> > >> > blur-site.properties");
> > >> > > +            System.exit(1);
> > >> > > +          }
> > >> > > +          throw e;
> > >> > >          }
> > >> > > -        throw e;
> > >> > > +        cache = new BlockDirectoryCache(blockCache);
> > >> > > +      } else {
> > >> > > +        cache = BlockDirectory.NO_CACHE;
> > >> > >        }
> > >> > > -      cache = new BlockDirectoryCache(blockCache);
> > >> > > +      blockCacheDirectoryFactory = new
> > >> > BlockCacheDirectoryFactoryV1(cache);
> > >> > >      } else {
> > >> > > -      cache = BlockDirectory.NO_CACHE;
> > >> > > +      long totalNumberOfBytes = VM.maxDirectMemory() - _64MB;
> > >> > > +      blockCacheDirectoryFactory = new
> > >> > BlockCacheDirectoryFactoryV2(configuration, totalNumberOfBytes);
> > >> > >      }
> > >> > >
> > >> > >      LOG.info("Shard Server using index [{0}] bind address [{1}]",
> > >> > serverIndex, bindAddress + ":" + bindPort);
> > >> > > @@ -185,7 +202,7 @@ public class ThriftBlurShardServer extends
> > >> > ThriftServer {
> > >> > >      BlurIndexWarmup indexWarmup = getIndexWarmup(configuration);
> > >> > >
> > >> > >      final DistributedIndexServer indexServer = new
> > >> > DistributedIndexServer();
> > >> > > -    indexServer.setCache(cache);
> > >> > > +
> > >> >
> >  indexServer.setBlockCacheDirectoryFactory(blockCacheDirectoryFactory);
> > >> > >      indexServer.setClusterStatus(clusterStatus);
> > >> > >
>  indexServer.setClusterName(configuration.get(BLUR_CLUSTER_NAME,
> > >> > BLUR_CLUSTER));
> > >> > >      indexServer.setConfiguration(config);
> > >> > > @@ -260,33 +277,15 @@ public class ThriftBlurShardServer extends
> > >> > ThriftServer {
> > >> > >    private static int getSlabCount(int slabCount, int
> > >> > numberOfBlocksPerSlab, int blockSize) {
> > >> > >      if (slabCount < 0) {
> > >> > >        long slabSize = numberOfBlocksPerSlab * blockSize;
> > >> > > -      List<String> inputArguments =
> > >> > ManagementFactory.getRuntimeMXBean().getInputArguments();
> > >> > > -      for (String arg : inputArguments) {
> > >> > > -        if (arg.startsWith("-XX:MaxDirectMemorySize")) {
> > >> > > -          long maxDirectMemorySize = getMaxDirectMemorySize(arg);
> > >> > > -          maxDirectMemorySize -= 64 * 1024 * 1024;
> > >> > > -          return (int) (maxDirectMemorySize / slabSize);
> > >> > > -        }
> > >> > > +      long maxDirectMemorySize = VM.maxDirectMemory() - _64MB;
> > >> > > +      if (maxDirectMemorySize < slabSize) {
> > >> > > +        throw new RuntimeException("Auto slab setup cannot
> happen,
> > >> JVM
> > >> > option -XX:MaxDirectMemorySize not set.");
> > >> > >        }
> > >> > > -      throw new RuntimeException("Auto slab setup cannot happen,
> > JVM
> > >> > option -XX:MaxDirectMemorySize not set.");
> > >> > > +      return (int) (maxDirectMemorySize / slabSize);
> > >> > >      }
> > >> > >      return slabCount;
> > >> > >    }
> > >> > >
> > >> > > -  private static long getMaxDirectMemorySize(String arg) {
> > >> > > -    int index = arg.lastIndexOf('=');
> > >> > > -    return parseNumber(arg.substring(index +
> > >> 1).toLowerCase().replace("
> > >> > ", ""));
> > >> > > -  }
> > >> > > -
> > >> > > -  private static long parseNumber(String number) {
> > >> > > -    if (number.endsWith("m")) {
> > >> > > -      return Long.parseLong(number.substring(0, number.length() -
> > >> 1)) *
> > >> > 1024 * 1024;
> > >> > > -    } else if (number.endsWith("g")) {
> > >> > > -      return Long.parseLong(number.substring(0, number.length() -
> > >> 1)) *
> > >> > 1024 * 1024 * 1024;
> > >> > > -    }
> > >> > > -    throw new RuntimeException("Cannot parse [" + number + "]");
> > >> > > -  }
> > >> > > -
> > >> > >    private static BlurFilterCache getFilterCache(BlurConfiguration
> > >> > configuration) {
> > >> > >      String _blurFilterCacheClass =
> > >> > configuration.get(BLUR_SHARD_FILTER_CACHE_CLASS);
> > >> > >      if (_blurFilterCacheClass != null) {
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactory.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactory.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactory.java
> > >> > > new file mode 100644
> > >> > > index 0000000..a8d31ab
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactory.java
> > >> > > @@ -0,0 +1,28 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one or
> > more
> > >> > > + * contributor license agreements.  See the NOTICE file
> distributed
> > >> with
> > >> > > + * this work for additional information regarding copyright
> > >> ownership.
> > >> > > + * The ASF licenses this file to You under the Apache License,
> > >> Version
> > >> > 2.0
> > >> > > + * (the "License"); you may not use this file except in
> compliance
> > >> with
> > >> > > + * the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store;
> > >> > > +
> > >> > > +import java.io.IOException;
> > >> > > +import java.util.Set;
> > >> > > +
> > >> > > +import org.apache.lucene.store.Directory;
> > >> > > +
> > >> > > +public interface BlockCacheDirectoryFactory {
> > >> > > +
> > >> > > +  Directory newDirectory(String name, Directory directory,
> > >> Set<String>
> > >> > blockCacheFileTypes) throws IOException;
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV1.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV1.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV1.java
> > >> > > new file mode 100644
> > >> > > index 0000000..182b494
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV1.java
> > >> > > @@ -0,0 +1,39 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one or
> > more
> > >> > > + * contributor license agreements.  See the NOTICE file
> distributed
> > >> with
> > >> > > + * this work for additional information regarding copyright
> > >> ownership.
> > >> > > + * The ASF licenses this file to You under the Apache License,
> > >> Version
> > >> > 2.0
> > >> > > + * (the "License"); you may not use this file except in
> compliance
> > >> with
> > >> > > + * the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store;
> > >> > > +
> > >> > > +import java.io.IOException;
> > >> > > +import java.util.Set;
> > >> > > +
> > >> > > +import org.apache.blur.store.blockcache.BlockDirectory;
> > >> > > +import org.apache.blur.store.blockcache.Cache;
> > >> > > +import org.apache.lucene.store.Directory;
> > >> > > +
> > >> > > +public class BlockCacheDirectoryFactoryV1 implements
> > >> > BlockCacheDirectoryFactory {
> > >> > > +
> > >> > > +  private final Cache _cache;
> > >> > > +
> > >> > > +  public BlockCacheDirectoryFactoryV1(Cache cache) {
> > >> > > +    _cache = cache;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public Directory newDirectory(String name, Directory directory,
> > >> > Set<String> blockCacheFileTypes) throws IOException {
> > >> > > +    return new BlockDirectory(name, directory, _cache,
> > >> > blockCacheFileTypes);
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV2.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV2.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV2.java
> > >> > > new file mode 100644
> > >> > > index 0000000..0f20900
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/BlockCacheDirectoryFactoryV2.java
> > >> > > @@ -0,0 +1,62 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one or
> > more
> > >> > > + * contributor license agreements.  See the NOTICE file
> distributed
> > >> with
> > >> > > + * this work for additional information regarding copyright
> > >> ownership.
> > >> > > + * The ASF licenses this file to You under the Apache License,
> > >> Version
> > >> > 2.0
> > >> > > + * (the "License"); you may not use this file except in
> compliance
> > >> with
> > >> > > + * the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store;
> > >> > > +
> > >> > > +import java.io.IOException;
> > >> > > +import java.util.Set;
> > >> > > +
> > >> > > +import org.apache.blur.BlurConfiguration;
> > >> > > +import org.apache.blur.store.blockcache_v2.BaseCache;
> > >> > > +import org.apache.blur.store.blockcache_v2.BaseCache.STORE;
> > >> > > +import org.apache.blur.store.blockcache_v2.Cache;
> > >> > > +import org.apache.blur.store.blockcache_v2.CacheDirectory;
> > >> > > +import org.apache.blur.store.blockcache_v2.FileNameBlockSize;
> > >> > > +import org.apache.blur.store.blockcache_v2.FileNameFilter;
> > >> > > +import org.apache.lucene.store.Directory;
> > >> > > +
> > >> > > +public class BlockCacheDirectoryFactoryV2 implements
> > >> > BlockCacheDirectoryFactory {
> > >> > > +
> > >> > > +  private Cache _cache;
> > >> > > +
> > >> > > +  public BlockCacheDirectoryFactoryV2(BlurConfiguration
> > >> configuration,
> > >> > long totalNumberOfBytes) {
> > >> > > +    int fileBufferSize = 8192;
> > >> > > +    FileNameBlockSize fileNameBlockSize = new
> FileNameBlockSize() {
> > >> > > +      @Override
> > >> > > +      public int getBlockSize(String directoryName, String
> > fileName)
> > >> {
> > >> > > +        return 8192;
> > >> > > +      }
> > >> > > +    };
> > >> > > +    FileNameFilter readFilter = new FileNameFilter() {
> > >> > > +      @Override
> > >> > > +      public boolean accept(String directoryName, String
> fileName)
> > {
> > >> > > +        if (fileName.endsWith(".fdt") ||
> > fileName.endsWith(".fdx")) {
> > >> > > +          return true;
> > >> > > +        }
> > >> > > +        return true;
> > >> > > +      }
> > >> > > +    };
> > >> > > +    FileNameFilter writeFilter = readFilter;
> > >> > > +    _cache = new BaseCache(totalNumberOfBytes, fileBufferSize,
> > >> > fileNameBlockSize, readFilter, writeFilter,
> > >> > > +        STORE.OFF_HEAP);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public Directory newDirectory(String name, Directory directory,
> > >> > Set<String> blockCacheFileTypes) throws IOException {
> > >> > > +    return new CacheDirectory(name, directory, _cache);
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/BaseCache.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/BaseCache.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/BaseCache.java
> > >> > > new file mode 100644
> > >> > > index 0000000..1ec2796
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/BaseCache.java
> > >> > > @@ -0,0 +1,210 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2;
> > >> > > +
> > >> > > +import java.io.IOException;
> > >> > > +import java.util.Iterator;
> > >> > > +import java.util.Map;
> > >> > > +import java.util.Map.Entry;
> > >> > > +import java.util.Set;
> > >> > > +import java.util.concurrent.ConcurrentHashMap;
> > >> > > +import java.util.concurrent.atomic.AtomicLong;
> > >> > > +
> > >> > > +import
> > >> > org.apache.blur.store.blockcache_v2.cachevalue.ByteArrayCacheValue;
> > >> > > +import
> > >> org.apache.blur.store.blockcache_v2.cachevalue.UnsafeCacheValue;
> > >> > > +import org.apache.lucene.store.IOContext;
> > >> > > +
> > >> > > +import
> > >> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap;
> > >> > > +import com.googlecode.concurrentlinkedhashmap.EvictionListener;
> > >> > > +import com.googlecode.concurrentlinkedhashmap.Weigher;
> > >> > > +
> > >> > > +public class BaseCache implements Cache {
> > >> > > +  public enum STORE {
> > >> > > +    ON_HEAP, OFF_HEAP
> > >> > > +  }
> > >> > > +
> > >> > > +  class BaseCacheEvictionListener implements
> > >> EvictionListener<CacheKey,
> > >> > CacheValue> {
> > >> > > +    @Override
> > >> > > +    public void onEviction(CacheKey key, CacheValue value) {
> > >> > > +      addToReleaseQueue(value);
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  class BaseCacheWeigher implements Weigher<CacheValue> {
> > >> > > +    @Override
> > >> > > +    public int weightOf(CacheValue value) {
> > >> > > +      return value.size();
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  private final ConcurrentLinkedHashMap<CacheKey, CacheValue>
> > >> _cacheMap;
> > >> > > +  private final int _fileBufferSize;
> > >> > > +  private final FileNameFilter _readFilter;
> > >> > > +  private final FileNameFilter _writeFilter;
> > >> > > +  private final STORE _store;
> > >> > > +  private final FileNameBlockSize _fileNameBlockSize;
> > >> > > +  private final Map<FileIdKey, Long> _fileNameToId = new
> > >> > ConcurrentHashMap<FileIdKey, Long>();
> > >> > > +  private final AtomicLong _fileId = new AtomicLong();
> > >> > > +
> > >> > > +  public BaseCache(long totalNumberOfBytes, int fileBufferSize,
> > >> > FileNameBlockSize fileNameBlockSize,
> > >> > > +      FileNameFilter readFilter, FileNameFilter writeFilter,
> STORE
> > >> > store) {
> > >> > > +    _cacheMap = new ConcurrentLinkedHashMap.Builder<CacheKey,
> > >> > CacheValue>().weigher(new BaseCacheWeigher())
> > >> > > +        .maximumWeightedCapacity(totalNumberOfBytes).listener(new
> > >> > BaseCacheEvictionListener()).build();
> > >> > > +    _fileBufferSize = fileBufferSize;
> > >> > > +    _readFilter = readFilter;
> > >> > > +    _writeFilter = writeFilter;
> > >> > > +    _store = store;
> > >> > > +    _fileNameBlockSize = fileNameBlockSize;
> > >> > > +  }
> > >> > > +
> > >> > > +  private void addToReleaseQueue(CacheValue value) {
> > >> > > +    if (value != null) {
> > >> > > +      if (value.refCount() == 0) {
> > >> > > +        value.release();
> > >> > > +        return;
> > >> > > +      }
> > >> > > +      // @TODO create clean up queue
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public CacheValue newInstance(CacheDirectory directory, String
> > >> > fileName) {
> > >> > > +    switch (_store) {
> > >> > > +    case ON_HEAP:
> > >> > > +      return new ByteArrayCacheValue(getCacheBlockSize(directory,
> > >> > fileName));
> > >> > > +    case OFF_HEAP:
> > >> > > +      return new UnsafeCacheValue(getCacheBlockSize(directory,
> > >> > fileName));
> > >> > > +    default:
> > >> > > +      throw new RuntimeException("Unknown type [" + _store +
> "]");
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public long getFileId(CacheDirectory directory, String
> fileName)
> > >> > throws IOException {
> > >> > > +    FileIdKey cachedFileName = getCacheFileName(directory,
> > fileName);
> > >> > > +    Long id = _fileNameToId.get(cachedFileName);
> > >> > > +    if (id != null) {
> > >> > > +      return id;
> > >> > > +    }
> > >> > > +    long newId = _fileId.incrementAndGet();
> > >> > > +    _fileNameToId.put(cachedFileName, id);
> > >> > > +    return newId;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void removeFile(CacheDirectory directory, String
> fileName)
> > >> > throws IOException {
> > >> > > +    FileIdKey cachedFileName = getCacheFileName(directory,
> > fileName);
> > >> > > +    _fileNameToId.remove(cachedFileName);
> > >> > > +  }
> > >> > > +
> > >> > > +  private FileIdKey getCacheFileName(CacheDirectory directory,
> > String
> > >> > fileName) throws IOException {
> > >> > > +    long fileModified = directory.getFileModified(fileName);
> > >> > > +    return new FileIdKey(directory.getDirectoryName(), fileName,
> > >> > fileModified);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public int getCacheBlockSize(CacheDirectory directory, String
> > >> > fileName) {
> > >> > > +    return
> > >> > _fileNameBlockSize.getBlockSize(directory.getDirectoryName(),
> > fileName);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public int getFileBufferSize(CacheDirectory directory, String
> > >> > fileName) {
> > >> > > +    return _fileBufferSize;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public boolean cacheFileForReading(CacheDirectory directory,
> > String
> > >> > fileName, IOContext context) {
> > >> > > +    return _readFilter.accept(directory.getDirectoryName(),
> > >> fileName);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public boolean cacheFileForWriting(CacheDirectory directory,
> > String
> > >> > fileName, IOContext context) {
> > >> > > +    return _writeFilter.accept(directory.getDirectoryName(),
> > >> fileName);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public CacheValue get(CacheKey key) {
> > >> > > +    return _cacheMap.get(key);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void put(CacheKey key, CacheValue value) {
> > >> > > +    addToReleaseQueue(_cacheMap.put(key, value));
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void releaseDirectory(String directoryName) {
> > >> > > +    Set<Entry<FileIdKey, Long>> entrySet =
> > _fileNameToId.entrySet();
> > >> > > +    Iterator<Entry<FileIdKey, Long>> iterator =
> > entrySet.iterator();
> > >> > > +    while (iterator.hasNext()) {
> > >> > > +      Entry<FileIdKey, Long> entry = iterator.next();
> > >> > > +      FileIdKey fileIdKey = entry.getKey();
> > >> > > +      if (fileIdKey._directoryName.equals(directoryName)) {
> > >> > > +        iterator.remove();
> > >> > > +      }
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  static class FileIdKey {
> > >> > > +    final String _directoryName;
> > >> > > +    final String _fileName;
> > >> > > +    final long _lastModified;
> > >> > > +
> > >> > > +    FileIdKey(String directoryName, String fileName, long
> > >> lastModified)
> > >> > {
> > >> > > +      _directoryName = directoryName;
> > >> > > +      _fileName = fileName;
> > >> > > +      _lastModified = lastModified;
> > >> > > +    }
> > >> > > +
> > >> > > +    @Override
> > >> > > +    public int hashCode() {
> > >> > > +      final int prime = 31;
> > >> > > +      int result = 1;
> > >> > > +      result = prime * result + ((_directoryName == null) ? 0 :
> > >> > _directoryName.hashCode());
> > >> > > +      result = prime * result + ((_fileName == null) ? 0 :
> > >> > _fileName.hashCode());
> > >> > > +      result = prime * result + (int) (_lastModified ^
> > (_lastModified
> > >> > >>> 32));
> > >> > > +      return result;
> > >> > > +    }
> > >> > > +
> > >> > > +    @Override
> > >> > > +    public boolean equals(Object obj) {
> > >> > > +      if (this == obj)
> > >> > > +        return true;
> > >> > > +      if (obj == null)
> > >> > > +        return false;
> > >> > > +      if (getClass() != obj.getClass())
> > >> > > +        return false;
> > >> > > +      FileIdKey other = (FileIdKey) obj;
> > >> > > +      if (_directoryName == null) {
> > >> > > +        if (other._directoryName != null)
> > >> > > +          return false;
> > >> > > +      } else if (!_directoryName.equals(other._directoryName))
> > >> > > +        return false;
> > >> > > +      if (_fileName == null) {
> > >> > > +        if (other._fileName != null)
> > >> > > +          return false;
> > >> > > +      } else if (!_fileName.equals(other._fileName))
> > >> > > +        return false;
> > >> > > +      if (_lastModified != other._lastModified)
> > >> > > +        return false;
> > >> > > +      return true;
> > >> > > +    }
> > >> > > +
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/Cache.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> > a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/Cache.java
> > >> >
> > >>
> > b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/Cache.java
> > >> > > new file mode 100644
> > >> > > index 0000000..66e60f7
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> > b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/Cache.java
> > >> > > @@ -0,0 +1,138 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2;
> > >> > > +
> > >> > > +import java.io.IOException;
> > >> > > +
> > >> > > +import org.apache.lucene.store.IOContext;
> > >> > > +
> > >> > > +public interface Cache {
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Creates a new instance of CacheValue, the cache capacity
> > should
> > >> be
> > >> > used for
> > >> > > +   * the given file.
> > >> > > +   *
> > >> > > +   * @param directory
> > >> > > +   *          the directory.
> > >> > > +   * @param fileName
> > >> > > +   *          the file name.
> > >> > > +   * @return the new CacheValue instance.
> > >> > > +   */
> > >> > > +  CacheValue newInstance(CacheDirectory directory, String
> > fileName);
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Gets unique id for the given file. This is assumed to be
> > unique
> > >> > even if the
> > >> > > +   * file is deleted and recreated.
> > >> > > +   *
> > >> > > +   * @param directory
> > >> > > +   *          the directory.
> > >> > > +   * @param fileName
> > >> > > +   *          the file name.
> > >> > > +   * @return the file id.
> > >> > > +   * @throws IOException
> > >> > > +   */
> > >> > > +  long getFileId(CacheDirectory directory, String fileName)
> throws
> > >> > IOException;
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Get capacity of each cache entry for the given file.
> > >> > > +   *
> > >> > > +   * @param directory
> > >> > > +   *          the directory.
> > >> > > +   * @param fileName
> > >> > > +   *          the file name.
> > >> > > +   * @return the capacity.
> > >> > > +   */
> > >> > > +  int getCacheBlockSize(CacheDirectory directory, String
> fileName);
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Gets buffer size of the buffer used while interacting with
> the
> > >> > underlying
> > >> > > +   * directory.
> > >> > > +   *
> > >> > > +   * @param directory
> > >> > > +   *          the directory.
> > >> > > +   * @param fileName
> > >> > > +   *          the file name.
> > >> > > +   * @return the buffer size.
> > >> > > +   */
> > >> > > +  int getFileBufferSize(CacheDirectory directory, String
> fileName);
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Checks whether file should be cached or not during reading.
> > >> > > +   *
> > >> > > +   * @param directory
> > >> > > +   *          the directory.
> > >> > > +   * @param fileName
> > >> > > +   *          the file name.
> > >> > > +   * @param context
> > >> > > +   *          the IOContext from Lucene.
> > >> > > +   * @return boolean.
> > >> > > +   */
> > >> > > +  boolean cacheFileForReading(CacheDirectory directory, String
> > >> > fileName, IOContext context);
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Checks whether file should be cached or not during writing.
> > >> > > +   *
> > >> > > +   * @param directory
> > >> > > +   *          the directory.
> > >> > > +   * @param fileName
> > >> > > +   *          the file name.
> > >> > > +   * @param context
> > >> > > +   *          the IOContext from Lucene.
> > >> > > +   * @return boolean.
> > >> > > +   */
> > >> > > +  boolean cacheFileForWriting(CacheDirectory directory, String
> > >> > fileName, IOContext context);
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Gets the cache value for the given key. Null if missing.
> > >> > > +   *
> > >> > > +   * @param key
> > >> > > +   *          the key.
> > >> > > +   * @return the cache value or null.
> > >> > > +   */
> > >> > > +  CacheValue get(CacheKey key);
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Puts the cache entry into the cache.
> > >> > > +   *
> > >> > > +   * @param key
> > >> > > +   *          the key.
> > >> > > +   * @param value
> > >> > > +   *          the value.
> > >> > > +   */
> > >> > > +  void put(CacheKey key, CacheValue value);
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Removes the file from the cache.
> > >> > > +   *
> > >> > > +   * @param directory
> > >> > > +   *          the directory.
> > >> > > +   * @param fileName
> > >> > > +   *          the file name.
> > >> > > +   * @throws IOException
> > >> > > +   */
> > >> > > +  void removeFile(CacheDirectory directory, String fileName)
> throws
> > >> > IOException;
> > >> > > +
> > >> > > +  /**
> > >> > > +   * This is called when the CacheDirectory is finalized.
> > >> > > +   *
> > >> > > +   * @param directoryName
> > >> > > +   *          the directory name.
> > >> > > +   */
> > >> > > +  void releaseDirectory(String directoryName);
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheDirectory.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheDirectory.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheDirectory.java
> > >> > > new file mode 100644
> > >> > > index 0000000..73cc147
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheDirectory.java
> > >> > > @@ -0,0 +1,147 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2;
> > >> > > +
> > >> > > +import java.io.IOException;
> > >> > > +import java.util.Collection;
> > >> > > +
> > >> > > +import org.apache.blur.store.blockcache.LastModified;
> > >> > > +import org.apache.blur.store.hdfs.DirectoryDecorator;
> > >> > > +import org.apache.lucene.store.Directory;
> > >> > > +import org.apache.lucene.store.IOContext;
> > >> > > +import org.apache.lucene.store.IndexInput;
> > >> > > +import org.apache.lucene.store.IndexOutput;
> > >> > > +import org.apache.lucene.store.Lock;
> > >> > > +import org.apache.lucene.store.LockFactory;
> > >> > > +
> > >> > > +public class CacheDirectory extends Directory implements
> > >> > DirectoryDecorator, LastModified {
> > >> > > +
> > >> > > +  private final Directory _internal;
> > >> > > +  private final String _directoryName;
> > >> > > +  private final Cache _cache;
> > >> > > +
> > >> > > +  public CacheDirectory(String directoryName, Directory
> directory,
> > >> > Cache cache) {
> > >> > > +    if (!(directory instanceof LastModified)) {
> > >> > > +      throw new RuntimeException("Directory [" + directory + "]
> > does
> > >> > not implement '" + LastModified.class.toString()
> > >> > > +          + "'");
> > >> > > +    }
> > >> > > +    _directoryName = notNull(directoryName);
> > >> > > +    _internal = notNull(directory);
> > >> > > +    _cache = notNull(cache);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  protected void finalize() throws Throwable {
> > >> > > +    _cache.releaseDirectory(getDirectoryName());
> > >> > > +  }
> > >> > > +
> > >> > > +  public IndexInput openInput(String name, IOContext context)
> > throws
> > >> > IOException {
> > >> > > +    IndexInput indexInput = _internal.openInput(name, context);
> > >> > > +    if (_cache.cacheFileForReading(this, name, context)) {
> > >> > > +      return new CacheIndexInput(this, name, indexInput, _cache);
> > >> > > +    }
> > >> > > +    return indexInput;
> > >> > > +  }
> > >> > > +
> > >> > > +  public IndexOutput createOutput(String name, IOContext context)
> > >> > throws IOException {
> > >> > > +    IndexOutput indexOutput = _internal.createOutput(name,
> > context);
> > >> > > +    if (_cache.cacheFileForWriting(this, name, context)) {
> > >> > > +      return new CacheIndexOutput(this, name, indexOutput,
> _cache);
> > >> > > +    }
> > >> > > +    return indexOutput;
> > >> > > +  }
> > >> > > +
> > >> > > +  public void deleteFile(String name) throws IOException {
> > >> > > +    _cache.removeFile(this, name);
> > >> > > +    _internal.deleteFile(name);
> > >> > > +  }
> > >> > > +
> > >> > > +  public String[] listAll() throws IOException {
> > >> > > +    return _internal.listAll();
> > >> > > +  }
> > >> > > +
> > >> > > +  public boolean fileExists(String name) throws IOException {
> > >> > > +    return _internal.fileExists(name);
> > >> > > +  }
> > >> > > +
> > >> > > +  public long fileLength(String name) throws IOException {
> > >> > > +    return _internal.fileLength(name);
> > >> > > +  }
> > >> > > +
> > >> > > +  public void sync(Collection<String> names) throws IOException {
> > >> > > +    _internal.sync(names);
> > >> > > +  }
> > >> > > +
> > >> > > +  public Lock makeLock(String name) {
> > >> > > +    return _internal.makeLock(name);
> > >> > > +  }
> > >> > > +
> > >> > > +  public void clearLock(String name) throws IOException {
> > >> > > +    _internal.clearLock(name);
> > >> > > +  }
> > >> > > +
> > >> > > +  public void close() throws IOException {
> > >> > > +    _internal.close();
> > >> > > +  }
> > >> > > +
> > >> > > +  public void setLockFactory(LockFactory lockFactory) throws
> > >> > IOException {
> > >> > > +    _internal.setLockFactory(lockFactory);
> > >> > > +  }
> > >> > > +
> > >> > > +  public LockFactory getLockFactory() {
> > >> > > +    return _internal.getLockFactory();
> > >> > > +  }
> > >> > > +
> > >> > > +  public String getLockID() {
> > >> > > +    return _internal.getLockID();
> > >> > > +  }
> > >> > > +
> > >> > > +  public String toString() {
> > >> > > +    return _internal.toString();
> > >> > > +  }
> > >> > > +
> > >> > > +  public void copy(Directory to, String src, String dest,
> IOContext
> > >> > context) throws IOException {
> > >> > > +    _internal.copy(to, src, dest, context);
> > >> > > +  }
> > >> > > +
> > >> > > +  public IndexInputSlicer createSlicer(String name, IOContext
> > >> context)
> > >> > throws IOException {
> > >> > > +    return _internal.createSlicer(name, context);
> > >> > > +  }
> > >> > > +
> > >> > > +  public String getDirectoryName() {
> > >> > > +    return _directoryName;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public long getFileModified(String name) throws IOException {
> > >> > > +    return ((LastModified) _internal).getFileModified(name);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public Directory getOriginalDirectory() {
> > >> > > +    return _internal;
> > >> > > +  }
> > >> > > +
> > >> > > +  private static <T> T notNull(T t) {
> > >> > > +    if (t == null) {
> > >> > > +      throw new IllegalArgumentException("Cannot be null");
> > >> > > +    }
> > >> > > +    return t;
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > > \ No newline at end of file
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexInput.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexInput.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexInput.java
> > >> > > new file mode 100644
> > >> > > index 0000000..5279292
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexInput.java
> > >> > > @@ -0,0 +1,171 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2;
> > >> > > +
> > >> > > +import java.io.IOException;
> > >> > > +
> > >> > > +import org.apache.blur.store.buffer.BufferStore;
> > >> > > +import org.apache.lucene.store.IndexInput;
> > >> > > +
> > >> > > +public class CacheIndexInput extends IndexInput {
> > >> > > +
> > >> > > +  private final long _fileLength;
> > >> > > +  private final long _fileId;
> > >> > > +  private final int _cacheBlockSize;
> > >> > > +  private final int _bufferSize;
> > >> > > +  private final CacheDirectory _directory;
> > >> > > +  private final String _fileName;
> > >> > > +  private final IndexInput _indexInput;
> > >> > > +  private final Cache _cache;
> > >> > > +  private final CacheKey _key = new CacheKey();
> > >> > > +
> > >> > > +  private CacheValue _cacheValue;
> > >> > > +  private long _position;
> > >> > > +  private int _blockPosition;
> > >> > > +
> > >> > > +  public CacheIndexInput(CacheDirectory directory, String
> fileName,
> > >> > IndexInput indexInput, Cache cache) throws IOException {
> > >> > > +    super(fileName);
> > >> > > +    _directory = directory;
> > >> > > +    _fileName = fileName;
> > >> > > +    _indexInput = indexInput;
> > >> > > +    _fileLength = indexInput.length();
> > >> > > +    _cache = cache;
> > >> > > +
> > >> > > +    _fileId = _cache.getFileId(_directory, _fileName);
> > >> > > +    _cacheBlockSize = _cache.getCacheBlockSize(_directory,
> > >> _fileName);
> > >> > > +    _bufferSize = _cache.getFileBufferSize(_directory,
> _fileName);
> > >> > > +    _key.setFileId(_fileId);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public byte readByte() throws IOException {
> > >> > > +    tryToFill();
> > >> > > +    byte b = _cacheValue.read(_blockPosition);
> > >> > > +    _position++;
> > >> > > +    _blockPosition++;
> > >> > > +    return b;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void readBytes(byte[] b, int offset, int len) throws
> > >> > IOException {
> > >> > > +    while (len > 0) {
> > >> > > +      tryToFill();
> > >> > > +      int remaining = remaining();
> > >> > > +      int length = Math.min(len, remaining);
> > >> > > +      _cacheValue.read(_blockPosition, b, offset, length);
> > >> > > +      offset += length;
> > >> > > +      len -= length;
> > >> > > +      _position += length;
> > >> > > +      _blockPosition += length;
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  private int remaining() {
> > >> > > +    return _cacheValue.length() - _blockPosition;
> > >> > > +  }
> > >> > > +
> > >> > > +  private void tryToFill() throws IOException {
> > >> > > +    if (_cacheValue == null) {
> > >> > > +      fill();
> > >> > > +    } else if (remaining() == 0) {
> > >> > > +      releaseCache();
> > >> > > +      fill();
> > >> > > +    } else {
> > >> > > +      return;
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  private void releaseCache() {
> > >> > > +    if (_cacheValue != null) {
> > >> > > +      _cacheValue.decRef();
> > >> > > +      _cacheValue = null;
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  private void fill() throws IOException {
> > >> > > +    _key.setBlockId(getBlockId());
> > >> > > +    _cacheValue = _cache.get(_key);
> > >> > > +    if (_cacheValue == null) {
> > >> > > +      _cacheValue = _cache.newInstance(_directory, _fileName);
> > >> > > +      long filePosition = getFilePosition();
> > >> > > +      _indexInput.seek(filePosition);
> > >> > > +      byte[] buffer = BufferStore.takeBuffer(_bufferSize);
> > >> > > +      int len = (int) Math.min(_cacheBlockSize, _fileLength -
> > >> > filePosition);
> > >> > > +      int cachePosition = 0;
> > >> > > +      while (len > 0) {
> > >> > > +        int length = Math.min(_bufferSize, len);
> > >> > > +        _indexInput.readBytes(buffer, 0, length);
> > >> > > +        _cacheValue.write(cachePosition, buffer, 0, length);
> > >> > > +        len -= length;
> > >> > > +        cachePosition += length;
> > >> > > +      }
> > >> > > +    }
> > >> > > +    _cache.put(_key.clone(), _cacheValue);
> > >> > > +    _cacheValue.incRef();
> > >> > > +    _blockPosition = getBlockPosition();
> > >> > > +  }
> > >> > > +
> > >> > > +  private int getBlockPosition() {
> > >> > > +    return (int) (_position % _cacheBlockSize);
> > >> > > +  }
> > >> > > +
> > >> > > +  private long getFilePosition() {
> > >> > > +    // make this a mask...?
> > >> > > +    return getBlockId() * _cacheBlockSize;
> > >> > > +  }
> > >> > > +
> > >> > > +  private long getBlockId(long pos) {
> > >> > > +    return pos / _cacheBlockSize;
> > >> > > +  }
> > >> > > +
> > >> > > +  private long getBlockId() {
> > >> > > +    return _position / _cacheBlockSize;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void close() throws IOException {
> > >> > > +    _indexInput.close();
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public long getFilePointer() {
> > >> > > +    return _position;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void seek(long pos) throws IOException {
> > >> > > +    if (_position >= _fileLength) {
> > >> > > +      throw new IOException("Can not seek past end of file [" +
> > pos +
> > >> > "] filelength [" + _fileLength + "]");
> > >> > > +    }
> > >> > > +    long oldBlockId = getBlockId();
> > >> > > +    _position = pos;
> > >> > > +    long newBlockId = getBlockId(pos);
> > >> > > +    if (newBlockId == oldBlockId) {
> > >> > > +      // need to set new block position
> > >> > > +      _blockPosition = getBlockPosition();
> > >> > > +    } else {
> > >> > > +      releaseCache();
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public long length() {
> > >> > > +    return _fileLength;
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexOutput.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexOutput.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexOutput.java
> > >> > > new file mode 100644
> > >> > > index 0000000..dbb346c
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheIndexOutput.java
> > >> > > @@ -0,0 +1,130 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2;
> > >> > > +
> > >> > > +import java.io.IOException;
> > >> > > +
> > >> > > +import org.apache.blur.store.buffer.BufferStore;
> > >> > > +import org.apache.lucene.store.IndexOutput;
> > >> > > +
> > >> > > +public class CacheIndexOutput extends IndexOutput {
> > >> > > +
> > >> > > +  private final IndexOutput _indexOutput;
> > >> > > +  private final int _fileBufferSize;
> > >> > > +  private final Cache _cache;
> > >> > > +  private final String _fileName;
> > >> > > +  private final CacheDirectory _directory;
> > >> > > +  private final long _fileId;
> > >> > > +
> > >> > > +  private long _position;
> > >> > > +  private byte[] _buffer;
> > >> > > +  private int _bufferPosition;
> > >> > > +  private int _lastWrittenPosition;
> > >> > > +  private int _cacheBlockSize;
> > >> > > +  private CacheValue _cacheValue;
> > >> > > +
> > >> > > +  public CacheIndexOutput(CacheDirectory directory, String
> > fileName,
> > >> > IndexOutput indexOutput, Cache cache) throws IOException {
> > >> > > +    _cache = cache;
> > >> > > +    _directory = directory;
> > >> > > +    _fileName = fileName;
> > >> > > +    _fileBufferSize = _cache.getFileBufferSize(_directory,
> > >> _fileName);
> > >> > > +    _cacheBlockSize = _cache.getCacheBlockSize(_directory,
> > >> _fileName);
> > >> > > +    _fileId = _cache.getFileId(_directory, _fileName);
> > >> > > +    _indexOutput = indexOutput;
> > >> > > +    _buffer = BufferStore.takeBuffer(_cacheBlockSize);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void writeByte(byte b) throws IOException {
> > >> > > +    tryToFlush();
> > >> > > +    _buffer[_bufferPosition] = b;
> > >> > > +    _bufferPosition++;
> > >> > > +    _position++;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void writeBytes(byte[] b, int offset, int len) throws
> > >> > IOException {
> > >> > > +    while (len > 0) {
> > >> > > +      tryToFlush();
> > >> > > +      int remaining = remaining();
> > >> > > +      int length = Math.min(len, remaining);
> > >> > > +      System.arraycopy(b, offset, _buffer, _bufferPosition,
> > length);
> > >> > > +      _bufferPosition += length;
> > >> > > +      _position += length;
> > >> > > +      len -= length;
> > >> > > +      offset += length;
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  private int remaining() {
> > >> > > +    return _fileBufferSize - _bufferPosition;
> > >> > > +  }
> > >> > > +
> > >> > > +  private void tryToFlush() throws IOException {
> > >> > > +    if (remaining() == 0) {
> > >> > > +      flushInternal();
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  private void flushInternal() throws IOException {
> > >> > > +    if (_cacheValue == null) {
> > >> > > +      _cacheValue = _cache.newInstance(_directory, _fileName);
> > >> > > +    }
> > >> > > +    int length = _bufferPosition - _lastWrittenPosition;
> > >> > > +    _indexOutput.writeBytes(_buffer, _lastWrittenPosition,
> length);
> > >> > > +    _cacheValue.write(_bufferPosition, _buffer,
> > _lastWrittenPosition,
> > >> > length);
> > >> > > +    if (_bufferPosition == _fileBufferSize) {
> > >> > > +      _cache.put(new CacheKey(_fileId, getBlockId()),
> _cacheValue);
> > >> > > +      _bufferPosition = 0;
> > >> > > +      _cacheValue = null;
> > >> > > +    }
> > >> > > +    _lastWrittenPosition = _bufferPosition;
> > >> > > +  }
> > >> > > +
> > >> > > +  private long getBlockId() {
> > >> > > +    return _position / _cacheBlockSize;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void seek(long pos) throws IOException {
> > >> > > +    throw new IOException("Seek is not supported.");
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void close() throws IOException {
> > >> > > +    flush();
> > >> > > +    _indexOutput.close();
> > >> > > +    BufferStore.putBuffer(_buffer);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void flush() throws IOException {
> > >> > > +    flushInternal();
> > >> > > +    _indexOutput.flush();
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public long getFilePointer() {
> > >> > > +    return _position;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public long length() throws IOException {
> > >> > > +    return getFilePointer();
> > >> > > +  }
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheKey.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheKey.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheKey.java
> > >> > > new file mode 100644
> > >> > > index 0000000..340e488
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheKey.java
> > >> > > @@ -0,0 +1,89 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2;
> > >> > > +
> > >> > > +public class CacheKey implements Cloneable {
> > >> > > +
> > >> > > +  private long _fileId;
> > >> > > +  private long _blockId;
> > >> > > +
> > >> > > +  public CacheKey() {
> > >> > > +
> > >> > > +  }
> > >> > > +
> > >> > > +  public CacheKey(long fileId, long blockId) {
> > >> > > +    _fileId = fileId;
> > >> > > +    _blockId = blockId;
> > >> > > +  }
> > >> > > +
> > >> > > +  public long getFileId() {
> > >> > > +    return _fileId;
> > >> > > +  }
> > >> > > +
> > >> > > +  public void setFileId(long fileId) {
> > >> > > +    _fileId = fileId;
> > >> > > +  }
> > >> > > +
> > >> > > +  public long getBlockId() {
> > >> > > +    return _blockId;
> > >> > > +  }
> > >> > > +
> > >> > > +  public void setBlockId(long blockId) {
> > >> > > +    _blockId = blockId;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public CacheKey clone() {
> > >> > > +    try {
> > >> > > +      return (CacheKey) super.clone();
> > >> > > +    } catch (CloneNotSupportedException e) {
> > >> > > +      throw new RuntimeException(e);
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public int hashCode() {
> > >> > > +    final int prime = 31;
> > >> > > +    int result = 1;
> > >> > > +    result = prime * result + (int) (_blockId ^ (_blockId >>>
> 32));
> > >> > > +    result = prime * result + (int) (_fileId ^ (_fileId >>> 32));
> > >> > > +    return result;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public boolean equals(Object obj) {
> > >> > > +    if (this == obj)
> > >> > > +      return true;
> > >> > > +    if (obj == null)
> > >> > > +      return false;
> > >> > > +    if (getClass() != obj.getClass())
> > >> > > +      return false;
> > >> > > +    CacheKey other = (CacheKey) obj;
> > >> > > +    if (_blockId != other._blockId)
> > >> > > +      return false;
> > >> > > +    if (_fileId != other._fileId)
> > >> > > +      return false;
> > >> > > +    return true;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public String toString() {
> > >> > > +    return "CacheKey [_fileId=" + _fileId + ", _blockId=" +
> > _blockId
> > >> +
> > >> > "]";
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheValue.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheValue.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheValue.java
> > >> > > new file mode 100644
> > >> > > index 0000000..0f7b976
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/CacheValue.java
> > >> > > @@ -0,0 +1,93 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2;
> > >> > > +
> > >> > > +public interface CacheValue {
> > >> > > +
> > >> > > +  /**
> > >> > > +   * The actual size of the the underlying resource.
> > >> > > +   *
> > >> > > +   * @return the size.
> > >> > > +   */
> > >> > > +  int size();
> > >> > > +
> > >> > > +  /**
> > >> > > +   * The length of the data in this block.
> > >> > > +   *
> > >> > > +   * @return the length.
> > >> > > +   */
> > >> > > +  int length();
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Writes data out to a given position in this block.
> > >> > > +   *
> > >> > > +   * @param position
> > >> > > +   *          the position.
> > >> > > +   * @param buf
> > >> > > +   *          the buffer.
> > >> > > +   * @param offset
> > >> > > +   *          the offset in the buffer.
> > >> > > +   * @param length
> > >> > > +   *          the length of bytes to write.
> > >> > > +   */
> > >> > > +  void write(int position, byte[] buf, int offset, int length);
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Reads data into the buffer given the position.
> > >> > > +   *
> > >> > > +   * @param position
> > >> > > +   *          the position to read.
> > >> > > +   * @param buf
> > >> > > +   *          the buffer to read into.
> > >> > > +   * @param offset
> > >> > > +   *          the offset within the buffer.
> > >> > > +   * @param length
> > >> > > +   *          the length of data to read.
> > >> > > +   */
> > >> > > +  void read(int position, byte[] buf, int offset, int length);
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Reads a byte from the given position.
> > >> > > +   *
> > >> > > +   * @param position
> > >> > > +   *          the position.
> > >> > > +   * @return the byte.
> > >> > > +   */
> > >> > > +  byte read(int position);
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Increments the reference.
> > >> > > +   */
> > >> > > +  void incRef();
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Decrements the reference.
> > >> > > +   */
> > >> > > +  void decRef();
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Gets the reference count.
> > >> > > +   */
> > >> > > +  long refCount();
> > >> > > +
> > >> > > +  /**
> > >> > > +   * Releases any underlying resources.
> > >> > > +   */
> > >> > > +  void release();
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameBlockSize.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameBlockSize.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameBlockSize.java
> > >> > > new file mode 100644
> > >> > > index 0000000..8890459
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameBlockSize.java
> > >> > > @@ -0,0 +1,24 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2;
> > >> > > +
> > >> > > +public interface FileNameBlockSize {
> > >> > > +
> > >> > > +  int getBlockSize(String directoryName, String fileName);
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameFilter.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameFilter.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameFilter.java
> > >> > > new file mode 100644
> > >> > > index 0000000..03a969c
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/FileNameFilter.java
> > >> > > @@ -0,0 +1,24 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2;
> > >> > > +
> > >> > > +public interface FileNameFilter {
> > >> > > +
> > >> > > +  boolean accept(String directoryName, String fileName);
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/BaseCacheValue.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/BaseCacheValue.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/BaseCacheValue.java
> > >> > > new file mode 100644
> > >> > > index 0000000..a5e10a3
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/BaseCacheValue.java
> > >> > > @@ -0,0 +1,83 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2.cachevalue;
> > >> > > +
> > >> > > +import java.util.concurrent.atomic.AtomicLong;
> > >> > > +
> > >> > > +import org.apache.blur.store.blockcache_v2.CacheValue;
> > >> > > +
> > >> > > +@SuppressWarnings("serial")
> > >> > > +public abstract class BaseCacheValue extends AtomicLong
> implements
> > >> > CacheValue {
> > >> > > +
> > >> > > +  private final int _length;
> > >> > > +
> > >> > > +  public BaseCacheValue(int length) {
> > >> > > +    _length = length;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public final int length() {
> > >> > > +    return _length;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void write(int position, byte[] buf, int offset, int
> > >> length) {
> > >> > > +    if (position + length > _length) {
> > >> > > +      throw new ArrayIndexOutOfBoundsException(position +
> length);
> > >> > > +    }
> > >> > > +    writeInternal(position, buf, offset, length);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void read(int position, byte[] buf, int offset, int
> > length)
> > >> {
> > >> > > +    if (position + length > _length) {
> > >> > > +      throw new ArrayIndexOutOfBoundsException(position +
> length);
> > >> > > +    }
> > >> > > +    readInternal(position, buf, offset, length);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public byte read(int position) {
> > >> > > +    if (position >= _length) {
> > >> > > +      throw new ArrayIndexOutOfBoundsException(position);
> > >> > > +    }
> > >> > > +    return readInternal(position);
> > >> > > +  }
> > >> > > +
> > >> > > +  protected abstract void writeInternal(int position, byte[] buf,
> > int
> > >> > offset, int length);
> > >> > > +
> > >> > > +  protected abstract byte readInternal(int position);
> > >> > > +
> > >> > > +  protected abstract void readInternal(int position, byte[] buf,
> > int
> > >> > offset, int length);
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public final void incRef() {
> > >> > > +    incrementAndGet();
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public final void decRef() {
> > >> > > +    decrementAndGet();
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public final long refCount() {
> > >> > > +    return get();
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValue.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValue.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValue.java
> > >> > > new file mode 100644
> > >> > > index 0000000..c4d84bf
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValue.java
> > >> > > @@ -0,0 +1,55 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2.cachevalue;
> > >> > > +
> > >> > > +@SuppressWarnings("serial")
> > >> > > +public class ByteArrayCacheValue extends BaseCacheValue {
> > >> > > +
> > >> > > +  private final byte[] _buffer;
> > >> > > +
> > >> > > +  public ByteArrayCacheValue(int length) {
> > >> > > +    super(length);
> > >> > > +    _buffer = new byte[length];
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  protected void writeInternal(int position, byte[] buf, int
> > offset,
> > >> > int length) {
> > >> > > +    System.arraycopy(buf, offset, _buffer, position, length);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  protected void readInternal(int position, byte[] buf, int
> offset,
> > >> int
> > >> > length) {
> > >> > > +    System.arraycopy(_buffer, position, buf, offset, length);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  protected byte readInternal(int position) {
> > >> > > +    return _buffer[position];
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void release() {
> > >> > > +    // no op
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public int size() {
> > >> > > +    return length();
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValue.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValue.java
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValue.java
> > >> > > new file mode 100644
> > >> > > index 0000000..a918250
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValue.java
> > >> > > @@ -0,0 +1,99 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2.cachevalue;
> > >> > > +
> > >> > > +import java.lang.reflect.Field;
> > >> > > +
> > >> > > +import sun.misc.Unsafe;
> > >> > > +
> > >> > > +@SuppressWarnings("serial")
> > >> > > +public class UnsafeCacheValue extends BaseCacheValue {
> > >> > > +
> > >> > > +  private static final int MINIMUM_SIZE = 1024;
> > >> > > +
> > >> > > +  private static final Unsafe _unsafe;
> > >> > > +
> > >> > > +  static {
> > >> > > +    try {
> > >> > > +      Class<?> clazz = Class.forName("java.nio.Bits");
> > >> > > +      Field field = clazz.getDeclaredField("unsafe");
> > >> > > +      field.setAccessible(true);
> > >> > > +      _unsafe = (Unsafe) field.get(null);
> > >> > > +    } catch (Exception e) {
> > >> > > +      throw new RuntimeException(e);
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  private static final int BYTE_ARRAY_BASE_OFFSET =
> > >> > _unsafe.arrayBaseOffset(byte[].class);
> > >> > > +
> > >> > > +  private static void copyFromArray(byte[] src, int srcOffset,
> int
> > >> > length, long destAddress) {
> > >> > > +    long offset = BYTE_ARRAY_BASE_OFFSET + srcOffset;
> > >> > > +    _unsafe.copyMemory(src, offset, null, destAddress, length);
> > >> > > +  }
> > >> > > +
> > >> > > +  private static void copyToArray(long srcAddress, byte[] dst,
> int
> > >> > dstOffset, int length) {
> > >> > > +    long offset = BYTE_ARRAY_BASE_OFFSET + dstOffset;
> > >> > > +    _unsafe.copyMemory(null, srcAddress, dst, offset, length);
> > >> > > +  }
> > >> > > +
> > >> > > +  private final long _address;
> > >> > > +  private final int _capacity;
> > >> > > +
> > >> > > +  public UnsafeCacheValue(int length) {
> > >> > > +    super(length);
> > >> > > +    _capacity = getCapacity(length);
> > >> > > +    _address = _unsafe.allocateMemory(_capacity);
> > >> > > +  }
> > >> > > +
> > >> > > +  private int getCapacity(int length) {
> > >> > > +    if (length < MINIMUM_SIZE) {
> > >> > > +      return MINIMUM_SIZE;
> > >> > > +    }
> > >> > > +    return length;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  protected void writeInternal(int position, byte[] buf, int
> > offset,
> > >> > int length) {
> > >> > > +    copyFromArray(buf, offset, length, resolveAddress(position));
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  protected void readInternal(int position, byte[] buf, int
> offset,
> > >> int
> > >> > length) {
> > >> > > +    copyToArray(resolveAddress(position), buf, offset, length);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  protected byte readInternal(int position) {
> > >> > > +    return _unsafe.getByte(resolveAddress(position));
> > >> > > +  }
> > >> > > +
> > >> > > +  private long resolveAddress(int position) {
> > >> > > +    return _address + position;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public void release() {
> > >> > > +    _unsafe.freeMemory(_address);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Override
> > >> > > +  public int size() {
> > >> > > +    return _capacity;
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexInputTest.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexInputTest.java
> > >> >
> > >>
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexInputTest.java
> > >> > > new file mode 100644
> > >> > > index 0000000..6e897bc
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexInputTest.java
> > >> > > @@ -0,0 +1,206 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2;
> > >> > > +
> > >> > > +import static org.junit.Assert.assertArrayEquals;
> > >> > > +import static org.junit.Assert.assertEquals;
> > >> > > +import static org.junit.Assert.fail;
> > >> > > +
> > >> > > +import java.io.IOException;
> > >> > > +import java.util.Random;
> > >> > > +
> > >> > > +import
> > >> > org.apache.blur.store.blockcache_v2.cachevalue.ByteArrayCacheValue;
> > >> > > +import org.apache.blur.store.buffer.BufferStore;
> > >> > > +import org.apache.lucene.store.IOContext;
> > >> > > +import org.apache.lucene.store.IndexInput;
> > >> > > +import org.apache.lucene.store.IndexOutput;
> > >> > > +import org.apache.lucene.store.RAMDirectory;
> > >> > > +import org.junit.Before;
> > >> > > +import org.junit.Test;
> > >> > > +
> > >> > > +import
> > >> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap;
> > >> > > +import com.googlecode.concurrentlinkedhashmap.EvictionListener;
> > >> > > +import com.googlecode.concurrentlinkedhashmap.Weigher;
> > >> > > +
> > >> > > +public class CacheIndexInputTest {
> > >> > > +
> > >> > > +  private long seed;
> > >> > > +
> > >> > > +  private final int sampleSize = 10000;
> > >> > > +  private final int maxBufSize = 10000;
> > >> > > +  private final int maxOffset = 1000;
> > >> > > +
> > >> > > +  @Before
> > >> > > +  public void setup() {
> > >> > > +    BufferStore.init(128, 128);
> > >> > > +    seed = new Random().nextLong();
> > >> > > +    System.out.println("Using seed [" + seed + "]");
> > >> > > +    // seed = -265282183286396219l;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test1() throws IOException {
> > >> > > +    RAMDirectory directory = new RAMDirectory();
> > >> > > +
> > >> > > +    String name = "test";
> > >> > > +
> > >> > > +    IndexOutput output = directory.createOutput(name,
> > >> > IOContext.DEFAULT);
> > >> > > +    byte[] bs = "hello world".getBytes();
> > >> > > +    output.writeBytes(bs, bs.length);
> > >> > > +    output.close();
> > >> > > +
> > >> > > +    IndexInput input = directory.openInput(name,
> > IOContext.DEFAULT);
> > >> > > +    Cache cache = getCache();
> > >> > > +    CacheIndexInput cacheInput = new CacheIndexInput(null, name,
> > >> input,
> > >> > cache);
> > >> > > +    byte[] buf = new byte[bs.length];
> > >> > > +    cacheInput.readBytes(buf, 0, buf.length);
> > >> > > +    cacheInput.close();
> > >> > > +
> > >> > > +    assertArrayEquals(bs, buf);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test2() throws IOException {
> > >> > > +    Cache cache = getCache();
> > >> > > +    RAMDirectory directory = new RAMDirectory();
> > >> > > +    Random random = new Random(seed);
> > >> > > +
> > >> > > +    String name = "test2";
> > >> > > +    long size = (10 * 1024 * 1024) + 13;
> > >> > > +
> > >> > > +    IndexOutput output = directory.createOutput(name,
> > >> > IOContext.DEFAULT);
> > >> > > +    writeRandomData(size, random, output);
> > >> > > +    output.close();
> > >> > > +
> > >> > > +    IndexInput input = directory.openInput(name,
> > IOContext.DEFAULT);
> > >> > > +    IndexInput testInput = new CacheIndexInput(null, name,
> > >> > input.clone(), cache);
> > >> > > +    readRandomData(input, testInput, random, sampleSize,
> > maxBufSize,
> > >> > maxOffset);
> > >> > > +    testInput.close();
> > >> > > +    input.close();
> > >> > > +  }
> > >> > > +
> > >> > > +  public static void readRandomData(IndexInput baseInput,
> > IndexInput
> > >> > testInput, Random random, int sampleSize,
> > >> > > +      int maxBufSize, int maxOffset) throws IOException {
> > >> > > +    assertEquals(baseInput.length(), testInput.length());
> > >> > > +    int fileLength = (int) baseInput.length();
> > >> > > +    for (int i = 0; i < sampleSize; i++) {
> > >> > > +      int position = random.nextInt(fileLength - maxBufSize);
> > >> > > +      int bufSize = random.nextInt(maxBufSize - maxOffset) + 1;
> > >> > > +      byte[] buf1 = new byte[bufSize];
> > >> > > +      byte[] buf2 = new byte[bufSize];
> > >> > > +
> > >> > > +      int offset = random.nextInt(Math.min(maxOffset, bufSize));
> > >> > > +      int len = Math.min(random.nextInt(bufSize - offset),
> > >> fileLength -
> > >> > position);
> > >> > > +
> > >> > > +      baseInput.seek(position);
> > >> > > +      baseInput.readBytes(buf1, offset, len);
> > >> > > +      testInput.seek(position);
> > >> > > +      testInput.readBytes(buf2, offset, len);
> > >> > > +      assertArrayEquals("Read [" + i + "] The position is [" +
> > >> position
> > >> > + "] and bufSize [" + bufSize + "]", buf1, buf2);
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  public static void writeRandomData(long size, Random random,
> > >> > IndexOutput... outputs) throws IOException {
> > >> > > +    byte[] buf = new byte[1024];
> > >> > > +    for (long l = 0; l < size; l += buf.length) {
> > >> > > +      random.nextBytes(buf);
> > >> > > +      int length = (int) Math.min(buf.length, size - l);
> > >> > > +      for (IndexOutput output : outputs) {
> > >> > > +        output.writeBytes(buf, length);
> > >> > > +      }
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +
> > >> > > +  public static Cache getCache() {
> > >> > > +    EvictionListener<CacheKey, CacheValue> listener = new
> > >> > EvictionListener<CacheKey, CacheValue>() {
> > >> > > +      @Override
> > >> > > +      public void onEviction(CacheKey key, CacheValue value) {
> > >> > > +        if (value.refCount() == 0) {
> > >> > > +          value.release();
> > >> > > +        } else {
> > >> > > +          // doing something else...
> > >> > > +          fail();
> > >> > > +        }
> > >> > > +      }
> > >> > > +    };
> > >> > > +    Weigher<CacheValue> weigher = new Weigher<CacheValue>() {
> > >> > > +      @Override
> > >> > > +      public int weightOf(CacheValue value) {
> > >> > > +        return value.length();
> > >> > > +      }
> > >> > > +    };
> > >> > > +    long maximumWeightedCapacity = 1 * 1024 * 1024;
> > >> > > +    final ConcurrentLinkedHashMap<CacheKey, CacheValue> cache =
> new
> > >> > ConcurrentLinkedHashMap.Builder<CacheKey, CacheValue>()
> > >> > > +
> > >> >
> > >>
> >
>  .weigher(weigher).maximumWeightedCapacity(maximumWeightedCapacity).listener(listener).build();
> > >> > > +    Cache cacheFactory = new Cache() {
> > >> > > +
> > >> > > +      @Override
> > >> > > +      public CacheValue newInstance(CacheDirectory directory,
> > String
> > >> > fileName) {
> > >> > > +        return new
> ByteArrayCacheValue(getCacheBlockSize(directory,
> > >> > fileName));
> > >> > > +      }
> > >> > > +
> > >> > > +      @Override
> > >> > > +      public long getFileId(CacheDirectory directory, String
> > >> fileName) {
> > >> > > +        return fileName.hashCode();
> > >> > > +      }
> > >> > > +
> > >> > > +      @Override
> > >> > > +      public int getFileBufferSize(CacheDirectory directory,
> String
> > >> > fileName) {
> > >> > > +        return 1024;
> > >> > > +      }
> > >> > > +
> > >> > > +      @Override
> > >> > > +      public int getCacheBlockSize(CacheDirectory directory,
> String
> > >> > fileName) {
> > >> > > +        return 8192;
> > >> > > +      }
> > >> > > +
> > >> > > +      @Override
> > >> > > +      public boolean cacheFileForReading(CacheDirectory
> directory,
> > >> > String name, IOContext context) {
> > >> > > +        return true;
> > >> > > +      }
> > >> > > +
> > >> > > +      @Override
> > >> > > +      public boolean cacheFileForWriting(CacheDirectory
> directory,
> > >> > String name, IOContext context) {
> > >> > > +        return true;
> > >> > > +      }
> > >> > > +
> > >> > > +      @Override
> > >> > > +      public CacheValue get(CacheKey key) {
> > >> > > +        return cache.get(key);
> > >> > > +      }
> > >> > > +
> > >> > > +      @Override
> > >> > > +      public void put(CacheKey key, CacheValue value) {
> > >> > > +        cache.put(key, value);
> > >> > > +      }
> > >> > > +
> > >> > > +      @Override
> > >> > > +      public void removeFile(CacheDirectory directory, String
> > >> fileName)
> > >> > throws IOException {
> > >> > > +
> > >> > > +      }
> > >> > > +
> > >> > > +      @Override
> > >> > > +      public void releaseDirectory(String directoryName) {
> > >> > > +
> > >> > > +      }
> > >> > > +    };
> > >> > > +    return cacheFactory;
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexOutputTest.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexOutputTest.java
> > >> >
> > >>
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexOutputTest.java
> > >> > > new file mode 100644
> > >> > > index 0000000..2dc8b5b
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/CacheIndexOutputTest.java
> > >> > > @@ -0,0 +1,101 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2;
> > >> > > +
> > >> > > +import static org.junit.Assert.assertArrayEquals;
> > >> > > +import static org.junit.Assert.assertEquals;
> > >> > > +
> > >> > > +import java.io.IOException;
> > >> > > +import java.util.Random;
> > >> > > +
> > >> > > +import org.apache.blur.store.buffer.BufferStore;
> > >> > > +import org.apache.lucene.store.IOContext;
> > >> > > +import org.apache.lucene.store.IndexInput;
> > >> > > +import org.apache.lucene.store.IndexOutput;
> > >> > > +import org.apache.lucene.store.RAMDirectory;
> > >> > > +import org.junit.Before;
> > >> > > +import org.junit.Test;
> > >> > > +
> > >> > > +public class CacheIndexOutputTest {
> > >> > > +
> > >> > > +  private long seed;
> > >> > > +
> > >> > > +  private final int sampleSize = 10000;
> > >> > > +  private final int maxBufSize = 10000;
> > >> > > +  private final int maxOffset = 1000;
> > >> > > +
> > >> > > +  @Before
> > >> > > +  public void setup() {
> > >> > > +    BufferStore.init(128, 128);
> > >> > > +    seed = new Random().nextLong();
> > >> > > +    System.out.println("Using seed [" + seed + "]");
> > >> > > +    // seed = -265282183286396219l;
> > >> > > +  }
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test1() throws IOException {
> > >> > > +    Random random = new Random(seed);
> > >> > > +    RAMDirectory directory = new RAMDirectory();
> > >> > > +    IndexOutput output = directory.createOutput("test",
> > >> > IOContext.DEFAULT);
> > >> > > +
> > >> > > +    Cache cache = CacheIndexInputTest.getCache();
> > >> > > +    CacheIndexOutput indexOutput = new CacheIndexOutput(null,
> > "test",
> > >> > output, cache);
> > >> > > +    indexOutput.writeByte((byte) 1);
> > >> > > +    indexOutput.writeByte((byte) 2);
> > >> > > +    byte[] b = new byte[16000];
> > >> > > +    random.nextBytes(b);
> > >> > > +    indexOutput.writeBytes(b, 16000);
> > >> > > +    indexOutput.close();
> > >> > > +
> > >> > > +    IndexInput input = directory.openInput("test",
> > >> IOContext.DEFAULT);
> > >> > > +    assertEquals(16002, input.length());
> > >> > > +    assertEquals(1, input.readByte());
> > >> > > +    assertEquals(2, input.readByte());
> > >> > > +
> > >> > > +    byte[] buf = new byte[16000];
> > >> > > +    input.readBytes(buf, 0, 16000);
> > >> > > +    input.close();
> > >> > > +    assertArrayEquals(b, buf);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test2() throws IOException {
> > >> > > +    Cache cache = CacheIndexInputTest.getCache();
> > >> > > +    RAMDirectory directory = new RAMDirectory();
> > >> > > +    RAMDirectory directory2 = new RAMDirectory();
> > >> > > +
> > >> > > +    Random random = new Random(seed);
> > >> > > +
> > >> > > +    String name = "test2";
> > >> > > +    long size = (10 * 1024 * 1024) + 13;
> > >> > > +
> > >> > > +    IndexOutput output = directory.createOutput(name,
> > >> > IOContext.DEFAULT);
> > >> > > +    IndexOutput output2 = directory2.createOutput(name,
> > >> > IOContext.DEFAULT);
> > >> > > +    CacheIndexOutput cacheIndexOutput = new
> CacheIndexOutput(null,
> > >> > name, output2, cache);
> > >> > > +    CacheIndexInputTest.writeRandomData(size, random, output,
> > >> > cacheIndexOutput);
> > >> > > +    output.close();
> > >> > > +    cacheIndexOutput.close();
> > >> > > +
> > >> > > +    IndexInput input = directory.openInput(name,
> > IOContext.DEFAULT);
> > >> > > +    IndexInput testInput = directory2.openInput(name,
> > >> > IOContext.DEFAULT);
> > >> > > +    CacheIndexInputTest.readRandomData(input, testInput, random,
> > >> > sampleSize, maxBufSize, maxOffset);
> > >> > > +    testInput.close();
> > >> > > +    input.close();
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValueTest.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValueTest.java
> > >> >
> > >>
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValueTest.java
> > >> > > new file mode 100644
> > >> > > index 0000000..8d33ece
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/ByteArrayCacheValueTest.java
> > >> > > @@ -0,0 +1,68 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2.cachevalue;
> > >> > > +
> > >> > > +import static org.junit.Assert.*;
> > >> > > +
> > >> > > +import org.junit.Test;
> > >> > > +
> > >> > > +public class ByteArrayCacheValueTest {
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test1() {
> > >> > > +    ByteArrayCacheValue value = new ByteArrayCacheValue(10);
> > >> > > +    byte[] buf = "hello world".getBytes();
> > >> > > +    value.write(0, buf, 0, 10);
> > >> > > +    byte[] buf2 = new byte[10];
> > >> > > +    value.read(0, buf2, 0, 10);
> > >> > > +    assertArrayEquals("hello worl".getBytes(), buf2);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test2() {
> > >> > > +    ByteArrayCacheValue value = new ByteArrayCacheValue(10);
> > >> > > +    byte[] buf = "hello world".getBytes();
> > >> > > +    try {
> > >> > > +      value.write(0, buf, 0, 11);
> > >> > > +      fail();
> > >> > > +    } catch (ArrayIndexOutOfBoundsException e) {
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test3() {
> > >> > > +    ByteArrayCacheValue value = new ByteArrayCacheValue(10);
> > >> > > +    byte[] buf = "hello world".getBytes();
> > >> > > +    try {
> > >> > > +      value.write(8, buf, 0, 3);
> > >> > > +      fail();
> > >> > > +    } catch (ArrayIndexOutOfBoundsException e) {
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test4() {
> > >> > > +    ByteArrayCacheValue value = new ByteArrayCacheValue(10);
> > >> > > +    byte[] buf = "hello world".getBytes();
> > >> > > +    value.write(8, buf, 0, 2);
> > >> > > +
> > >> > > +    assertEquals('h', value.read(8));
> > >> > > +    assertEquals('e', value.read(9));
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValueTest.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> >
> > >>
> >
> a/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValueTest.java
> > >> >
> > >>
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValueTest.java
> > >> > > new file mode 100644
> > >> > > index 0000000..37f5926
> > >> > > --- /dev/null
> > >> > > +++
> > >> >
> > >>
> >
> b/blur-store/src/test/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValueTest.java
> > >> > > @@ -0,0 +1,68 @@
> > >> > > +/**
> > >> > > + * Licensed to the Apache Software Foundation (ASF) under one
> > >> > > + * or more contributor license agreements.  See the NOTICE file
> > >> > > + * distributed with this work for additional information
> > >> > > + * regarding copyright ownership.  The ASF licenses this file
> > >> > > + * to you under the Apache License, Version 2.0 (the
> > >> > > + * "License"); you may not use this file except in compliance
> > >> > > + * with the License.  You may obtain a copy of the License at
> > >> > > + *
> > >> > > + *     http://www.apache.org/licenses/LICENSE-2.0
> > >> > > + *
> > >> > > + * Unless required by applicable law or agreed to in writing,
> > >> software
> > >> > > + * distributed under the License is distributed on an "AS IS"
> > BASIS,
> > >> > > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > >> > implied.
> > >> > > + * See the License for the specific language governing
> permissions
> > >> and
> > >> > > + * limitations under the License.
> > >> > > + */
> > >> > > +package org.apache.blur.store.blockcache_v2.cachevalue;
> > >> > > +
> > >> > > +import static org.junit.Assert.*;
> > >> > > +
> > >> > > +import org.junit.Test;
> > >> > > +
> > >> > > +public class UnsafeCacheValueTest {
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test1() {
> > >> > > +    UnsafeCacheValue value = new UnsafeCacheValue(10);
> > >> > > +    byte[] buf = "hello world".getBytes();
> > >> > > +    value.write(0, buf, 0, 10);
> > >> > > +    byte[] buf2 = new byte[10];
> > >> > > +    value.read(0, buf2, 0, 10);
> > >> > > +    assertArrayEquals("hello worl".getBytes(), buf2);
> > >> > > +  }
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test2() {
> > >> > > +    UnsafeCacheValue value = new UnsafeCacheValue(10);
> > >> > > +    byte[] buf = "hello world".getBytes();
> > >> > > +    try {
> > >> > > +      value.write(0, buf, 0, 11);
> > >> > > +      fail();
> > >> > > +    } catch (ArrayIndexOutOfBoundsException e) {
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test3() {
> > >> > > +    UnsafeCacheValue value = new UnsafeCacheValue(10);
> > >> > > +    byte[] buf = "hello world".getBytes();
> > >> > > +    try {
> > >> > > +      value.write(8, buf, 0, 3);
> > >> > > +      fail();
> > >> > > +    } catch (ArrayIndexOutOfBoundsException e) {
> > >> > > +    }
> > >> > > +  }
> > >> > > +
> > >> > > +  @Test
> > >> > > +  public void test4() {
> > >> > > +    UnsafeCacheValue value = new UnsafeCacheValue(10);
> > >> > > +    byte[] buf = "hello world".getBytes();
> > >> > > +    value.write(8, buf, 0, 2);
> > >> > > +
> > >> > > +    assertEquals('h', value.read(8));
> > >> > > +    assertEquals('e', value.read(9));
> > >> > > +  }
> > >> > > +
> > >> > > +}
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-util/src/main/java/org/apache/blur/utils/BlurConstants.java
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git
> > >> > a/blur-util/src/main/java/org/apache/blur/utils/BlurConstants.java
> > >> > b/blur-util/src/main/java/org/apache/blur/utils/BlurConstants.java
> > >> > > index 60368f8..fbe86d0 100644
> > >> > > ---
> > a/blur-util/src/main/java/org/apache/blur/utils/BlurConstants.java
> > >> > > +++
> > b/blur-util/src/main/java/org/apache/blur/utils/BlurConstants.java
> > >> > > @@ -44,6 +44,7 @@ public class BlurConstants {
> > >> > >    public static final String BLUR_SHARD_BIND_ADDRESS =
> > >> > "blur.shard.bind.address";
> > >> > >    public static final String
> > >> > BLUR_SHARD_BLOCKCACHE_DIRECT_MEMORY_ALLOCATION =
> > >> > "blur.shard.blockcache.direct.memory.allocation";
> > >> > >    public static final String BLUR_SHARD_BLOCKCACHE_SLAB_COUNT =
> > >> > "blur.shard.blockcache.slab.count";
> > >> > > +  public static final String BLUR_SHARD_EXPERIMENTAL_BLOCK_CACHE
> =
> > >> > "blur.shard.experimental.block.cache";
> > >> > >    public static final String BLUR_SHARD_SAFEMODEDELAY =
> > >> > "blur.shard.safemodedelay";
> > >> > >    public static final String BLUR_CONTROLLER_HOSTNAME =
> > >> > "blur.controller.hostname";
> > >> > >    public static final String BLUR_CONTROLLER_BIND_PORT =
> > >> > "blur.controller.bind.port";
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/blur-util/src/main/resources/blur-default.properties
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git a/blur-util/src/main/resources/blur-default.properties
> > >> > b/blur-util/src/main/resources/blur-default.properties
> > >> > > index d28f39d..4a1ef8b 100644
> > >> > > --- a/blur-util/src/main/resources/blur-default.properties
> > >> > > +++ b/blur-util/src/main/resources/blur-default.properties
> > >> > > @@ -66,6 +66,9 @@ blur.shard.index.warmup.throttle=30000000
> > >> > >  # By default the block cache using off heap memory
> > >> > >  blur.shard.blockcache.direct.memory.allocation=true
> > >> > >
> > >> > > +# By default the experimental block cache is off
> > >> > > +blur.shard.experimental.block.cache=false
> > >> > > +
> > >> > >  # The slabs in the blockcache are automatically configured by
> > default
> > >> > (-1) otherwise 1 slab equals 128MB
> > >> > >  # The auto config is detected through the MaxDirectoryMemorySize
> > >> > provided to the JVM
> > >> > >  blur.shard.blockcache.slab.count=-1
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://git-wip-us.apache.org/repos/asf/incubator-blur/blob/5ea9c30e/docs/cluster-setup.html
> > >> > >
> > ----------------------------------------------------------------------
> > >> > > diff --git a/docs/cluster-setup.html b/docs/cluster-setup.html
> > >> > > index d6ba58b..e63e4f5 100644
> > >> > > --- a/docs/cluster-setup.html
> > >> > > +++ b/docs/cluster-setup.html
> > >> > > @@ -284,6 +284,9 @@ blur.shard.index.warmup.throttle=30000000
> > >> > >  # By default the block cache using off heap memory
> > >> > >  blur.shard.blockcache.direct.memory.allocation=true
> > >> > >
> > >> > > +# By default the experimental block cache is off
> > >> > > +blur.shard.experimental.block.cache=false
> > >> > > +
> > >> > >  # The slabs in the blockcache are automatically configured by
> > >> > >  # default (-1) otherwise 1 slab equals 128MB. The auto config
> > >> > >  # is detected through the MaxDirectoryMemorySize provided to
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message