hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1597) Prevent unnecessary caching of blocks during compactions
Date Fri, 03 Jul 2009 17:17:47 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727054#action_12727054

stack commented on HBASE-1597:

This seems odd:

+    public CompactionReader(Reader reader) {
+      super(reader.istream, reader.fileSize, reader.cache);
+      super.blockIndex = reader.blockIndex;
+      super.trailer = reader.trailer;
+      super.lastkey = reader.lastkey;
+      super.avgKeyLen = reader.avgKeyLen;
+      super.avgValueLen = reader.avgValueLen;
+      super.comparator = reader.comparator;
+      super.metaIndex = reader.metaIndex;
+      super.fileInfoLoaded = reader.fileInfoLoaded;
+      super.compressAlgo = reader.compressAlgo;
+    }

Why not make a constructor that takes above args?

Too much of readBlock is duplicated in CompactionReader.  Would suggest changing backing readBlock
so it makes call to a new method that does the add-to-cache... then subclass this new method

Otherwise, nice patch.

> Prevent unnecessary caching of blocks during compactions
> --------------------------------------------------------
>                 Key: HBASE-1597
>                 URL: https://issues.apache.org/jira/browse/HBASE-1597
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>             Fix For: 0.20.0
>         Attachments: HBASE-1597-v1.patch
> When running any kind of compaction, we read every block of every storefile being compacted
into the block cache.
> We would like to reuse any already cached blocks, if available, but otherwise we should
not bog down the LRU with unnecessary blocks.
> This is not as bad as it was with the old LRU because the latest LRU implementation (HBASE-1460)
is scan-resistant.  This ensures that we are not causing massive eviction of the blocks that
are being read multiple times or from in-memory tables.  However, this does add to the GC-woes
of an import because each block gets further referenced, and for longer periods of time. 
There is also overhead in running the LRU evictions.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message