hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HBASE-1597) Prevent unnecessary caching of blocks during compactions
Date Fri, 03 Jul 2009 00:39:47 GMT

     [ https://issues.apache.org/jira/browse/HBASE-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Gray updated HBASE-1597:
---------------------------------

    Attachment: HBASE-1597-v1.patch

Creates an additional HFile.Reader subclass called HFile.CompactionReader

Overrides a single method:  ByteBuffer readBlock(int)

Same code in that method except at the end it doesn't cache the block.

Adds a method StoreFile.getCompactionReader() which instantiates a new CompactionReader, taking
the normal Reader as input.

Switched the necessary calls from getReader() to getCompactionReader().  There were about
6 of em.

The nasty part of this now is that it has to instantiate a new CompactionReader rather than
just casting or setting a flag.  Lots of other ways to implement this, probably more cleanly
and more efficient... Just wanted to put something up so we can have a basis for continued
discussion


Compaction test passes.

> Prevent unnecessary caching of blocks during compactions
> --------------------------------------------------------
>
>                 Key: HBASE-1597
>                 URL: https://issues.apache.org/jira/browse/HBASE-1597
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>             Fix For: 0.20.0
>
>         Attachments: HBASE-1597-v1.patch
>
>
> When running any kind of compaction, we read every block of every storefile being compacted
into the block cache.
> We would like to reuse any already cached blocks, if available, but otherwise we should
not bog down the LRU with unnecessary blocks.
> This is not as bad as it was with the old LRU because the latest LRU implementation (HBASE-1460)
is scan-resistant.  This ensures that we are not causing massive eviction of the blocks that
are being read multiple times or from in-memory tables.  However, this does add to the GC-woes
of an import because each block gets further referenced, and for longer periods of time. 
There is also overhead in running the LRU evictions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message