hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@apache.org>
Subject Re: Query for OldWals and use of WAl for Hbase indexer
Date Thu, 12 Jul 2018 15:17:30 GMT
On Wed, Jul 11, 2018 at 12:49 PM, Manjeet Singh
<manjeet.chandhok@gmail.com> wrote:
> Thanks Sean for your reply
>
> I still have some question un answered like
> Q1: How Hbase syncronized with Hbase indexer.

The Lily Indexer for HBase (sometimes referred to as the "Lily HBase
Indexer") is an independent project, linked to in my previous email.
They would be best positioned to answer questions you have about how
it works. I suggest you talk with that project.

My understanding is that the Lily Indexer leverages the HBase
replication system to get observe edits as they come into the system
and then applies a corresponding change to the index it maintains.
That means it has all the same configurable options and warnings that
come with any HBase replication set up:

http://hbase.apache.org/1.2/book.html#_cluster_replication


> Q2 What optimization I can apply.

You need to dig into the logs to find out if there is a problem
talking to the indexers or if they are just lagging. If they're
lagging, then I believe you can add more indexer nodes to scale up
effective throughput. The Lily Indexer for HBase project would be
better suited to answer that though.

> Q3 As it's clear from my stats, data in OldWals is quite huge so it's not
> getting clear my HMaster., how can I improve my HDFS space issue due to
> this?

The only way to safely decrease the size of the retained wals is to
make it so they are no longer needed. That means either getting the
Lily Indexer for HBase to catch up or reseting things and using a
batch indexing method to fill in the gap.

Mime
View raw message