Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of varun@pinterest.com
 designates 209.85.223.177 as permitted sender)
MIME-Version: 1.0
Date: Mon, 10 Dec 2012 05:58:21 -0800
Message-ID: 
 <CAKxWWm3-MDqCydZcZmLPK-E583WtCNSX1kcJq2aOjApewYKOXg@mail.gmail.com>
Subject: Filtering/Collection columns during Major Compaction
From: Varun Sharma <varun@pinterest.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=14dae93403d94e554b04d07ff539

--14dae93403d94e554b04d07ff539
Content-Type: text/plain; charset=ISO-8859-1

Hi,

My understanding of major compaction is that it rewrites one store file and
does a merge of the memstore, store files on disk and cleans out delete
tombstones and puts prior to them and cleans out excess versions. We want
to limit the number of columns per row in hbase. Also, we want to limit
them in lexicographically sorted order - which means we take the top, say
100 smallest columns (in lexicographical sense) and only keep them while
discard the rest.

One way to do this would be to clean out columns in a daily mapreduce job.
Or another way is to clean them out during the major compaction which can
be run daily too. I see, from the code that a major compaction essentially
invokes a Scan over the region - so if the Scan is invoked with the
appropriate filter (say ColumnCountGetFilter) - would that do the trick ?

Thanks
Varun

--14dae93403d94e554b04d07ff539--