hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Meaning of storefileIndexSize
Date Tue, 18 May 2010 15:51:21 GMT
On Tue, May 18, 2010 at 2:15 AM, Renaud Delbru <renaud.delbru@deri.org> wrote:
> Hi,
>
> after some tuning, like increasing the hfile block size to 128KB,  I have
> noticed that the storefileIndexSize is now half of what it was before
> (~250). Do storefileIndexSize is the size of the in-memory hfile block index
> ?

Yes.

So, yes, doubling the block size should halve the index size.

How come your index is so big?  Do you have big keys?  Lots of data?
Lots of storefiles?

Looking in HRegionServer I see that its calculated so:

 storefileIndexSizeMB = (int)(store.getStorefilesIndexSize()/1024/1024);

In the Store, we do this:

  /**
   * @return The size of the store file indexes, in bytes.
   */
  long getStorefilesIndexSize() {
    long size = 0;
    for (StoreFile s: storefiles.values()) {
      Reader r = s.getReader();
      if (r == null) {
        LOG.warn("StoreFile " + s + " has a null Reader");
        continue;
      }
      size += r.indexSize();
    }
    return size;
  }

The indexSize is out of the HFile metadata.

St.Ack

>
> Thanks
> --
> Renaud Delbru
>
> On 17/05/10 15:27, Renaud Delbru wrote:
>>
>> Hi,
>>
>> I would like to understand the meaning of the storefileIndexSize metric,
>> could someone point me to a definition or explain me what does that mean ?
>>
>> Also, we are performing a large table import (90M rows, size of the row
>> varying between hundreds of kb to 8 MB), and we are encountering memory
>> problem (OOME). My observation is that it always happens after a while, when
>> the storefileIndexSize starts to be large (> 500). Is there a way to reduce
>> it ?
>>
>> Thanks,
>
>

Mime
View raw message