asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jianfeng Jia <jianfeng....@gmail.com>
Subject Re: How to set the lsm component size?
Date Mon, 12 Sep 2016 21:19:02 GMT
Clear. Thanks.

And Ian’s parameters works. I can have a on-disk components around 128M. Thanks!

> On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee <salsubaiee@gmail.com> wrote:
> 
> This is the total memory size given for all datasets. Think of it as the
> buffer cache for all memory components of all indexes in that machine. When
> it is exhausted, a victim dataset must be evicted and closed to have a
> space for another dataset.
> 
> On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia <jianfeng.jia@gmail.com>
> wrote:
> 
>> I was a little confused, there is another configuration:
>> 
>> storage.memorycomponent.globalbudget ( which I set to 4G)
>> 
>> I was thinking this is the budget that every component on one partition is
>> shared. Is that the case?
>> 
>>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee <salsubaiee@gmail.com>
>> wrote:
>>> 
>>> The 128M is shared by all the memory components of the primary index and
>>> all its secondary indexes across all io devices on that node.
>>> Also the in-memory components usually usually has fill factor of 75%
>> since
>>> the pages are 75% full and the remaining 25% is un-utilized.
>>> 
>>> The page size that you have set 128KB looks reasonable for most cases.
>> Your
>>> best bet is to increase the value of storage.memorycomponent.numpage to
>> a
>>> higher number.
>>> 
>>> Sattam
>>> 
>>> 
>>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia <jianfeng.jia@gmail.com>
>>> wrote:
>>> 
>>>> Dear devs,
>>>> 
>>>> I’m using the `no-merge` compaction policy and find that the physical
>>>> flushed on-disk component is smaller than I was expected.
>>>> 
>>>> Here are my related configurations
>>>> 
>>>> <property>
>>>>   <name>storage.memorycomponent.pagesize</name>
>>>>   <value>128KB</value>
>>>>   <description>The page size in bytes for pages allocated to memory
>>>>     components. (Default = "131072" // 128KB)
>>>>   </description>
>>>> </property>
>>>> 
>>>> <property>
>>>>   <name>storage.memorycomponent.numpages</name>
>>>>   <value>1024</value>
>>>>   <description>The number of pages to allocate for a memory component.
>>>>     (Default = 256)
>>>>   </description>
>>>> </property>
>>>> 
>>>> With these two settings, I’m expecting the lsm component should be 128M.
>>>> However, the flushed one is about 16M~ 20M. Do we have some compression
>> for
>>>> the on-disk components? If so, it will be good. Otherwise, could someone
>>>> help me to increase the component size? Thanks!
>>>> 
>>>> Best,
>>>> 
>>>> Jianfeng Jia
>>>> PhD Candidate of Computer Science
>>>> University of California, Irvine
>>>> 
>>>> 
>> 
>> 
>> 
>> Best,
>> 
>> Jianfeng Jia
>> PhD Candidate of Computer Science
>> University of California, Irvine
>> 
>> 



Best,

Jianfeng Jia
PhD Candidate of Computer Science
University of California, Irvine


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message