asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taewoo Kim <wangs...@gmail.com>
Subject Re: How to set the lsm component size?
Date Tue, 13 Sep 2016 00:02:43 GMT
Thanks to Sattam, here is the revised version. Feel free to revise this. I
will upload a patch set after some revision is done.

*storage.memorycomponent.numpages*

The number of pages to allocate for a memory component. (Default = 256)
This budget is shared by all the memory components of the primary index and
all its secondary indexes across all I/O devices on a node.
Note: in-memory components usually has fill factor of 75% since the pages
are 75% full and the remaining 25% is un-utilized.


*storage.memorycomponent.globalbudget*

[4GB + 100MB] The total size of memory in bytes that the sum of all open
memory components cannot exceed. (Default = 512MB)
Consider this as the buffer cache for all memory components of all indexes
in a node.
When this budget is fully used, a victim dataset will be chosen. It must be
evicted and closed to make a space for another dataset.


Best,
Taewoo

On Mon, Sep 12, 2016 at 4:10 PM, Mike Carey <dtabass@gmail.com> wrote:

> +1
>
>
>
> On 9/12/16 3:42 PM, Taewoo Kim wrote:
>
>> It would be really helpful this conversation can be applied in the
>> description of each parameter. Currently, I think that is too short.
>>
>> Best,
>> Taewoo
>>
>> On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia <jianfeng.jia@gmail.com>
>> wrote:
>>
>> Clear. Thanks.
>>>
>>> And Ian’s parameters works. I can have a on-disk components around 128M.
>>> Thanks!
>>>
>>> On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee <salsubaiee@gmail.com>
>>>>
>>> wrote:
>>>
>>>> This is the total memory size given for all datasets. Think of it as the
>>>> buffer cache for all memory components of all indexes in that machine.
>>>>
>>> When
>>>
>>>> it is exhausted, a victim dataset must be evicted and closed to have a
>>>> space for another dataset.
>>>>
>>>> On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia <jianfeng.jia@gmail.com>
>>>> wrote:
>>>>
>>>> I was a little confused, there is another configuration:
>>>>>
>>>>> storage.memorycomponent.globalbudget ( which I set to 4G)
>>>>>
>>>>> I was thinking this is the budget that every component on one partition
>>>>>
>>>> is
>>>
>>>> shared. Is that the case?
>>>>>
>>>>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee <salsubaiee@gmail.com>
>>>>>>
>>>>> wrote:
>>>>>
>>>>>> The 128M is shared by all the memory components of the primary index
>>>>>>
>>>>> and
>>>
>>>> all its secondary indexes across all io devices on that node.
>>>>>> Also the in-memory components usually usually has fill factor of
75%
>>>>>>
>>>>> since
>>>>>
>>>>>> the pages are 75% full and the remaining 25% is un-utilized.
>>>>>>
>>>>>> The page size that you have set 128KB looks reasonable for most cases.
>>>>>>
>>>>> Your
>>>>>
>>>>>> best bet is to increase the value of storage.memorycomponent.numpage
>>>>>>
>>>>> to
>>>
>>>> a
>>>>>
>>>>>> higher number.
>>>>>>
>>>>>> Sattam
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia <
>>>>>> jianfeng.jia@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>> Dear devs,
>>>>>>>
>>>>>>> I’m using the `no-merge` compaction policy and find that the
physical
>>>>>>> flushed on-disk component is smaller than I was expected.
>>>>>>>
>>>>>>> Here are my related configurations
>>>>>>>
>>>>>>> <property>
>>>>>>>    <name>storage.memorycomponent.pagesize</name>
>>>>>>>    <value>128KB</value>
>>>>>>>    <description>The page size in bytes for pages allocated
to memory
>>>>>>>      components. (Default = "131072" // 128KB)
>>>>>>>    </description>
>>>>>>> </property>
>>>>>>>
>>>>>>> <property>
>>>>>>>    <name>storage.memorycomponent.numpages</name>
>>>>>>>    <value>1024</value>
>>>>>>>    <description>The number of pages to allocate for a memory
>>>>>>> component.
>>>>>>>      (Default = 256)
>>>>>>>    </description>
>>>>>>> </property>
>>>>>>>
>>>>>>> With these two settings, I’m expecting the lsm component should
be
>>>>>>>
>>>>>> 128M.
>>>
>>>> However, the flushed one is about 16M~ 20M. Do we have some
>>>>>>>
>>>>>> compression
>>>
>>>> for
>>>>>
>>>>>> the on-disk components? If so, it will be good. Otherwise, could
>>>>>>>
>>>>>> someone
>>>
>>>> help me to increase the component size? Thanks!
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Jianfeng Jia
>>>>>>> PhD Candidate of Computer Science
>>>>>>> University of California, Irvine
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>> Best,
>>>>>
>>>>> Jianfeng Jia
>>>>> PhD Candidate of Computer Science
>>>>> University of California, Irvine
>>>>>
>>>>>
>>>>>
>>>
>>> Best,
>>>
>>> Jianfeng Jia
>>> PhD Candidate of Computer Science
>>> University of California, Irvine
>>>
>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message