asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Carey <dtab...@gmail.com>
Subject Re: How to set the lsm component size?
Date Mon, 12 Sep 2016 23:10:55 GMT
+1


On 9/12/16 3:42 PM, Taewoo Kim wrote:
> It would be really helpful this conversation can be applied in the
> description of each parameter. Currently, I think that is too short.
>
> Best,
> Taewoo
>
> On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia <jianfeng.jia@gmail.com>
> wrote:
>
>> Clear. Thanks.
>>
>> And Ian’s parameters works. I can have a on-disk components around 128M.
>> Thanks!
>>
>>> On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee <salsubaiee@gmail.com>
>> wrote:
>>> This is the total memory size given for all datasets. Think of it as the
>>> buffer cache for all memory components of all indexes in that machine.
>> When
>>> it is exhausted, a victim dataset must be evicted and closed to have a
>>> space for another dataset.
>>>
>>> On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia <jianfeng.jia@gmail.com>
>>> wrote:
>>>
>>>> I was a little confused, there is another configuration:
>>>>
>>>> storage.memorycomponent.globalbudget ( which I set to 4G)
>>>>
>>>> I was thinking this is the budget that every component on one partition
>> is
>>>> shared. Is that the case?
>>>>
>>>>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee <salsubaiee@gmail.com>
>>>> wrote:
>>>>> The 128M is shared by all the memory components of the primary index
>> and
>>>>> all its secondary indexes across all io devices on that node.
>>>>> Also the in-memory components usually usually has fill factor of 75%
>>>> since
>>>>> the pages are 75% full and the remaining 25% is un-utilized.
>>>>>
>>>>> The page size that you have set 128KB looks reasonable for most cases.
>>>> Your
>>>>> best bet is to increase the value of storage.memorycomponent.numpage
>> to
>>>> a
>>>>> higher number.
>>>>>
>>>>> Sattam
>>>>>
>>>>>
>>>>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia <jianfeng.jia@gmail.com
>>>>> wrote:
>>>>>
>>>>>> Dear devs,
>>>>>>
>>>>>> I’m using the `no-merge` compaction policy and find that the physical
>>>>>> flushed on-disk component is smaller than I was expected.
>>>>>>
>>>>>> Here are my related configurations
>>>>>>
>>>>>> <property>
>>>>>>    <name>storage.memorycomponent.pagesize</name>
>>>>>>    <value>128KB</value>
>>>>>>    <description>The page size in bytes for pages allocated
to memory
>>>>>>      components. (Default = "131072" // 128KB)
>>>>>>    </description>
>>>>>> </property>
>>>>>>
>>>>>> <property>
>>>>>>    <name>storage.memorycomponent.numpages</name>
>>>>>>    <value>1024</value>
>>>>>>    <description>The number of pages to allocate for a memory
component.
>>>>>>      (Default = 256)
>>>>>>    </description>
>>>>>> </property>
>>>>>>
>>>>>> With these two settings, I’m expecting the lsm component should
be
>> 128M.
>>>>>> However, the flushed one is about 16M~ 20M. Do we have some
>> compression
>>>> for
>>>>>> the on-disk components? If so, it will be good. Otherwise, could
>> someone
>>>>>> help me to increase the component size? Thanks!
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Jianfeng Jia
>>>>>> PhD Candidate of Computer Science
>>>>>> University of California, Irvine
>>>>>>
>>>>>>
>>>>
>>>>
>>>> Best,
>>>>
>>>> Jianfeng Jia
>>>> PhD Candidate of Computer Science
>>>> University of California, Irvine
>>>>
>>>>
>>
>>
>> Best,
>>
>> Jianfeng Jia
>> PhD Candidate of Computer Science
>> University of California, Irvine
>>
>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message