asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taewoo Kim <wangs...@gmail.com>
Subject Re: How to set the lsm component size?
Date Mon, 12 Sep 2016 22:42:31 GMT
It would be really helpful this conversation can be applied in the
description of each parameter. Currently, I think that is too short.

Best,
Taewoo

On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia <jianfeng.jia@gmail.com>
wrote:

> Clear. Thanks.
>
> And Ian’s parameters works. I can have a on-disk components around 128M.
> Thanks!
>
> > On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee <salsubaiee@gmail.com>
> wrote:
> >
> > This is the total memory size given for all datasets. Think of it as the
> > buffer cache for all memory components of all indexes in that machine.
> When
> > it is exhausted, a victim dataset must be evicted and closed to have a
> > space for another dataset.
> >
> > On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia <jianfeng.jia@gmail.com>
> > wrote:
> >
> >> I was a little confused, there is another configuration:
> >>
> >> storage.memorycomponent.globalbudget ( which I set to 4G)
> >>
> >> I was thinking this is the budget that every component on one partition
> is
> >> shared. Is that the case?
> >>
> >>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee <salsubaiee@gmail.com>
> >> wrote:
> >>>
> >>> The 128M is shared by all the memory components of the primary index
> and
> >>> all its secondary indexes across all io devices on that node.
> >>> Also the in-memory components usually usually has fill factor of 75%
> >> since
> >>> the pages are 75% full and the remaining 25% is un-utilized.
> >>>
> >>> The page size that you have set 128KB looks reasonable for most cases.
> >> Your
> >>> best bet is to increase the value of storage.memorycomponent.numpage
> to
> >> a
> >>> higher number.
> >>>
> >>> Sattam
> >>>
> >>>
> >>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia <jianfeng.jia@gmail.com
> >
> >>> wrote:
> >>>
> >>>> Dear devs,
> >>>>
> >>>> I’m using the `no-merge` compaction policy and find that the physical
> >>>> flushed on-disk component is smaller than I was expected.
> >>>>
> >>>> Here are my related configurations
> >>>>
> >>>> <property>
> >>>>   <name>storage.memorycomponent.pagesize</name>
> >>>>   <value>128KB</value>
> >>>>   <description>The page size in bytes for pages allocated to memory
> >>>>     components. (Default = "131072" // 128KB)
> >>>>   </description>
> >>>> </property>
> >>>>
> >>>> <property>
> >>>>   <name>storage.memorycomponent.numpages</name>
> >>>>   <value>1024</value>
> >>>>   <description>The number of pages to allocate for a memory component.
> >>>>     (Default = 256)
> >>>>   </description>
> >>>> </property>
> >>>>
> >>>> With these two settings, I’m expecting the lsm component should be
> 128M.
> >>>> However, the flushed one is about 16M~ 20M. Do we have some
> compression
> >> for
> >>>> the on-disk components? If so, it will be good. Otherwise, could
> someone
> >>>> help me to increase the component size? Thanks!
> >>>>
> >>>> Best,
> >>>>
> >>>> Jianfeng Jia
> >>>> PhD Candidate of Computer Science
> >>>> University of California, Irvine
> >>>>
> >>>>
> >>
> >>
> >>
> >> Best,
> >>
> >> Jianfeng Jia
> >> PhD Candidate of Computer Science
> >> University of California, Irvine
> >>
> >>
>
>
>
> Best,
>
> Jianfeng Jia
> PhD Candidate of Computer Science
> University of California, Irvine
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message