arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe L. Korn (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (ARROW-464) C++: More intelligent array growing
Date Sat, 07 Jan 2017 17:16:58 GMT

     [ https://issues.apache.org/jira/browse/ARROW-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Uwe L. Korn updated ARROW-464:
------------------------------
    Description: 
Three things to consider:

 * Instead of always growing the memory to twice the size in the Builders, we should consider
1.5 as a growth factor as this still leads to less memory wastage. As the allocation costly
is mostly linear in the number of newly requested pages, this shouldn't be a noticeable impact.
Due to memory below the size of a single page (i.e. 4KiB) not being able to be expanded in
place, we should keep the factor of there.
 * In the case of jemalloc, we can ask the allocator with the function {{nallocx}} for the
size it would actually allocate for the requested size and then allocate this as the new capacity
for the Builder/Buffer/... and not trigger an reallocation as long as the output of {{nallocx}} doesn't
change.
 * If not using {{jemalloc}}, we should be careful about the cost of minor allocation size
changes. It would be preferable in these cases to have also an implementation along the lines
of {{nallocx}} that does some rounding to avoid unnecessary reallocations.

  was:
Two things to consider:

 * Instead of always growing the memory to twice the size in the Builders, we should consider
1.5 as a growth factor as this still leads to less memory wastage. As the allocation costly
is mostly linear in the number of newly requested pages, this shouldn't be a noticeable impact.
Due to memory below the size of a single page (i.e. 4KiB) not being able to be expanded in
place, we should keep the factor of there.
 * In the case of jemalloc, we can ask the allocator with the function {{nallocx}} for the
size it would actually allocate for the requested size and then allocate this as the new capacity
for the Builder/Buffer/... and not trigger an reallocation as long as the output of {{nallocx}} doesn't
change.


> C++: More intelligent array growing
> -----------------------------------
>
>                 Key: ARROW-464
>                 URL: https://issues.apache.org/jira/browse/ARROW-464
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>            Reporter: Uwe L. Korn
>            Assignee: Uwe L. Korn
>
> Three things to consider:
>  * Instead of always growing the memory to twice the size in the Builders, we should
consider 1.5 as a growth factor as this still leads to less memory wastage. As the allocation
costly is mostly linear in the number of newly requested pages, this shouldn't be a noticeable
impact. Due to memory below the size of a single page (i.e. 4KiB) not being able to be expanded
in place, we should keep the factor of there.
>  * In the case of jemalloc, we can ask the allocator with the function {{nallocx}} for
the size it would actually allocate for the requested size and then allocate this as the new
capacity for the Builder/Buffer/... and not trigger an reallocation as long as the output
of {{nallocx}} doesn't change.
>  * If not using {{jemalloc}}, we should be careful about the cost of minor allocation
size changes. It would be preferable in these cases to have also an implementation along the
lines of {{nallocx}} that does some rounding to avoid unnecessary reallocations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message