arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe L. Korn (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ARROW-1123) C++: Make jemalloc the default allocator
Date Fri, 16 Jun 2017 17:46:00 GMT
Uwe L. Korn created ARROW-1123:
----------------------------------

             Summary: C++: Make jemalloc the default allocator
                 Key: ARROW-1123
                 URL: https://issues.apache.org/jira/browse/ARROW-1123
             Project: Apache Arrow
          Issue Type: New Feature
          Components: C++
            Reporter: Uwe L. Korn


While one of the advertised point of Arrow Java is that it comes with an efficient allocator
that is based on jemalloc, we only provide jemalloc as an alternative allocator in C++. I
would propose to switch to it as the default allocator on all platforms jemalloc is available
(i.e. MacOS and Linux) as it provides the following benefits:

 * in-place resizing while ensuring 64byte alignment, for allocations >4kb this won't touch
the data but rather only changes some pointers in kernel internal structs pointing to the
respecitive memory pages.
 * better allocation algorithm, see https://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919/
that should lead to less fragmentation and support fast multithreaded allocations

Especially for the builders, simply changing the allocator shows speedups between 2x and 10x.
In the optimal case, this puts builders nearly on the same level as if one would have pre-allocated
the final size.
 
This change should also include a blog post that outlines the benefits.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message