Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E8D2D200CA9 for ; Fri, 16 Jun 2017 19:46:05 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id E7810160BDD; Fri, 16 Jun 2017 17:46:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 42ABE160BEB for ; Fri, 16 Jun 2017 19:46:05 +0200 (CEST) Received: (qmail 41597 invoked by uid 500); 16 Jun 2017 17:46:04 -0000 Mailing-List: contact dev-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@arrow.apache.org Delivered-To: mailing list dev@arrow.apache.org Received: (qmail 41519 invoked by uid 99); 16 Jun 2017 17:46:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jun 2017 17:46:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id DC6DAC03DE for ; Fri, 16 Jun 2017 17:46:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id E6go54hEm_Bg for ; Fri, 16 Jun 2017 17:46:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id ED27D5FCDA for ; Fri, 16 Jun 2017 17:46:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 20848E08F4 for ; Fri, 16 Jun 2017 17:46:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 42C432400C for ; Fri, 16 Jun 2017 17:46:00 +0000 (UTC) Date: Fri, 16 Jun 2017 17:46:00 +0000 (UTC) From: "Uwe L. Korn (JIRA)" To: dev@arrow.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (ARROW-1123) C++: Make jemalloc the default allocator MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 16 Jun 2017 17:46:06 -0000 Uwe L. Korn created ARROW-1123: ---------------------------------- Summary: C++: Make jemalloc the default allocator Key: ARROW-1123 URL: https://issues.apache.org/jira/browse/ARROW-1123 Project: Apache Arrow Issue Type: New Feature Components: C++ Reporter: Uwe L. Korn While one of the advertised point of Arrow Java is that it comes with an efficient allocator that is based on jemalloc, we only provide jemalloc as an alternative allocator in C++. I would propose to switch to it as the default allocator on all platforms jemalloc is available (i.e. MacOS and Linux) as it provides the following benefits: * in-place resizing while ensuring 64byte alignment, for allocations >4kb this won't touch the data but rather only changes some pointers in kernel internal structs pointing to the respecitive memory pages. * better allocation algorithm, see https://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919/ that should lead to less fragmentation and support fast multithreaded allocations Especially for the builders, simply changing the allocator shows speedups between 2x and 10x. In the optimal case, this puts builders nearly on the same level as if one would have pre-allocated the final size. This change should also include a blog post that outlines the benefits. -- This message was sent by Atlassian JIRA (v6.4.14#64029)