From dev-return-15777-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Fri Nov 1 13:36:23 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id C9A59180626 for ; Fri, 1 Nov 2019 14:36:22 +0100 (CET) Received: (qmail 83365 invoked by uid 500); 1 Nov 2019 13:36:21 -0000 Mailing-List: contact dev-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@arrow.apache.org Delivered-To: mailing list dev@arrow.apache.org Received: (qmail 83297 invoked by uid 99); 1 Nov 2019 13:36:20 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Nov 2019 13:36:20 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 3D87BC2F12 for ; Fri, 1 Nov 2019 13:36:20 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.003 X-Spam-Level: X-Spam-Status: No, score=0.003 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=dremio.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id XeP8-0vyfuw0 for ; Fri, 1 Nov 2019 13:36:19 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.160.173; helo=mail-qt1-f173.google.com; envelope-from=ravindra@dremio.com; receiver= Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 9A58AC1B4C for ; Fri, 1 Nov 2019 13:36:18 +0000 (UTC) Received: by mail-qt1-f173.google.com with SMTP id t26so12949482qtr.5 for ; Fri, 01 Nov 2019 06:36:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dremio.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=BikBmOGEgO+4noRHaqBw2hJpMJajw1BsIccQyKFa+LU=; b=iVkeVXvUcoci5eZ0UK39lYIk+wDMX963te33R1JEtK/d8NiFclzIU6K0yKscsskglL yVyQ9DDjfXtB0Ssb2eG1dGFVTV50PCNQnOYlnDgrm7070D7dZU8YRrLuxqCKOuJKRnG0 6O6LO83Zj0PSFqpxcCpY6tJzsGHNetPQRz3apklpsKl7suuG7b1zJxBNDK7KpMt73Xke KzeD0ry5gloJClgKEBD1hY2E/2O3SyYg25DIlNZOt42UrtroF6bxqJ97x3KcTC9xwRbg f4DzBLJBUZWMj7vRQSYKTSv+Oug61/LqhWBRKW6r25yyxIgcja64Egy2y6sO02aLRPcI 0ubw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=BikBmOGEgO+4noRHaqBw2hJpMJajw1BsIccQyKFa+LU=; b=IWcbl73UIwTjY9CwZFZxZ3FEx+ALLRTnwv7jW5Qf6+okyZ4Ka84TLgodWkMmbbgXhA ngfW6b0mmVlVVliaTNBMhYCQBLMxlOtEV1Twj/puYeDOkfNa4U0LHe3V5FOwonjqKz7t TDQOYtBi1agjoA0oJM+oWRp/MOmxPBNNt2r6DXcrMVjiFrYgpfMuvpAnMPKsAeKTBXoa GVlQ4wCWV3113iMcvZuQ+ccEqkIFuAYZEN141uEsrSS+H9Vul1rkLqwr2j7szRTlMDKs evy8hTa7o4h64N4CUIIgxVDTyzBGEQJZuiJiQIFPK4XSJwPjAy1+RA1e0DKPdpdnJa4A dYKA== X-Gm-Message-State: APjAAAUO32Jh37jbcrf9icypak0b1STDRZ0DYkr/wUrbhuQms5DEW8nT nf5XbcqLjfwDLm9HU8wzUqNailuZ5iLxgPB0213AYU4Adn4= X-Google-Smtp-Source: APXvYqxS+L0nDbJ2oIQ/12ySj2M1HVX7x1Nxh0LaGJWOOFmgfpE2Y5Og1GU2tvi7OxMjzQUEEHu9Cd5Q1Q0DSb8nxAg= X-Received: by 2002:ac8:1663:: with SMTP id x32mr25844qtk.368.1572615377859; Fri, 01 Nov 2019 06:36:17 -0700 (PDT) MIME-Version: 1.0 References: <4b078b57-c293-a28e-d754-6eeec4cf9b3c@arm.com> <887871ff-5278-a455-84ec-ea7074eca76a@arm.com> In-Reply-To: <887871ff-5278-a455-84ec-ea7074eca76a@arm.com> From: Ravindra Pindikura Date: Fri, 1 Nov 2019 19:06:06 +0530 Message-ID: Subject: Re: questions about Gandiva To: dev Cc: Wes McKinney Content-Type: multipart/alternative; boundary="000000000000f7b1fa0596490c39" --000000000000f7b1fa0596490c39 Content-Type: text/plain; charset="UTF-8" On Fri, Nov 1, 2019 at 10:41 AM Yibo Cai wrote: > Thanks Wes. Arrow is a very exciting project. > I'm from Arm. We are interested in arrow and would like to study and help > improving arrow. > If you are familiar with LLVM/JIT, you could help us with improving the optimisation passes in gandiva (tweaking existing ones or adding new ones or any other tricks ..) > > Yibo > > On 11/1/19 1:25 AM, Wes McKinney wrote: > > hi > > > > On Thu, Oct 31, 2019 at 12:11 AM Yibo Cai wrote: > >> > >> Hi, > >> > >> Arrow cpp integrates Gandiva to provide low level operations on arrow > buffers. [1][2] > >> I have some questions, any help is appreciated: > >> - Arrow cpp already has a compute kernel[3], does it duplicate what > Gandiva provides? I see a Jira talk about it.[4] > > > > No. There are some cases of functional overlap but we are servicing a > > spectrum of use cases beyond the scope of Gandiva. Additionally, it is > > unclear to me that an LLVM JIT compilation step should be required to > > evaluate simple expressions such as "a > 5" -- in addition to > > introducing latency (due to the compilation step) it is also a heavy > > dependency to require the LLVM runtime in all applications. > > > > Personally I'm interested in supporting a wide gamut of analytics > > workloads, from data frame / data science type libraries to SQL-like > > systems. Gandiva is designed for the needs of a SQL-based execution > > engine where chunks of data are fed into Projection or Filter nodes in > > a computation graph -- Gandiva generates a specialized kernel to > > perform a unit of work inside those nodes. Realistically, I expect > > many real world applications will contain a mixture of pre-compiled > > analytic kernels and JIT-compiled kernels. > > > > Rome wasn't built in a day, so I'm expecting several years of work > > ahead of us at the present rate. We need more help in this domain. > > > >> - Is Gandiva only for arrow cpp? What about other languages(go, rust, > ...)? > > > > It's being used in Java via JNI. The same approach could be applied > > for the other languages as they have their own C FFI mechanisms. > > > >> - Gandiva leverages SIMD for vectorized operations[1], but I didn't see > any related code. Am I missing something? > > > > My understanding is that LLVM inserts many SIMD instructions > > automatically based on the host CPU architecture version. Gandiva > > developers may have some comments / pointers about this > > > >> > >> [1] > https://www.dremio.com/announcing-gandiva-initiative-for-apache-arrow/ > >> [2] https://github.com/apache/arrow/tree/master/cpp/src/gandiva > >> [3] https://github.com/apache/arrow/tree/master/cpp/src/arrow/compute > >> [4] https://issues.apache.org/jira/browse/ARROW-7017 > >> > >> Thanks, > >> Yibo > -- Thanks and regards, Ravindra. --000000000000f7b1fa0596490c39--