From user-return-698-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Wed Oct 7 05:42:47 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 6316D180644 for ; Wed, 7 Oct 2020 07:42:47 +0200 (CEST) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id 9502D123159 for ; Wed, 7 Oct 2020 05:42:46 +0000 (UTC) Received: (qmail 83029 invoked by uid 500); 7 Oct 2020 05:42:45 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 83019 invoked by uid 99); 7 Oct 2020 05:42:45 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Oct 2020 05:42:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id EB163BFE0C for ; Wed, 7 Oct 2020 05:42:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id bsODnTFNuWTR for ; Wed, 7 Oct 2020 05:42:44 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::534; helo=mail-ed1-x534.google.com; envelope-from=emkornfield@gmail.com; receiver= Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 5A2B37FB70 for ; Wed, 7 Oct 2020 05:42:44 +0000 (UTC) Received: by mail-ed1-x534.google.com with SMTP id 33so803174edq.13 for ; Tue, 06 Oct 2020 22:42:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:reply-to:from:date:message-id :subject:to; bh=NQtxP/u+1rt7NQughCplqoLSXqpptpXoV1s180yxHls=; b=SNY9azyqyA46DkGxnv6EpCLzo6NxIo6c+YtcOcXw50Rmv2VGhVVMKS37LiFMEduKPZ dxPTiV++ZuKO0SeZXM1GLxdfyEDTOscAB/VYQbjJN3RY0BU4sHSeWmOyPp0b7cGg/qpk BI2mPc0M3gp5g4rJBHLEBMC4CTQjs7zUXydLDMkQusYgHnGVIL6XtHFKBS/29lbMyoEx jf3a354LTP6hGCJ+/lYc18eVE++v3c6tUiCaJ2Jl1iUQhFLa2CefwKp9NdIfgAqg2/m6 hxQ5Mag8DWhB3jHZpNS6gX0DjombN3alwR4KDRyO060kMCQvr+tG3LEOHJAT3Q6Jh0GT qxmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to; bh=NQtxP/u+1rt7NQughCplqoLSXqpptpXoV1s180yxHls=; b=kO/h0ZMEX0A+q41mPl9Zp+zZmYNqNAltV//IhoA3naaWR19adjMX4lllZRsBgCj/iM mqCKP9IZZ4jXx/Pb6OIJbaWAU4tlKNNwN7QhfpPqXS6nRBLHiSpMTfFE9One/BlozhTv yRFCFuc6Aa8Zm4AhdOEl2HZ4ah+xB9tzQbyT8+dP0i5OJKLwsK1vW4FaFT6fScPVnlvX booByPXyHCDX6CugrdvKtksUF4uADotJjmA88wXw6L8wmLv/555ARwefw1l2gSZ2MoJB YNgvGCo0wmDLB8WEk5A8N0VlJa81igDdVUoFtzJdYCU07wU8TfmHIcpVQyGMpbWLLdsb 6yXg== X-Gm-Message-State: AOAM531Ag/dBEqIRA5ffvaKvhvl+54VDHlRPm4KqjQJHaHpC0iMCg4q+ ra5F4v+C+srsMj+6TWzSU7wlYGloQ1cUegu49DsAAt1QFRs= X-Google-Smtp-Source: ABdhPJyMtMETFWdck57RVQNonWmfeeVzdWgwNy+RBndxTlr+rBRSzh7PANnW5jDKaMW0Xj872+T38sI7JfsF9z361DA= X-Received: by 2002:a05:6402:6d8:: with SMTP id n24mr1535364edy.168.1602049363537; Tue, 06 Oct 2020 22:42:43 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: Reply-To: emkornfield@gmail.com From: Micah Kornfield Date: Tue, 6 Oct 2020 22:42:32 -0700 Message-ID: Subject: Re: Arrow and HBase To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="00000000000039f12805b10e2f38" --00000000000039f12805b10e2f38 Content-Type: text/plain; charset="UTF-8" Hi Matt, I'm not sure which presentations you are referring to, but if this is in reference to the diagram on the Overview page [1]. That is aspirational there are some integrations in various levels of depth for some of the technologies but it isn't the final state. To my knowledge there isn't a library within the Arrow project itself that currently connects to HBase and does any sort of HBase schema to Arrow schema conversion, but i think there might be some interest in building one at some point. There was a thread a little while ago on using Arrow within Redis [2] which might be of interest depending on how you are using HBase. Hope this helps. -Micah [1] https://arrow.apache.org/overview/ [2] https://mail-archives.apache.org/mod_mbox/arrow-user/202007.mbox/%3CCAKg4KDy3uji7dsZiLjRQhs%3DTfnBP%2BJanbdr-LFtNnBZftHKsdw%40mail.gmail.com%3E On Tue, Oct 6, 2020 at 8:21 AM Matt Duffy wrote: > I've seen several presentations listing HBase as a technology that > integrates with Arrow dataframes. Is there an example for reading/writing > to HBase from an Arrow dataframe? I'm hoping to utilize Arrow to expedite > the translation between HBase and Spark or Pandas. > --00000000000039f12805b10e2f38 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Matt,
I'm not sure which presentations you are = referring to, but if this is in reference to the diagram on the Overview pa= ge [1].=C2=A0 That is aspirational there are some integrations in various l= evels of depth for some of the technologies but it isn't the final stat= e.

To my knowledge there isn't a library withi= n the Arrow project itself that currently connects to HBase and does any so= rt of HBase schema to Arrow schema conversion, but i think there might be s= ome interest in building one at some point.=C2=A0=C2=A0

There was a thread a little while ago on using Arrow within Redis [2]= which might be of interest depending on how you are using HBase.

Hope this helps.

-Micah
=

On Tue, Oct = 6, 2020 at 8:21 AM Matt Duffy <matt.w.duffy@gmail.com> wrote:
I've seen several presentations= listing HBase as a technology that integrates with Arrow dataframes.=C2=A0= Is there an example for reading/writing to HBase from an Arrow dataframe?= =C2=A0 I'm hoping to utilize Arrow to expedite the translation between = HBase and Spark or Pandas.
--00000000000039f12805b10e2f38--