Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Thu, 26 Oct 2017 00:29:00 +0000 (UTC)
From: "Solomon Duskis (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13102024.1505327259000.63563.1508977740344@Atlassian.JIRA>
In-Reply-To: <JIRA.13102024.1505327259000@Atlassian.JIRA>
References: <JIRA.13102024.1505327259000@Atlassian.JIRA> <JIRA.13102024.1505327259705@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-2955) Create a Cloud Bigtable HBase
 connector
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 26 Oct 2017 00:29:08 -0000


    [ https://issues.apache.org/jira/browse/BEAM-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16219783#comment-16219783 ] 

Solomon Duskis commented on BEAM-2955:
--------------------------------------

The problem is that Cloud Bigtable needs the following things:

# A different method for splitting.
# A different configuration mechanism for Cloud Bigtable specific configuration.  The configuration mechanism would also require the use of ValueProvider for templating purposes.
# A custom Cloud Bigtable oriented metric for expressing throttling.
# A custom way to use MultiRowRangeFilter (which is different between Cloud Bigtable and HBase)

There are probably other differences I'm missing.

A Service works for issue #1, but not for the rest.  There definitely is room for reuse, but I'm not sure if passing a Service to HBaseIO is the right way to do it.

> Create a Cloud Bigtable HBase connector
> ---------------------------------------
>
>                 Key: BEAM-2955
>                 URL: https://issues.apache.org/jira/browse/BEAM-2955
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-gcp
>            Reporter: Solomon Duskis
>            Assignee: Solomon Duskis
>
> The Cloud Bigtable (CBT) team has had a Dataflow connector maintained in a different repo for awhile. Recently, we did some reworking of the Cloud Bigtable client that would allow it to better coexist in the Beam ecosystem, and we also released a Beam connector in our repository that exposes HBase idioms rather than the Protobuf idioms of BigtableIO.  More information about the customer experience of the HBase connector can be found here: [https://cloud.google.com/bigtable/docs/dataflow-hbase].
> The Beam repo is a much better place to house a Cloud Bigtable HBase connector.  There are a couple of ways we can implement this new connector:
> # The CBT connector depends on artifacts in the io/hbase maven project.  We can create a new extend HBaseIO for the purposes of CBT.  We would have to add some features to HBaseIO to make that work (dynamic rebalancing, and a way for HBase and CBT's size estimation models to coexist)
> # The BigtableIO connector works well, and we can add an adapter layer on top of it.  I have a proof of concept of it here: [https://github.com/sduskis/cloud-bigtable-client/tree/add_beam/bigtable-dataflow-parent/bigtable-hbase-beam].
> # We can build a separate CBT HBase connector.
> I'm happy to do the work.  I would appreciate some guidance and discussion about the right approach.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)