From dev-return-70506-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Mon Aug 6 17:04:45 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9AA5E180627 for ; Mon, 6 Aug 2018 17:04:44 +0200 (CEST) Received: (qmail 67332 invoked by uid 500); 6 Aug 2018 15:04:43 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 67321 invoked by uid 99); 6 Aug 2018 15:04:43 -0000 Received: from mail-relay.apache.org (HELO mailrelay2-lw-us.apache.org) (207.244.88.137) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Aug 2018 15:04:43 +0000 Received: from hw13390.local (pool-96-244-88-135.bltmmd.fios.verizon.net [96.244.88.135]) by mailrelay2-lw-us.apache.org (ASF Mail Server at mailrelay2-lw-us.apache.org) with ESMTPSA id CB5FA1F11 for ; Mon, 6 Aug 2018 15:04:42 +0000 (UTC) Subject: Re: [DISCUSS] Expanded "work items" for HBase-in-the-Cloud doc To: dev@hbase.apache.org References: <49735383-871b-b916-a1b6-523b26558c08@apache.org> From: Josh Elser Message-ID: <91ed134b-3f14-531f-8440-05a83f04277f@apache.org> Date: Mon, 6 Aug 2018 11:04:42 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Yup, replication is a big one to "unravel". Repeating myself from a branch in the thread, but I'd expect some initial suggestions on what a new API could be this week. Certainly the first draft won't be the final -- would be great to get your input after your AsyncWAL work, Duo. Using AWS SimpleQueryService, or much anything else, would be great. I want to make sure that, while we try to "scratch this one itch", we pave the way for whatever else folks want to experiment with. On 8/4/18 5:10 AM, 张铎(Duo Zhang) wrote: > Yes, maybe we could write WAL to SQS and HFile to S3, then we can deploy > HBase on AWS without any local storage volumes... > > But we also need a good abstraction for Replication, as the current design > is file based... > > 2018-07-27 1:28 GMT+08:00 Zach York : > >> I would REALLY hope that the WAL interface/API changes would go into master >> even if the feature work for Ratis is going in a feature branch. Not only >> would this enable other backends to be developed in parallel with the Ratis >> solution if there are other good fits for a non-HDFS WAL, but also it would >> save the burden of having to rebase these core changes onto the latest >> master to maintain compatibility. I'm assuming the Ratis portion of the >> code would be mostly new files so these would be less of a concern. >> >> On Thu, Jul 26, 2018 at 9:24 AM, Josh Elser wrote: >> >>> On 7/26/18 1:00 AM, Stack wrote: >>> >>>> All this said, I'd like to start moving toward the point where we start >>>>> breaking out this work into a feature-branch off of master and start >>>>> building code. My hope is that this is amenable to everyone, with the >>>>> acknowledge that the Ratis work is considered "experimental" and not an >>>>> attempt to make all of HBase use Ratis-backed WALs. >>>>> >>>>> >>>>> Go for it. >>>> >>>> The branch would have WAL API changes only or would it include Ratis WAL >>>> dev? (If the latter, would that be better done over on Ratis project?). >>>> >>> >>> I think we would start with WAL API changes, get those "blessed", and >> then >>> continue Ratis WAL dev after that. >>> >> >