From dev-return-19121-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Thu Apr 25 21:26:05 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id E51DD180638 for ; Thu, 25 Apr 2019 23:26:04 +0200 (CEST) Received: (qmail 63762 invoked by uid 500); 25 Apr 2019 21:26:04 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 63717 invoked by uid 99); 25 Apr 2019 21:26:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Apr 2019 21:26:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 4DEE9C24AC for ; Thu, 25 Apr 2019 21:26:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.2 X-Spam-Level: X-Spam-Status: No, score=-0.2 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id DT6qijsvKfMO for ; Thu, 25 Apr 2019 21:26:00 +0000 (UTC) Received: from mail-oi1-f180.google.com (mail-oi1-f180.google.com [209.85.167.180]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id E973B5F6C3 for ; Thu, 25 Apr 2019 21:25:59 +0000 (UTC) Received: by mail-oi1-f180.google.com with SMTP id l7so1244336oil.11 for ; Thu, 25 Apr 2019 14:25:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=n+JtcZAl1rlF6QAA++tG/84fJyCpb64/dlbAmgD2FEw=; b=WsIrkP2yTkYgMfV3E1W86UWArvHaGxrB3pmQC0/S/HAeooIAXjzpERgUqXgWTB8cIV lUdqPZ1lT/XpiBIhtWCpyIsofSio5iRE5yfGMAHpU8LIkZLGhDzR4QmK1P5/BztzEu76 cgfW1pVOxoCW4Lqb3fGWzM0yhCoFX8Gnu2dlaxjgmv+hrdV2k6ciBMnGNEJfLpOBX8RY 4Ljz5XBliHs1gyLUdeJYlwgiZc/20kAwKU7c/xS0TsADBGhmm+/uy9TnrA6EICPz/RGx X/pQefDUTYLFlqqWrOeUsSQEDitQ9bOQZBZVVHQm65L+cBUK0WTkL6e+G+cnAGBga6Wb dFQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=n+JtcZAl1rlF6QAA++tG/84fJyCpb64/dlbAmgD2FEw=; b=ozTQoO8SnHYIAEubb2ywfqSKVgO4B6ci6w3S5RlmaXMC27dROGhHSSpsrbd+F5z2lJ asLNGZSxeEVqSFe4qNWpmIMEbCRVX8Q0JuGIpYokIqh5p8CLdLMbGXgms6KcaI8f5z7u liH+qdVpTyEiE7pHh/c1ZKsZu+D8JCEfusdKfNIdU1gvgFHDVXkiciTFdYS91L7xxAup ddUYebGvArwvHmbl6kPEFM55u3sHInu3imaJJDyDjU7IPTG2TO5++35nRMnBf/8FzrwW 0LtYkb4lj9bSEEe2Mp0ZNznpFmzC8wmJfzC3RcUH+F1701zL3trkzpprxFKaiK0RA6iy TDOg== X-Gm-Message-State: APjAAAVJWDW32fIJypxBHyTPvyZoOinFVEJKfc0CL0h0hGDVK9Nr6NQR U1bw+mBIISj44aGNxLTocbJVhYQl3vEMBFrzKPCXLK1V X-Google-Smtp-Source: APXvYqxO6nXmH/XHq6DbT7bDQtgi04ZtsYXblS2ClGMAArRjthBtN1lxm3usMMnxKOW+c6p6aVzTVBI12360sZeofpU= X-Received: by 2002:aca:b156:: with SMTP id a83mr4542135oif.119.1556227558350; Thu, 25 Apr 2019 14:25:58 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Bryan Bende Date: Thu, 25 Apr 2019 17:25:46 -0400 Message-ID: Subject: Re: Adding HBase Support for AtomicDistributedMapCacheClient To: dev@nifi.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks, I'm following now... I think adding the new method to the interface and throwing UnsupportedOperationException for 1_1_2, or using the original checkAndPut and implementing it in both services, would both be fine solutions. I guess another variation might be to introduce the new method in the interface, but in the 1_1_2 implementation just delegate back to the original checkAndPut and ignore the timestamp, and document that it isn't used in that implementation. I don't love this, but it does allow both services to implement the functionality and still leverage the better solution for 2_x. On Thu, Apr 25, 2019 at 3:54 PM Shawn Weeks wro= te: > > Here is what I think the new checkAndPut or checkAndMutate method would l= ook like. This also shows what the new mutate api looks like. > > @Override > public boolean checkAndPut(String tableName, byte[] rowId, byte[] fam= ily, byte[] qualifier, byte[] value, long timestamp, PutColumn column) thro= ws IOException { > try (final Table table =3D connection.getTable(TableName.valueOf(= tableName))) { > Put put =3D new Put(rowId); > put.addColumn( > column.getColumnFamily(), > column.getColumnQualifier(), > column.getBuffer()); > return table.checkAndMutate(rowId, family).qualifier(qualifie= r).ifEquals(value).timeRange(TimeRange.at(timestamp)).thenPut(put); > } > } > > If the atomic guarantee for the original checkAndPut is good enough then = there is no reason I can't implement the atomic map cache for both versions= of HBase. > > Thanks > Shawn > > -----Original Message----- > From: Bryan Bende > Sent: Thursday, April 25, 2019 12:39 PM > To: dev@nifi.apache.org > Subject: Re: Adding HBase Support for AtomicDistributedMapCacheClient > > I'm not totally if would matter if there were changes in between, as long= as the current value is what we thought it was then the changes we are sen= ding back should be accurate as a replacement. As a simplified scenario, if= the current value is 1 and thread-A retrieves that value, thread-B then ch= anges it to 2 and back to 1 before thread-A can do anything, then thread-A = sends in 2 with a previous of 1, that is still the correct replacement. > > I can see the argument for using the timestamp though... can you show the= method signature of the new checkAndMutate method that would need to be ad= ded to the client service, and also which method of the HBase client it nee= ds to call? > > Just so I can get an idea of the differences between 1.x and 2.x. > > On Thu, Apr 25, 2019 at 1:00 PM Shawn Weeks w= rote: > > > > While checkAndPut is atomic as it's built now it doesn't support also c= hecking the timestamp range which is included in the new checkAndMutate API= . I had planned on using the cell's timestamp as the revision along with th= e value to ensure not only that the value hadn't been changed but that ther= e hadn't been changes in between that just happened to put the value back. > > > > As I was looking at everything I had another question. Why is the cache= currently using a scan instead of a get to fetch values from HBase. It see= ms like that would be much less performant considering we know the row key = we're looking for. > > > > > > Thanks > > Shawn > > > > -----Original Message----- > > From: Bryan Bende > > Sent: Thursday, April 25, 2019 11:56 AM > > To: dev@nifi.apache.org > > Subject: Re: Adding HBase Support for AtomicDistributedMapCacheClient > > > > Can it not be done with the existing checkAndPut method? [1] > > > > I think if you use the value as the revision it should work. Would be s= imilar to how the Redis implementation works [2]. > > > > [1] > > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-stand > > ard-services/nifi-hbase-client-service-api/src/main/java/org/apache/ni > > fi/hbase/HBaseClientService.java#L65 > > [2] > > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-redis > > -bundle/nifi-redis-extensions/src/main/java/org/apache/nifi/redis/serv > > ice/RedisDistributedMapCacheClientService.java#L271 > > > > On Thu, Apr 25, 2019 at 12:38 PM Shawn Weeks wrote: > > > > > > I'll need to add a check and mutate method to the HBaseClientService = Interface, should I just extend with a HBase2ClientService or add checkAndM= utate to the existing interface and just make it raise an exception if you = try and use it against hbase 1? While Hbase 1.x supports checkAndMutate it = doesn't provide a way to filter on timestamp which is part of how I was goi= ng to implement the revision requirement for AtomicMapCache. > > > > > > Thanks > > > Shawn > > > > > > -----Original Message----- > > > From: Bryan Bende > > > Sent: Thursday, April 25, 2019 9:11 AM > > > To: dev@nifi.apache.org > > > Subject: Re: Adding HBase Support for > > > AtomicDistributedMapCacheClient > > > > > > I'm not aware of a JIRA, so I'd say go for it. > > > > > > On Wed, Apr 24, 2019 at 9:27 PM Shawn Weeks wrote: > > > > > > > > Seems like this should be fairly easy for HBase 2.x with the checkA= ndMutate functionality and I was wondering if there is already a Jira for t= his. Otherwise I might make an attempt at it. It would be good to be able t= o support Wait/Notify and other things that need AtomicDistributedMapCacheC= lient using an Apache developed product commonly found in a Hadoop Cluster. > > > > > > > > Thanks > > > > Shawn