From dev-return-20147-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Thu Nov 14 14:17:22 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id DF7B3180607 for ; Thu, 14 Nov 2019 15:17:21 +0100 (CET) Received: (qmail 49627 invoked by uid 500); 14 Nov 2019 14:17:21 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 49615 invoked by uid 99); 14 Nov 2019 14:17:20 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.42) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Nov 2019 14:17:20 +0000 Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id AD47C1003 for ; Thu, 14 Nov 2019 14:17:20 +0000 (UTC) Received: by mail-ot1-f45.google.com with SMTP id w24so4480810otk.6 for ; Thu, 14 Nov 2019 06:17:20 -0800 (PST) X-Gm-Message-State: APjAAAWjxkvUiXocWjYUf0PgNZAdLB/isIQqEp/eivRcbsotFvFFC0fv C1mCrfmXtQx0rrqe3hz/cD8lsGvz6CNGluJwveA= X-Google-Smtp-Source: APXvYqzAs/iGxqY+8Fk6dxRrJxI7GRbXTMnRHwKEnBimRV1er1gheMz4exspPylkG9/q5KIUGGSCLdlHSUdIQ8icMvE= X-Received: by 2002:a9d:4c15:: with SMTP id l21mr7951465otf.204.1573741040251; Thu, 14 Nov 2019 06:17:20 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Matt Burgess Date: Thu, 14 Nov 2019 09:16:44 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: New Distributed Map Cache Impementations To: dev@nifi.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Shawn, There are also Redis and Couchbase distributed map cache clients already in NiFi. I don't see any Jiras or PRs related to a DynamoDB or JDBC ones. I thought about making ones for JDBC, Hazelcast and/or Nitrite [1] (with or without a DataGate server), but never got around to it. I think DynamoDB and JDBC implementations would be helpful, the latter could support DynamoDB in the meantime using the Simba JDBC driver [2]. While thinking about the JDBC one, I figured it might be nice to be able to cache the table locally for X amount of time or N number of entries, in case you pre-populate the cache and are just reading it with the client. Any write (from the NiFi client) would invalidate the cache and the table would be re-fetched on the next read operation. I did something similar for the DatabaseRecordLookupService, but that's a read-only service so I didn't have to worry about writes, I was just trying to improve performance where possible. Regards, Matt [1] https://www.dizitart.org/nitrite-database.html [2] https://www.simba.com/drivers/dynamodb-odbc-jdbc/ On Thu, Nov 14, 2019 at 8:31 AM Shawn Weeks wro= te: > > Has anyone already done some work on adding new services for distribute m= ap cache? I=E2=80=99m looking at moving to aws and I really don=E2=80=99t w= ant to have to run emr just for hbase. I=E2=80=99ve been thinking about sta= rting on either a DynamoDB or simple jdbc implementation. > > Thanks > Shawn > > Sent from my iPhone