nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Burgess <mattyb...@apache.org>
Subject Re: New Distributed Map Cache Impementations
Date Thu, 14 Nov 2019 14:16:44 GMT
Shawn,

There are also Redis and Couchbase distributed map cache clients
already in NiFi. I don't see any Jiras or PRs related to a DynamoDB or
JDBC ones. I thought about making ones for JDBC, Hazelcast and/or
Nitrite [1] (with or without a DataGate server), but never got around
to it. I think DynamoDB and JDBC implementations would be helpful, the
latter could support DynamoDB in the meantime using the Simba JDBC
driver [2].

While thinking about the JDBC one, I figured it might be nice to be
able to cache the table locally for X amount of time or N number of
entries, in case you pre-populate the cache and are just reading it
with the client. Any write (from the NiFi client) would invalidate the
cache and the table would be re-fetched on the next read operation. I
did something similar for the DatabaseRecordLookupService, but that's
a read-only service so I didn't have to worry about writes, I was just
trying to improve performance where possible.

Regards,
Matt

[1] https://www.dizitart.org/nitrite-database.html
[2] https://www.simba.com/drivers/dynamodb-odbc-jdbc/

On Thu, Nov 14, 2019 at 8:31 AM Shawn Weeks <sweeks@weeksconsulting.us> wrote:
>
> Has anyone already done some work on adding new services for distribute map cache? I’m
looking at moving to aws and I really don’t want to have to run emr just for hbase. I’ve
been thinking about starting on either a DynamoDB or simple jdbc implementation.
>
> Thanks
> Shawn
>
> Sent from my iPhone

Mime
View raw message