nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Knapp, Michael" <>
Subject Re: [DISCUSS] Encrypted repositories (content, flowfile, provenance)
Date Tue, 24 Jan 2017 17:11:38 GMT

I and several of my co-workers are very interested in seeing this feature added to NIFI. 
I have been trying to write an EncryptedFileSystemRepository myself but unfortunately did
not get very far before running into issues.

So far I just tried a simple solution: I essentially extended FileSystemRepository, and overrode
the "read" and "write" methods to wrap them with java's CipherInputStream or CipherOutputStream.
 I also added ways to pass in a secret key.  If that had worked, I would have updated it to
use AWS's KMS service to provide secret keys.  I do think we would need to roll keys periodically,
but still haven't figured out how that would work.

I ran into a lot of classpath issues.  First I tried making a thin jar and putting it on NIFI's
lib, but unfortunately NIFI's classloader ensured that my jar did not see any of nifi's framework
classes.  I tried making an uber jar but encountered a lot of other classpath issues.  I tried
making a nar, but NIFI still could not find my EncryptedRepository when spring was wiring
up the application.  It seems you can only extend repositories from within the source code.
 So I tried putting my code into the source and re-building.  Unfortunately my work's proxy
prevented me from getting some essential apache artifacts so I am unable to build NIFI from
source.  I might try this again from a personal account.

So it seems that with NIFI it is not easy to implement things that are not processors or controller
services, since you have to add that to the source and re-build it.  Perhaps NIFI needs an
easier way to add these implementations without re-building from source.

Since I don't want to put the entire AWS core artifact and its transitive dependencies inside
NIFI, I was thinking of using java's SPI framework to provide encryption services to the EncryptedRepository.

I'm not sure I really answered any of your questions, but hopefully I gave you an idea of
how we might use this.  I might be available to help develop and/or test the solution.

Michael Knapp

Capital One

From: Andy LoPresto <>
Sent: Tuesday, January 24, 2017 12:13:07 AM
Subject: [DISCUSS] Encrypted repositories (content, flowfile, provenance)

I am working on building drop-in encrypted repositories for NiFi [1]. I am currently in the
planning stages and have written up a fairly extensive Jira ticket documenting my plans (high-level)
and some concerns (much bigger section). I would welcome community feedback in order to capture
expectations, concerns (for security, performance, and usability), and any other valuable
contributions. All of us are smarter than one of us, and I am not that smart. Thanks.


Andy LoPresto<><>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message