nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre Villard <pierre.villard...@gmail.com>
Subject Re: encryption before writing to disk
Date Fri, 23 Dec 2016 16:42:25 GMT
Kind of.

WriteAheadFlowFileRepository is for FlowFile repository, it only stores
attributes and flow file states [1].
For flow files content, you would need to extend FileSystemRepository.
And if you need to also encrypt provenance data, that's an additional
repository to extend.

Just a quick remark: I didn't say it in my previous mail, but a short term
solution for you could be to use the volatile repositories to keep
everything in memory instead of disks, but be aware of the limitations (in
case of a NiFi restart for example).

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#flowfile-repository

2016-12-23 17:26 GMT+01:00 Knapp, Michael <Michael.Knapp@capitalone.com>:

> Pierre,
>
> OK so let me see if I am interpreting your advice correctly:
> 1. Extend WriteAheadFlowFileRepository.
> 2. Override methods as necessary.
> 3. Make it encrypt everything that goes to disk.  I could have it use
> AWS’s KMS service to do that.  I might even have custom encryption logic
> here.
>
> Is that right?
>
> On 12/23/16, 10:23 AM, "Pierre Villard" <pierre.villard.fr@gmail.com>
> wrote:
>
>     Hi Michael,
>
>     The feature you describe is on the roadmap and you may find more
> details
>     here [1] and/or participate into the discussion. Regarding the
>     implementation you suggest, I think that implementing a custom
> repository
>     instead of a controller service would be an easier approach (no need to
>     change the processors). In any case, this would be a great feature and
> it
>     is possible that Andy already started some developments on this
> subject.
>
>     [1]
>     https://cwiki.apache.org/confluence/display/NIFI/
> Security+Feature+Roadmap
>
>     -Pierre
>
>
>     2016-12-23 15:43 GMT+01:00 Knapp, Michael <
> Michael.Knapp@capitalone.com>:
>
>     > Nifi Developers,
>     >
>     > So I have a somewhat interesting task.  I want to run Nifi on AWS,
> but at
>     > the same time there is a lot of red tape involved with putting data
> on
>     > AWS.  Some data may not be placed on an AWS disk unless it is
> encrypted.
>     > Running Nifi on top of an encrypted EBS is not considered good
> enough in my
>     > case.  The ListenHTTP processor does not let people encrypt content
> before
>     > it is written to disk.  You can encrypt the content downstream, but
> at that
>     > point it has already been written to disk so that is too late.
> People can
>     > encrypt content before it is sent to Nifi, but in some situations
> that may
>     > be very challenging for the developer, as they may have limited
> access to
>     > the source of the data.
>     >
>     > I was thinking of modifying the ListenHTTP processor and other
> similar
>     > processors.  I want to create a ControllerService interface that
> merely
>     > returns a StreamCallback implementation.  The ListenHTTP processor
> would
>     > take this as an optional property.  If that property is set, then the
>     > processor will use that to modify/encrypt content before it is even
> written
>     > to disk.  If the property is not set, then it will operate the same
> way it
>     > does now.
>     >
>     > I looked for a good project to place this controller service
> interface in,
>     > I feel like this service is so basic that it should really be part
> of the
>     > framework’s core, but I don’t see any other controller services
> there.  So
>     > my best guess for now is to put this in the
> nifi-ssl-context-service-(api|bundle)
>     > projects.  I feel like this is not really related to SSL, but that
> is the
>     > only project I found that has controller services listed and is a
>     > dependency of both the nifi-kafka projects and the
> nifi-standard-processors
>     > projects.  I think it would be a waste to set up a new api/bundle
> pair just
>     > for one interface.
>     >
>     > So my questions are:
>     >
>     > 1.       Do you think this is a good idea?
>     >
>     > 2.       Where should I put this code if I write it?
>     >
>     > Michael Knapp
>     > Capital One
>     > ________________________________________________________
>     >
>     > The information contained in this e-mail is confidential and/or
>     > proprietary to Capital One and/or its affiliates and may only be used
>     > solely in performance of work or services for Capital One. The
> information
>     > transmitted herewith is intended only for use by the individual or
> entity
>     > to which it is addressed. If the reader of this message is not the
> intended
>     > recipient, you are hereby notified that any review, retransmission,
>     > dissemination, distribution, copying or other use of, or taking of
> any
>     > action in reliance upon this information is strictly prohibited. If
> you
>     > have received this communication in error, please contact the sender
> and
>     > delete the material from your computer.
>     >
>
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message