nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Knapp, Michael" <Michael.Kn...@capitalone.com>
Subject Re: encryption before writing to disk
Date Fri, 23 Dec 2016 16:46:51 GMT
Great advice!  I just might wind up using that.

Is there any way to have some processors use one repository and other processors use the default
repository?  If not, then I may need to have two separate nifi instances running for that.

On 12/23/16, 11:42 AM, "Pierre Villard" <pierre.villard.fr@gmail.com> wrote:

    Kind of.
    
    WriteAheadFlowFileRepository is for FlowFile repository, it only stores
    attributes and flow file states [1].
    For flow files content, you would need to extend FileSystemRepository.
    And if you need to also encrypt provenance data, that's an additional
    repository to extend.
    
    Just a quick remark: I didn't say it in my previous mail, but a short term
    solution for you could be to use the volatile repositories to keep
    everything in memory instead of disks, but be aware of the limitations (in
    case of a NiFi restart for example).
    
    [1]
    https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#flowfile-repository
    
    2016-12-23 17:26 GMT+01:00 Knapp, Michael <Michael.Knapp@capitalone.com>:
    
    > Pierre,
    >
    > OK so let me see if I am interpreting your advice correctly:
    > 1. Extend WriteAheadFlowFileRepository.
    > 2. Override methods as necessary.
    > 3. Make it encrypt everything that goes to disk.  I could have it use
    > AWS’s KMS service to do that.  I might even have custom encryption logic
    > here.
    >
    > Is that right?
    >
    > On 12/23/16, 10:23 AM, "Pierre Villard" <pierre.villard.fr@gmail.com>
    > wrote:
    >
    >     Hi Michael,
    >
    >     The feature you describe is on the roadmap and you may find more
    > details
    >     here [1] and/or participate into the discussion. Regarding the
    >     implementation you suggest, I think that implementing a custom
    > repository
    >     instead of a controller service would be an easier approach (no need to
    >     change the processors). In any case, this would be a great feature and
    > it
    >     is possible that Andy already started some developments on this
    > subject.
    >
    >     [1]
    >     https://cwiki.apache.org/confluence/display/NIFI/
    > Security+Feature+Roadmap
    >
    >     -Pierre
    >
    >
    >     2016-12-23 15:43 GMT+01:00 Knapp, Michael <
    > Michael.Knapp@capitalone.com>:
    >
    >     > Nifi Developers,
    >     >
    >     > So I have a somewhat interesting task.  I want to run Nifi on AWS,
    > but at
    >     > the same time there is a lot of red tape involved with putting data
    > on
    >     > AWS.  Some data may not be placed on an AWS disk unless it is
    > encrypted.
    >     > Running Nifi on top of an encrypted EBS is not considered good
    > enough in my
    >     > case.  The ListenHTTP processor does not let people encrypt content
    > before
    >     > it is written to disk.  You can encrypt the content downstream, but
    > at that
    >     > point it has already been written to disk so that is too late.
    > People can
    >     > encrypt content before it is sent to Nifi, but in some situations
    > that may
    >     > be very challenging for the developer, as they may have limited
    > access to
    >     > the source of the data.
    >     >
    >     > I was thinking of modifying the ListenHTTP processor and other
    > similar
    >     > processors.  I want to create a ControllerService interface that
    > merely
    >     > returns a StreamCallback implementation.  The ListenHTTP processor
    > would
    >     > take this as an optional property.  If that property is set, then the
    >     > processor will use that to modify/encrypt content before it is even
    > written
    >     > to disk.  If the property is not set, then it will operate the same
    > way it
    >     > does now.
    >     >
    >     > I looked for a good project to place this controller service
    > interface in,
    >     > I feel like this service is so basic that it should really be part
    > of the
    >     > framework’s core, but I don’t see any other controller services
    > there.  So
    >     > my best guess for now is to put this in the
    > nifi-ssl-context-service-(api|bundle)
    >     > projects.  I feel like this is not really related to SSL, but that
    > is the
    >     > only project I found that has controller services listed and is a
    >     > dependency of both the nifi-kafka projects and the
    > nifi-standard-processors
    >     > projects.  I think it would be a waste to set up a new api/bundle
    > pair just
    >     > for one interface.
    >     >
    >     > So my questions are:
    >     >
    >     > 1.       Do you think this is a good idea?
    >     >
    >     > 2.       Where should I put this code if I write it?
    >     >
    >     > Michael Knapp
    >     > Capital One
    >     > ________________________________________________________
    >     >
    >     > The information contained in this e-mail is confidential and/or
    >     > proprietary to Capital One and/or its affiliates and may only be used
    >     > solely in performance of work or services for Capital One. The
    > information
    >     > transmitted herewith is intended only for use by the individual or
    > entity
    >     > to which it is addressed. If the reader of this message is not the
    > intended
    >     > recipient, you are hereby notified that any review, retransmission,
    >     > dissemination, distribution, copying or other use of, or taking of
    > any
    >     > action in reliance upon this information is strictly prohibited. If
    > you
    >     > have received this communication in error, please contact the sender
    > and
    >     > delete the material from your computer.
    >     >
    >
    >
    > ________________________________________________________
    >
    > The information contained in this e-mail is confidential and/or
    > proprietary to Capital One and/or its affiliates and may only be used
    > solely in performance of work or services for Capital One. The information
    > transmitted herewith is intended only for use by the individual or entity
    > to which it is addressed. If the reader of this message is not the intended
    > recipient, you are hereby notified that any review, retransmission,
    > dissemination, distribution, copying or other use of, or taking of any
    > action in reliance upon this information is strictly prohibited. If you
    > have received this communication in error, please contact the sender and
    > delete the material from your computer.
    >
    

________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.
Mime
View raw message