apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: Encrypted Streams
Date Tue, 15 Dec 2015 17:07:03 GMT
Encrytion over wire within Hadoop may be a legal requirement. This does
mean that the customer understands performance penalty. As a feature though
it will help some customers ger security certification much easier. If we
have both per-stream attribute and app-wide attribute, users can choose the
one they want.

Thks,
Amol


On Tue, Dec 15, 2015 at 6:51 AM, Aniruddha Thombare <
aniruddha@datatorrent.com> wrote:

> +1 For encrypted streams...
>
>
> Thanks,
>
>
> Aniruddha
>
> On Tue, Dec 15, 2015 at 7:43 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com>
> wrote:
>
> > Agreed. Some really good points.
> >
> > But first I want to ask community whether encrypted data flow over stream
> > sound like a good and useful addition to apex platform OR not.
> > If yes, I can create a Jira for the same and then we can probably
> continue
> > the discussion for requirement and design there.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > ~ Chinmay.
> >
> > On Tue, Dec 15, 2015 at 5:55 PM, Pramod Immaneni <pramod@datatorrent.com
> >
> > wrote:
> >
> > > Looks like one would need to configure the EncryptedInputPort with the
> > > encryption algorithm and related input parameters so it is not very
> > > automatic in the end isn't it as compared to setting a StreamCodec.
> Also
> > > not all streams in a DAG may need to be encrypted, some streams between
> > > operators may carry sensitive information and others may not. There is
> a
> > > performance penalty to encryption so a DAG level attribute may be too
> > > restrictive.
> > >
> > > There is a point to be made about operator level setting as opposed to
> > > doing the encryption in the StreamCodec. Doing it in the StreamCodec
> > > encrypts only the operator data but not all communication between the
> > > operators. Furthermore, if done in StreamCodec, at every checkpoint the
> > > encryption state would have to be reset providing attackers information
> > > some boundary information. If the encryption is handled at the
> transport
> > > layer, like using SSL for example, this problem is taken care of. There
> > > still may be other uses of StreamCodec for data conversion like
> > compression
> > > for example.
> > >
> > > Chinmay,
> > >
> > >
> > >
> > > On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <tim@datatorrent.com>
> > > wrote:
> > >
> > > > I think encryption of data sent across the wire and operator logic
> are
> > > > orthogonal. The user should just have to set DAG level attribute to
> > > > enable/disable encryption, without having to write any encryption
> > related
> > > > code. I think this would require changes to the Buffer Server
> publisher
> > > and
> > > > subscriber though.
> > > >
> > > > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <
> > chandni@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > When we are dealing with secured data, the usual scenarios are that
> > you
> > > > get
> > > > > encrypted data.
> > > > > This data need to decrypt and then perform other functions on it.
> The
> > > > > output of the dag is then encrypted.
> > > > >
> > > > > In the past we have solved these use cases by performing
> > > > > decryption/encryption in the operator.
> > > > > IMO the operator approach works better because these processes may
> > > > require
> > > > > invoking utilities and also operators can be configured easily
> using
> > > > > properties.
> > > > >
> > > > > Chandni
> > > > >
> > > > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> > > sandesh@datatorrent.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Well we have committers from bank, their feedback will be really
> > > > > valuable.
> > > > > >
> > > > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> > > > > priyanka@datatorrent.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Sounds good. This is good feature for banks and security
> domain.
> > > > > > > One suggestion: We can do key management ourself at application
> > > (may
> > > > be
> > > > > > by
> > > > > > > providing default keys) and there should be an option to
> override
> > > > keys
> > > > > if
> > > > > > > user really want to do so.
> > > > > > >
> > > > > > > -Priyanka
> > > > > > >
> > > > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > > > > > > chinmay@datatorrent.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > I wanted to propose an idea using which one can have
> encrypted
> > > > stream
> > > > > > > > flowing in a DAG.
> > > > > > > >
> > > > > > > > Basically, the idea is to create a new EncryptedInputPort
> which
> > > > will
> > > > > > > extend
> > > > > > > > from DefaultInputPort and will return a StreamCodec
object
> > which
> > > > will
> > > > > > > take
> > > > > > > > care of encryption/decryption.
> > > > > > > > As the same StreamCodec object will be used at OutputPort,
> the
> > > > > > encryption
> > > > > > > > can be done in toByteArray method at Output port and
> decryption
> > > can
> > > > > be
> > > > > > > done
> > > > > > > > in fromByteArray at Input port.
> > > > > > > >
> > > > > > > > By default we can support some basic encryption algorithms
> like
> > > RSA
> > > > > and
> > > > > > > DSA
> > > > > > > > where user need to provide the key(s) to EncryptedInputPort.
> > > > > > > >
> > > > > > > > Any thoughts?
> > > > > > > >
> > > > > > > > ~ Chinmay.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message