apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Farkas <...@datatorrent.com>
Subject Re: Encrypted Streams
Date Tue, 15 Dec 2015 18:28:54 GMT
Found this, thought it was relevant and useful.

http://blog.cloudera.com/blog/2013/03/how-to-set-up-a-hadoop-cluster-with-network-encryption/
<http://www.google.com/url?q=http%3A%2F%2Fblog.cloudera.com%2Fblog%2F2013%2F03%2Fhow-to-set-up-a-hadoop-cluster-with-network-encryption%2F&sa=D&sntz=1&usg=AFQjCNHrWGUnzE4STD6eRpEaiIwYIhVHPA>

Thanks,
Tim

On Tue, Dec 15, 2015 at 9:07 AM, Amol Kekre <amol@datatorrent.com> wrote:

> Encrytion over wire within Hadoop may be a legal requirement. This does
> mean that the customer understands performance penalty. As a feature though
> it will help some customers ger security certification much easier. If we
> have both per-stream attribute and app-wide attribute, users can choose the
> one they want.
>
> Thks,
> Amol
>
>
> On Tue, Dec 15, 2015 at 6:51 AM, Aniruddha Thombare <
> aniruddha@datatorrent.com> wrote:
>
> > +1 For encrypted streams...
> >
> >
> > Thanks,
> >
> >
> > Aniruddha
> >
> > On Tue, Dec 15, 2015 at 7:43 PM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com>
> > wrote:
> >
> > > Agreed. Some really good points.
> > >
> > > But first I want to ask community whether encrypted data flow over
> stream
> > > sound like a good and useful addition to apex platform OR not.
> > > If yes, I can create a Jira for the same and then we can probably
> > continue
> > > the discussion for requirement and design there.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > > ~ Chinmay.
> > >
> > > On Tue, Dec 15, 2015 at 5:55 PM, Pramod Immaneni <
> pramod@datatorrent.com
> > >
> > > wrote:
> > >
> > > > Looks like one would need to configure the EncryptedInputPort with
> the
> > > > encryption algorithm and related input parameters so it is not very
> > > > automatic in the end isn't it as compared to setting a StreamCodec.
> > Also
> > > > not all streams in a DAG may need to be encrypted, some streams
> between
> > > > operators may carry sensitive information and others may not. There
> is
> > a
> > > > performance penalty to encryption so a DAG level attribute may be too
> > > > restrictive.
> > > >
> > > > There is a point to be made about operator level setting as opposed
> to
> > > > doing the encryption in the StreamCodec. Doing it in the StreamCodec
> > > > encrypts only the operator data but not all communication between the
> > > > operators. Furthermore, if done in StreamCodec, at every checkpoint
> the
> > > > encryption state would have to be reset providing attackers
> information
> > > > some boundary information. If the encryption is handled at the
> > transport
> > > > layer, like using SSL for example, this problem is taken care of.
> There
> > > > still may be other uses of StreamCodec for data conversion like
> > > compression
> > > > for example.
> > > >
> > > > Chinmay,
> > > >
> > > >
> > > >
> > > > On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <
> tim@datatorrent.com>
> > > > wrote:
> > > >
> > > > > I think encryption of data sent across the wire and operator logic
> > are
> > > > > orthogonal. The user should just have to set DAG level attribute
to
> > > > > enable/disable encryption, without having to write any encryption
> > > related
> > > > > code. I think this would require changes to the Buffer Server
> > publisher
> > > > and
> > > > > subscriber though.
> > > > >
> > > > > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <
> > > chandni@datatorrent.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > When we are dealing with secured data, the usual scenarios are
> that
> > > you
> > > > > get
> > > > > > encrypted data.
> > > > > > This data need to decrypt and then perform other functions on
it.
> > The
> > > > > > output of the dag is then encrypted.
> > > > > >
> > > > > > In the past we have solved these use cases by performing
> > > > > > decryption/encryption in the operator.
> > > > > > IMO the operator approach works better because these processes
> may
> > > > > require
> > > > > > invoking utilities and also operators can be configured easily
> > using
> > > > > > properties.
> > > > > >
> > > > > > Chandni
> > > > > >
> > > > > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> > > > sandesh@datatorrent.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Well we have committers from bank, their feedback will
be
> really
> > > > > > valuable.
> > > > > > >
> > > > > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> > > > > > priyanka@datatorrent.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Sounds good. This is good feature for banks and security
> > domain.
> > > > > > > > One suggestion: We can do key management ourself at
> application
> > > > (may
> > > > > be
> > > > > > > by
> > > > > > > > providing default keys) and there should be an option
to
> > override
> > > > > keys
> > > > > > if
> > > > > > > > user really want to do so.
> > > > > > > >
> > > > > > > > -Priyanka
> > > > > > > >
> > > > > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar
<
> > > > > > > > chinmay@datatorrent.com
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > I wanted to propose an idea using which one can
have
> > encrypted
> > > > > stream
> > > > > > > > > flowing in a DAG.
> > > > > > > > >
> > > > > > > > > Basically, the idea is to create a new EncryptedInputPort
> > which
> > > > > will
> > > > > > > > extend
> > > > > > > > > from DefaultInputPort and will return a StreamCodec
object
> > > which
> > > > > will
> > > > > > > > take
> > > > > > > > > care of encryption/decryption.
> > > > > > > > > As the same StreamCodec object will be used at
OutputPort,
> > the
> > > > > > > encryption
> > > > > > > > > can be done in toByteArray method at Output port
and
> > decryption
> > > > can
> > > > > > be
> > > > > > > > done
> > > > > > > > > in fromByteArray at Input port.
> > > > > > > > >
> > > > > > > > > By default we can support some basic encryption
algorithms
> > like
> > > > RSA
> > > > > > and
> > > > > > > > DSA
> > > > > > > > > where user need to provide the key(s) to
> EncryptedInputPort.
> > > > > > > > >
> > > > > > > > > Any thoughts?
> > > > > > > > >
> > > > > > > > > ~ Chinmay.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message