flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Max Michels <...@apache.org>
Subject Re: Contributing to Flink
Date Fri, 27 Feb 2015 09:19:44 GMT
Hi Niraj,

Pleased to here you want to start contributing to Flink :)

In terms of security, there are some open issues. Like Robert
metioned, it would be great if you could implement proper HDFS
Kerberos authentication. Basically, the HDFS Delegation Token needs to
be transferred to the workers so that they don't have to be
authenticated themselves. Also, proper renewal of the Token needs to
be taken care of.

Another security task could be to implement authentication support in
Flink. This could be done by asking the user for some kind of shared
secret. Alternatively, authentication could also be performed by
Kerberos' authentication method.

Let me know what you find interesting.

Best regards,

On Fri, Feb 27, 2015 at 2:31 AM, Rai, Niraj <niraj.rai@intel.com> wrote:
> Hi Robert,
> Thanks for the detailed response. I worked on the encryption of HDFS as well as the crypto
file system in HDFS, so, I am aware of how it is done in Hadoop. Let me sync up with Max to
get started on it.
> I will also start looking into the current implementations.
> Niraj
> From: Robert Metzger [mailto:rmetzger@apache.org]
> Sent: Thursday, February 26, 2015 3:11 PM
> To: dev@flink.apache.org
> Cc: Rai, Niraj
> Subject: Re: Contributing to Flink
> Hi Niraj,
> Welcome to the Flink community ;)
> I'm really excited that you want to contribute to our project, and since you've asked
for something in the security area, I actually have something very concrete in mind.
> We recently added some support for accessing (Kerberos) secured HDFS clusters in Flink:
> However, the implementation is very simple because it assumes that every worker of Flink
(TaskManager) is authenticated with Kerberos (kinit). Its not very practical for large setups
because you have to ssh to all machines to log into Kerberos.
> What I would really like to have in Flink would be an way to transfer the authentication
tokens form the JobManager (master) to the TaskManagers. This way, users only have to be authenticated
with Kerberos at the JobManager, and Flink is taking care of the rest.
> As far as I understood it, Hadoop has already all the utilities in place for getting
and transferring the delegation tokens.
> Max Michels, another committer in our project has quite a good understanding of the details
there. It would be great if you (Max) could chime in if I forgot something.
> If you are interested in working on this, you can file a JIRA (https://issues.apache.org/jira/browse/FLINK)
for tracking the progress and discussing the details.
> If not I'm sure we'll come up with more interesting ideas.
> Robert
> On Thu, Feb 26, 2015 at 11:07 PM, Henry Saputra <henry.saputra@gmail.com<mailto:henry.saputra@gmail.com>>
> Hi Niraj,
> Thanks for your interest at Apache Flink. The quickest is to just give
> Flink a spin and figure out how it works.
> This would get you start on how it works before actually doing work on Flink =)
> Please do visit Flink how to contribute page [1] and subscribe to dev
> mailing list [2] to start following up.
> Welcome =)
> [1] http://flink.apache.org/how-to-contribute.html
> [2] http://flink.apache.org/community.html#mailing-lists
> On Thu, Feb 26, 2015 at 1:45 PM, Rai, Niraj <niraj.rai@intel.com<mailto:niraj.rai@intel.com>>
>> Hi Flink Dev,
>> I am looking to contribute to Flink, especially in the area of security. In the past,
I have contributed to Pig, Hive and HDFS. I would really appreciate, if I can get some work
assigned to me. Looking forward to hear back from the development community of Flink.
>> Thanks
>> Niraj

View raw message