hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jitendra Pandey <jiten...@hortonworks.com>
Subject Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Date Mon, 23 Oct 2017 18:10:41 GMT
I have filed https://issues.apache.org/jira/browse/HDFS-12697 to ensure ozone stays disabled
in a secure environment.
Since ozone is disabled by default and will not come with security on, it will not expose
any new attack surface in a Hadoop deployment.
Ozone security effort will need a detailed design and discussion on a community jira. Hopefully,
that effort will start soon after the merge.

Thanks
jitendra

On 10/20/17, 2:40 PM, "larry mccay" <lmccay@apache.org> wrote:

    All -
    
    I broke this list of questions out into a separate DISCUSS thread where we
    can iterate over how a security audit process at merge time might look and
    whether it is even something that we want to take on.
    
    I will try and continue discussion on that thread and drive that to some
    conclusion before bringing it into any particular merge discussion.
    
    thanks,
    
    --larry
    
    On Fri, Oct 20, 2017 at 12:37 PM, larry mccay <lmccay@apache.org> wrote:
    
    > I previously sent this same email from my work email and it doesn't seem
    > to have gone through - resending from apache account (apologizing up from
    > for the length)....
    >
    > For such sizable merges in Hadoop, I would like to start doing security
    > audits in order to have an initial idea of the attack surface, the
    > protections available for known threats, what sort of configuration is
    > being used to launch processes, etc.
    >
    > I dug into the architecture documents while in the middle of this list -
    > nice docs!
    > I do intend to try and make a generic check list like this for such
    > security audits in the future so a lot of this is from that but I tried to
    > also direct specific questions from those docs as well.
    >
    > 1. UIs
    > I see there are at least two UIs - Storage Container Manager and Key Space
    > Manager. There are a number of typical vulnerabilities that we find in UIs
    >
    > 1.1. What sort of validation is being done on any accepted user input?
    > (pointers to code would be appreciated)
    > 1.2. What explicit protections have been built in for (pointers to code
    > would be appreciated):
    >   1.2.1. cross site scripting
    >   1.2.2. cross site request forgery
    >   1.2.3. click jacking (X-Frame-Options)
    > 1.3. What sort of authentication is required for access to the UIs?
    > 1.4. What authorization is available for determining who can access what
    > capabilities of the UIs for either viewing, modifying data or affecting
    > object stores and related processes?
    > 1.5. Are the UIs built with proxying in mind by leveraging X-Forwarded
    > headers?
    > 1.6. Is there any input that will ultimately be persisted in configuration
    > for executing shell commands or processes?
    > 1.7. Do the UIs support the trusted proxy pattern with doas impersonation?
    > 1.8. Is there TLS/SSL support?
    >
    > 2. REST APIs
    >
    > 2.1. Do the REST APIs support the trusted proxy pattern with doas
    > impersonation capabilities?
    > 2.2. What explicit protections have been built in for:
    >   2.2.1. cross site scripting (XSS)
    >   2.2.2. cross site request forgery (CSRF)
    >   2.2.3. XML External Entity (XXE)
    > 2.3. What is being used for authentication - Hadoop Auth Module?
    > 2.4. Are there separate processes for the HTTP resources (UIs and REST
    > endpoints) or are the part of existing HDFS processes?
    > 2.5. Is there TLS/SSL support?
    > 2.6. Are there new CLI commands and/or clients for access the REST APIs?
    > 2.7. Bucket Level API allows for setting of ACLs on a bucket - what
    > authorization is required here - is there a restrictive ACL set on creation?
    > 2.8. Bucket Level API allows for deleting a bucket - I assume this is
    > dependent on ACLs based access control?
    > 2.9. Bucket Level API to list bucket returns up to 1000 keys - is there
    > paging available?
    > 2.10. Storage Level APIs indicate “Signed with User Authorization” what
    > does this refer to exactly?
    > 2.11. Object Level APIs indicate that there is no ACL support and only
    > bucket owners can read and write - but there are ACL APIs on the Bucket
    > Level are they meaningless for now?
    > 2.12. How does a REST client know which Ozone Handler to connect to or am
    > I missing some well known NN type endpoint in the architecture doc
    > somewhere?
    >
    > 3. Encryption
    >
    > 3.1. Is there any support for encryption of persisted data?
    > 3.2. If so, is KMS and the hadoop key command used for key management?
    >
    > 4. Configuration
    >
    > 4.1. Are there any passwords or secrets being added to configuration?
    > 4.2. If so, are they accessed via Configuration.getPassword() to allow for
    > provisioning in credential providers?
    > 4.3. Are there any settings that are used to launch docker containers or
    > shell out any commands, etc?
    >
    > 5. HA
    >
    > 5.1. Are there provisions for HA?
    > 5.2. Are we leveraging the existing HA capabilities in HDFS?
    > 5.3. Is Storage Container Manager a SPOF?
    > 5.4. I see HA listed in future work in the architecture doc - is this
    > still an open issue?
    >
    > On Fri, Oct 20, 2017 at 11:19 AM, Anu Engineer <aengineer@hortonworks.com>
    > wrote:
    >
    >> Hi Steve,
    >>
    >> In addition to everything Weiwei mentioned (chapter 3 of user guide), if
    >> you really want to drill down to REST protocol you might want to apply this
    >> patch and build ozone.
    >>
    >> https://issues.apache.org/jira/browse/HDFS-12690
    >>
    >> This will generate an Open API (https://www.openapis.org ,
    >> http://swagger.io) based specification which can be accessed from KSM UI
    >> or just as a json file.
    >> Unfortunately, this patch is still at code review stage, so you will have
    >> to apply the patch and build it yourself.
    >>
    >> Thanks
    >> Anu
    >>
    >>
    >> On 10/20/17, 6:09 AM, "Yang Weiwei" <cheersyang@hotmail.com> wrote:
    >>
    >>     Hi Steve
    >>
    >>
    >>     The code is available in HDFS-7240 feature branch, public git repo
    >> here<https://github.com/apache/hadoop/tree/HDFS-7240>.
    >>
    >>     I am not sure if there is a "public" API for object stores, but the
    >> design doc<https://issues.apache.org/jira/secure/attachment/1279954
    >> 9/ozone_user_v0.pdf> uses most common syntax so I believe it should be
    >> compliance. You can find the rest API doc here<https://github.com/apache
    >> /hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/
    >> site/markdown/OzoneRest.md> (with some example usages), and commandline
    >> API here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop-
    >> hdfs-project/hadoop-hdfs/src/site/markdown/OzoneCommandShell.md>.
    >>
    >>
    >>     Look forward for your feedback!
    >>
    >>
    >>     --Weiwei
    >>
    >>
    >>     ________________________________
    >>     发件人: Steve Loughran <stevel@hortonworks.com>
    >>     发送时间: 2017年10月20日 11:49
    >>     收件人: Yang Weiwei
    >>     抄送: hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org;
    >> yarn-dev@hadoop.apache.org; common-dev@hadoop.apache.org
    >>     主题: Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
    >>
    >>
    >>     Wow, big piece of work
    >>
    >>     1. Where is a PR/branch on github with rendered docs for us to look
    >> at?
    >>     2. Have you made any public APi changes related to object stores?
    >> That's probably something I'll have opinions on more than implementation
    >> details.
    >>
    >>     thanks
    >>
    >>     > On 19 Oct 2017, at 02:54, Yang Weiwei <cheersyang@hotmail.com>
    >> wrote:
    >>     >
    >>     > Hello everyone,
    >>     >
    >>     >
    >>     > I would like to start this thread to discuss merging Ozone
    >> (HDFS-7240) to trunk. This feature implements an object store which can
    >> co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with
    >> cluster sizes varying from 1 to 100 data nodes.
    >>     >
    >>     >
    >>     >
    >>     > The merge payload includes the following:
    >>     >
    >>     >  1.  All services, management scripts
    >>     >  2.  Object store APIs, exposed via both REST and RPC
    >>     >  3.  Master service UIs, command line interfaces
    >>     >  4.  Pluggable pipeline Integration
    >>     >  5.  Ozone File System (Hadoop compatible file system
    >> implementation, passes all FileSystem contract tests)
    >>     >  6.  Corona - a load generator for Ozone.
    >>     >  7.  Essential documentation added to Hadoop site.
    >>     >  8.  Version specific Ozone Documentation, accessible via service
    >> UI.
    >>     >  9.  Docker support for ozone, which enables faster development
    >> cycles.
    >>     >
    >>     >
    >>     > To build Ozone and run ozone using docker, please follow
    >> instructions in this wiki page. https://cwiki.apache.org/confl
    >> uence/display/HADOOP/Dev+cluster+with+docker.
    >>     Dev cluster with docker - Hadoop - Apache Software Foundation<
    >> https://cwiki.apache.org/confluence/display/HADOO
    >> P/Dev+cluster+with+docker>
    >>     cwiki.apache.org
    >>     First, it uses a much more smaller common image which doesn't
    >> contains Hadoop. Second, the real Hadoop should be built from the source
    >> and the dist director should be ...
    >>
    >>
    >>
    >>     >
    >>     >
    >>     > We have built a passionate and diverse community to drive this
    >> feature development. As a team, we have achieved significant progress in
    >> past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far,
    >> we have resolved almost 400 JIRAs by 20+ contributors/committers from
    >> different countries and affiliations. We also want to thank the large
    >> number of community members who were supportive of our efforts and
    >> contributed ideas and participated in the design of ozone.
    >>     >
    >>     >
    >>     > Please share your thoughts, thanks!
    >>     >
    >>     >
    >>     > -- Weiwei Yang
    >>
    >>
    >>
    >>
    >> ---------------------------------------------------------------------
    >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
    >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
    >>
    >
    >
    

Mime
View raw message