hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haohui Mai <h...@hortonworks.com>
Subject Re: Replacing the JSP web UIs to HTML 5 applications
Date Tue, 29 Oct 2013 00:48:49 GMT
It seems more appealing to me that the UI should JMX directly, because:

* We're support the JMX in the long term for other management software.
* The information provided by the JMX API will be mostly identical of the
JSON API. Today the Web UI covers most of the information provided by JMX.
The Web UI does some trivial work to extract the information and renders it
as HTML.
* We can compatibility and unit tests for free.

I do agree that the JMX APIs are imperfect and they should be revisited in
the 3.0 timeframe. However, this is orthogonal of the discussions of
transitioning from JSP-based Web UI to client-side JavaScript Web UI. The
architecture of the new Web UI allows easy migration to any JSON-based APIs
whenever they land in the trunk.

Thanks,
Haohui


On Mon, Oct 28, 2013 at 5:13 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:

> Isn't using JMX to expose JSON for the web UI misusing JMX?
>
> I would think a more appropriate approach would be having /JMX for
> monitoring integration and a /JSON end point for the UI data.
>
> Thanks.
>
>
> On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hmai@hortonworks.com> wrote:
>
> > Alejandro,
> >
> > If I understand correctly, that is the exact approach that the new web UI
> > is taking. The new web UI takes the output from JMX and renders them as
> > HTML at the client side.
> >
> > ~Haohui
> >
> >
> > On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > Haohui,
> > >
> > > If you have NN and DNs producing JSON instead HTML, then you can build
> JS
> > > based web UIs. Take for example Oozie, Oozie produces JSON, it has a
> > built
> > > in JS web ui that consumes JSON and Hue has built an external web UI
> that
> > > also consumes JSON. In the case of Hue UI, Oozie didn't have to change
> > > anything to get that UI and improvements on the Hue UI don't require
> > > changes in Oozie unless it is to produce additional information.
> > >
> > > hope this clarifies.
> > >
> > > Thx
> > >
> > >
> > > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hmai@hortonworks.com>
> > wrote:
> > >
> > > > Echo my comments on HDFS-5402:
> > > >
> > > > bq. If we're going to remove the old web UI, I think the new web UI
> has
> > > > to have the same level of unit testing. We shouldn't go backwards in
> > > > terms of unit testing.
> > > >
> > > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> > > > TestClusterJspHelper. It seems to me that we can merge these tests
> with
> > > the
> > > > unit tests on JMX.
> > > >
> > > > bq. If we are going to
> > > > remove this capability, we need to add some other command-line tools
> > > > to get the same functionality. These tools could use REST if we have
> > > > that, or JMX, but they need to exist before we can consider removing
> > > > the old UI.
> > > >
> > > > This is a good point. Since all information are available through
> JMX,
> > > the
> > > > easiest way to approach it is to write some scripts using Node.js.
> The
> > > > architecture of the new Web UIs is ready for this.
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > > >wrote:
> > > >
> > > > > Producing JSON would be great. Agree with Colin that we should
> leave
> > > for
> > > > > now the current JSP based web ui.
> > > > >
> > > > > thx
> > > > >
> > > > >
> > > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> > cmccabe@alumni.cmu.edu
> > > > > >wrote:
> > > > >
> > > > > > This is a really interesting project, Haohui.  I think it will
> make
> > > > > > our web UI much nicer.
> > > > > >
> > > > > > I have a few concerns about removing the old web UI, however:
> > > > > >
> > > > > > * If we're going to remove the old web UI, I think the new web
UI
> > has
> > > > > > to have the same level of unit testing.  We shouldn't go
> backwards
> > in
> > > > > > terms of unit testing.
> > > > > >
> > > > > > * Most of the deployments of elinks and links out there don't
> > support
> > > > > > Javascript.  This is just a reality of life when using CentOS
5
> or
> > 6,
> > > > > > which many users are still using.  I have used "links" to
> diagnose
> > > > > > problems through the web UI in the past, in systems where access
> to
> > > > > > the cluster was available only through telnet.  If we are going
> to
> > > > > > remove this capability, we need to add some other command-line
> > tools
> > > > > > to get the same functionality.  These tools could use REST if
we
> > have
> > > > > > that, or JMX, but they need to exist before we can consider
> > removing
> > > > > > the old UI.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <
> hmai@hortonworks.com>
> > > > > wrote:
> > > > > > > Thanks for the reply, Luke. Here I just echo my response
from
> the
> > > > jira:
> > > > > > >
> > > > > > > bq. this client-side js only approach, which is less secure
> than
> > a
> > > > > > > progressively enhanced hybrid approach used by YARN. The
recent
> > > gmail
> > > > > > > XSS fiasco highlights the issue.
> > > > > > >
> > > > > > > I'm presenting an informal security analysis to compare
the
> > > security
> > > > of
> > > > > > the
> > > > > > > old and the new web UIs.
> > > > > > >
> > > > > > > An attacker launches an XSS attack by injecting malicious
code
> > > which
> > > > > are
> > > > > > > usually HTML or JavaScript fragments into the web page,
so that
> > the
> > > > > > > malicious code can have the same privileges of the web
page.
> > > > > > >
> > > > > > > First, in the scope of XSS attacks, note that the threat
models
> > of
> > > > > > > launching XSS attacks on Internet sites Gmail/Linkedin
and the
> > one
> > > of
> > > > > the
> > > > > > > Hadoop UIs are different. They have fundamental different
sets
> of
> > > > > > external
> > > > > > > inputs that the attackers have control to. Internet sites
have
> > > little
> > > > > > > control of these inputs. In the case of Gmail / Linkedin,
an
> > attack
> > > > can
> > > > > > > send you a crafted e-mail, or put malicious description
in his
> /
> > > > > > > her Linkedin profile. The sets of external inputs are
> > *restricted*
> > > in
> > > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs.
The
> > > > > > > attacker has to launch a XSS attack by:
> > > > > > >
> > > > > > > * Compromise the jars so that the output of JMX / WebHDFS
have
> > the
> > > > > > > malicious code.
> > > > > > > * Replace the web UIs completely to include the malicious
code.
> > > > > > >
> > > > > > > In either case *the attacker has to compromise the hadoop
core
> or
> > > the
> > > > > > > namenode*. That means the new web UIs are at least as secure
as
> > the
> > > > > > hadoop
> > > > > > > core, and the namenode machine.
> > > > > > >
> > > > > > > Second, I argue that using client-side templates are more
> secure
> > > than
> > > > > the
> > > > > > > current JSP-based server-side templates. To defend against
XSS
> > > > > > > attacks, both techniques have to filter the external inputs
at
> > > > *every*
> > > > > > > possible execution paths. Several facts much be taken into
> > > > > > > plays when evaluating the security of both approaches in
> > real-world
> > > > > > > environments:
> > > > > > >
> > > > > > > * The JavaScript libraries used in the new web UIs have
> survived
> > in
> > > > > > > extremely large-scale production tests. jQuery is used
by
> Google
> > > and
> > > > > > >  Microsoft, bootstrap is used by Twitter, and dust.js is
used
> by
> > > > > > Linkedin.
> > > > > > > All libraries survived from hundreds of thousands of
> > > > > > >  attack attempts on a daily basis. I agree that the libraries
> > might
> > > > > still
> > > > > > > be imperfect, but there's no way that we can test the JSP
web
> > > > > > >  UIs to achieve the same level of assurances given the
amount
> of
> > > > > > resources
> > > > > > > the community has.
> > > > > > > * Client-side templates consolidate all filtering logic
in one
> > > > central
> > > > > > > place. Recall that the goal is to filter all external inputs
at
> > > every
> > > > > > >  execution paths, this is a much more systematic approach
> > compared
> > > to
> > > > > the
> > > > > > > server-side templates we have today. It is difficult (if
not
> > > > > > >  impossible) to do it in a JSP/ASP/PHP application, since
such
> > > > > filtering
> > > > > > > can be only achieved via ad-hoc approaches ([1] shows some
> > > > > > >  empirical data). Also, HDFS-4901 recently describes a
XSS
> > > > > vulnerability
> > > > > > in
> > > > > > > browseDirectory.jsp.
> > > > > > >
> > > > > > > bq. You'd require proper SSL (not self signed) setup to
avoid
> JS
> > > > > > > injection
> > > > > > >
> > > > > > > Commodity browsers enforce Same-Origin Policy to defend
against
> > > code
> > > > > > > injections. It has nothing to do with what kinds of SSL
> > > certificates
> > > > > > > you hold.
> > > > > > >
> > > > > > > bq.  I also have concerns that we commit these changes
without
> > > > matching
> > > > > > > unit tests
> > > > > > >
> > > > > > > The JavaScript code can be automatically tested. The same
code
> > can
> > > be
> > > > > run
> > > > > > > by node.js and the test can compared with pre-defined
> > > > > > > results. It is also possible to write an adapter to use
Rhino
> to
> > > > > > accomplish
> > > > > > > the same task. We can discuss how to integrate them into
> > > > > > > the maven test routines in a different thread.
> > > > > > >
> > > > > > > bq. Client side rendering completely breaks the workflows
for
> ops
> > > who
> > > > > > rely
> > > > > > > on text based terminal/emacs/vim browsers (no js support)
to
> > > > > > > monitor component UI.
> > > > > > >
> > > > > > > links / elinks (http://elinks.or.cz/) are text-based web
> > browsers
> > > > that
> > > > > > > support JavaScript.
> > > > > > >
> > > > > > > bq. The priority/requirements for UI in core Hadoop should
be
> > > > security
> > > > > > and
> > > > > > > correctness, which client side templating cannot address
> properly
> > > > > > > so far.
> > > > > > >
> > > > > > > I agree that we should focus on security and correctness.
The
> > > > > paragraphs
> > > > > > > above explain that how the architecture of the new UIs
> > > > > > > makes the UIs more secure in real-world settings compared
to
> the
> > UI
> > > > we
> > > > > > have
> > > > > > > today.
> > > > > > >
> > > > > > > References:
> > > > > > >
> > > > > > > 1. A. Yip et al. Improving Application Security with Data
Flow
> > > > > > Assertions.
> > > > > > > In SOSP'2009.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <llu@apache.org>
> > wrote:
> > > > > > >
> > > > > > >> Echoing my comments on HDFS-3555:
> > > > > > >>
> > > > > > >> I have concerns with this client-side js only approach,
which
> is
> > > > less
> > > > > > >> secure than a progressively enhanced hybrid approach
used by
> > YARN.
> > > > The
> > > > > > >> recent gmail XSS fiasco highlights the issue. I also
have
> > concerns
> > > > > that
> > > > > > we
> > > > > > >> commit these changes without matching unit tests –
the fact
> you
> > > > cannot
> > > > > > >> effectively unit test these changes should tell you
something
> > > about
> > > > > this
> > > > > > >> approach.
> > > > > > >>
> > > > > > >> *Requiring* JS means that an admin cannot turn off
js to
> > > (partially)
> > > > > use
> > > > > > >> core Hadoop UI. You'd *require* proper SSL (not self
signed)
> > setup
> > > > to
> > > > > > avoid
> > > > > > >> JS injection, even if security of js libraries used
is
> perfect,
> > > > which
> > > > > I
> > > > > > >> doubt (search gmail/linkedin XSS). Client side rendering
> > > completely
> > > > > > breaks
> > > > > > >> the workflows for ops who rely on text based
> terminal/emacs/vim
> > > > > browsers
> > > > > > >> (no js support) to monitor component UI.
> > > > > > >>
> > > > > > >> IMO, JS-only rendering belongs to social networking
sites
> and/or
> > > > SaaS
> > > > > > >> front-ends, where full time UI/security specialists
babysits
> UI
> > > > > > changes. I
> > > > > > >> think eventually most users will use a self servicing
UI in a
> > SaaS
> > > > > > >> front-end that uses REST/JMX API to get data from back-end
> > > > components,
> > > > > > >> besides their own app master/service UI. The
> > priority/requirements
> > > > for
> > > > > > UI
> > > > > > >> in core Hadoop should be security and correctness,
which
> client
> > > side
> > > > > > >> templating cannot address properly so far.
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> > hmai@hortonworks.com
> > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > Hi all,
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Jing Zhao and I recently have reimplemented the
JSP-based
> web
> > > UIs
> > > > in
> > > > > > >> HTML 5
> > > > > > >> > applications (HDFS-5333). Based on our prelimanary
testing
> > > results
> > > > > we
> > > > > > >> > believe thst the new web UIs of the namenodes
and the
> datanode
> > > are
> > > > > > ready
> > > > > > >> > for everyday uses.
> > > > > > >> >
> > > > > > >> > You're more than welcome to try it out on trunk
by visiting
> > > > http://
> > > > > > >> > <namenode>/dfshealth.html
> > > > > > >> >
> > > > > > >> > There are a number of benefits from this transition.
From a
> > > > > > developer's
> > > > > > >> > prospective, the most notable one is *maintainability*:
> > > > > > >> >
> > > > > > >> > (1) The abstractions between the UI and the core
server are
> > > > > > well-defined,
> > > > > > >> > decoupling the UI and the core hadoop servers.
> > > > > > >> >
> > > > > > >> > (2) It allows us to deprecate the logic in the
JSP pages.
> The
> > > old
> > > > > web
> > > > > > UIs
> > > > > > >> > have to duplicate the logic in the JSPs. The logic
is often
> > > > > > out-of-dated
> > > > > > >> > and not well-tested, which leads to broken pages
and
> security
> > > > > > >> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308,
> > HDFS-5317
> > > > and
> > > > > > >> > HDFS-4901). The architecture of the new UIs prevent
these
> bugs
> > > at
> > > > > the
> > > > > > >> very
> > > > > > >> > beginning.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > I propose that deprecate the old, JSP-based web
UIs in 2.3.
> I
> > > > opened
> > > > > > >> > HDFS-5402 to track the relevant discussions.
> > > > > > >> >
> > > > > > >> > Your feedbacks are highly appreciated.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Sincerely,
> > > > > > >> >
> > > > > > >> > Haohui
> > > > > > >> >
> > > > > > >> > --
> > > > > > >> > CONFIDENTIALITY NOTICE
> > > > > > >> > NOTICE: This message is intended for the use of
the
> individual
> > > or
> > > > > > entity
> > > > > > >> to
> > > > > > >> > which it is addressed and may contain information
that is
> > > > > > confidential,
> > > > > > >> > privileged and exempt from disclosure under applicable
law.
> If
> > > the
> > > > > > reader
> > > > > > >> > of this message is not the intended recipient,
you are
> hereby
> > > > > notified
> > > > > > >> that
> > > > > > >> > any printing, copying, dissemination, distribution,
> disclosure
> > > or
> > > > > > >> > forwarding of this communication is strictly prohibited.
If
> > you
> > > > have
> > > > > > >> > received this communication in error, please contact
the
> > sender
> > > > > > >> immediately
> > > > > > >> > and delete it from your system. Thank You.
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > > --
> > > > > > > CONFIDENTIALITY NOTICE
> > > > > > > NOTICE: This message is intended for the use of the individual
> or
> > > > > entity
> > > > > > to
> > > > > > > which it is addressed and may contain information that
is
> > > > confidential,
> > > > > > > privileged and exempt from disclosure under applicable
law. If
> > the
> > > > > reader
> > > > > > > of this message is not the intended recipient, you are
hereby
> > > > notified
> > > > > > that
> > > > > > > any printing, copying, dissemination, distribution, disclosure
> or
> > > > > > > forwarding of this communication is strictly prohibited.
If you
> > > have
> > > > > > > received this communication in error, please contact the
sender
> > > > > > immediately
> > > > > > > and delete it from your system. Thank You.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alejandro
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> > >
> > >
> > > --
> > > Alejandro
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> Alejandro
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message