hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haohui Mai <h...@hortonworks.com>
Subject Re: Replacing the JSP web UIs to HTML 5 applications
Date Tue, 29 Oct 2013 01:09:26 GMT
Neither of them will go through JMX.

The new Web UI implements hdfs browsing through WebHDFS.

The logs are available through the static servlets, which is exactly the
same as what we have today.

Thanks,
Haohui


On Mon, Oct 28, 2013 at 6:01 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:

> are you planning to expose things like hdfs browsing and nn/dn logs over
> jmx?
>
> thx
>
> Alejandro
> (phone typing)
>
> On Oct 28, 2013, at 17:48, Haohui Mai <hmai@hortonworks.com> wrote:
>
> > It seems more appealing to me that the UI should JMX directly, because:
> >
> > * We're support the JMX in the long term for other management software.
> > * The information provided by the JMX API will be mostly identical of the
> > JSON API. Today the Web UI covers most of the information provided by
> JMX.
> > The Web UI does some trivial work to extract the information and renders
> it
> > as HTML.
> > * We can compatibility and unit tests for free.
> >
> > I do agree that the JMX APIs are imperfect and they should be revisited
> in
> > the 3.0 timeframe. However, this is orthogonal of the discussions of
> > transitioning from JSP-based Web UI to client-side JavaScript Web UI. The
> > architecture of the new Web UI allows easy migration to any JSON-based
> APIs
> > whenever they land in the trunk.
> >
> > Thanks,
> > Haohui
> >
> >
> > On Mon, Oct 28, 2013 at 5:13 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> Isn't using JMX to expose JSON for the web UI misusing JMX?
> >>
> >> I would think a more appropriate approach would be having /JMX for
> >> monitoring integration and a /JSON end point for the UI data.
> >>
> >> Thanks.
> >>
> >>
> >> On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hmai@hortonworks.com>
> wrote:
> >>
> >>> Alejandro,
> >>>
> >>> If I understand correctly, that is the exact approach that the new web
> UI
> >>> is taking. The new web UI takes the output from JMX and renders them as
> >>> HTML at the client side.
> >>>
> >>> ~Haohui
> >>>
> >>>
> >>> On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tucu@cloudera.com
> >>>> wrote:
> >>>
> >>>> Haohui,
> >>>>
> >>>> If you have NN and DNs producing JSON instead HTML, then you can build
> >> JS
> >>>> based web UIs. Take for example Oozie, Oozie produces JSON, it has a
> >>> built
> >>>> in JS web ui that consumes JSON and Hue has built an external web UI
> >> that
> >>>> also consumes JSON. In the case of Hue UI, Oozie didn't have to change
> >>>> anything to get that UI and improvements on the Hue UI don't require
> >>>> changes in Oozie unless it is to produce additional information.
> >>>>
> >>>> hope this clarifies.
> >>>>
> >>>> Thx
> >>>>
> >>>>
> >>>> On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hmai@hortonworks.com>
> >>> wrote:
> >>>>
> >>>>> Echo my comments on HDFS-5402:
> >>>>>
> >>>>> bq. If we're going to remove the old web UI, I think the new web
UI
> >> has
> >>>>> to have the same level of unit testing. We shouldn't go backwards
in
> >>>>> terms of unit testing.
> >>>>>
> >>>>> I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> >>>>> TestClusterJspHelper. It seems to me that we can merge these tests
> >> with
> >>>> the
> >>>>> unit tests on JMX.
> >>>>>
> >>>>> bq. If we are going to
> >>>>> remove this capability, we need to add some other command-line tools
> >>>>> to get the same functionality. These tools could use REST if we
have
> >>>>> that, or JMX, but they need to exist before we can consider removing
> >>>>> the old UI.
> >>>>>
> >>>>> This is a good point. Since all information are available through
> >> JMX,
> >>>> the
> >>>>> easiest way to approach it is to write some scripts using Node.js.
> >> The
> >>>>> architecture of the new Web UIs is ready for this.
> >>>>>
> >>>>>
> >>>>> On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <
> >> tucu@cloudera.com
> >>>>>> wrote:
> >>>>>
> >>>>>> Producing JSON would be great. Agree with Colin that we should
> >> leave
> >>>> for
> >>>>>> now the current JSP based web ui.
> >>>>>>
> >>>>>> thx
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> >>> cmccabe@alumni.cmu.edu
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> This is a really interesting project, Haohui.  I think it
will
> >> make
> >>>>>>> our web UI much nicer.
> >>>>>>>
> >>>>>>> I have a few concerns about removing the old web UI, however:
> >>>>>>>
> >>>>>>> * If we're going to remove the old web UI, I think the new
web UI
> >>> has
> >>>>>>> to have the same level of unit testing.  We shouldn't go
> >> backwards
> >>> in
> >>>>>>> terms of unit testing.
> >>>>>>>
> >>>>>>> * Most of the deployments of elinks and links out there
don't
> >>> support
> >>>>>>> Javascript.  This is just a reality of life when using CentOS
5
> >> or
> >>> 6,
> >>>>>>> which many users are still using.  I have used "links" to
> >> diagnose
> >>>>>>> problems through the web UI in the past, in systems where
access
> >> to
> >>>>>>> the cluster was available only through telnet.  If we are
going
> >> to
> >>>>>>> remove this capability, we need to add some other command-line
> >>> tools
> >>>>>>> to get the same functionality.  These tools could use REST
if we
> >>> have
> >>>>>>> that, or JMX, but they need to exist before we can consider
> >>> removing
> >>>>>>> the old UI.
> >>>>>>>
> >>>>>>> best,
> >>>>>>> Colin
> >>>>>>>
> >>>>>>> On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <
> >> hmai@hortonworks.com>
> >>>>>> wrote:
> >>>>>>>> Thanks for the reply, Luke. Here I just echo my response
from
> >> the
> >>>>> jira:
> >>>>>>>>
> >>>>>>>> bq. this client-side js only approach, which is less
secure
> >> than
> >>> a
> >>>>>>>> progressively enhanced hybrid approach used by YARN.
The recent
> >>>> gmail
> >>>>>>>> XSS fiasco highlights the issue.
> >>>>>>>>
> >>>>>>>> I'm presenting an informal security analysis to compare
the
> >>>> security
> >>>>> of
> >>>>>>> the
> >>>>>>>> old and the new web UIs.
> >>>>>>>>
> >>>>>>>> An attacker launches an XSS attack by injecting malicious
code
> >>>> which
> >>>>>> are
> >>>>>>>> usually HTML or JavaScript fragments into the web page,
so that
> >>> the
> >>>>>>>> malicious code can have the same privileges of the web
page.
> >>>>>>>>
> >>>>>>>> First, in the scope of XSS attacks, note that the threat
models
> >>> of
> >>>>>>>> launching XSS attacks on Internet sites Gmail/Linkedin
and the
> >>> one
> >>>> of
> >>>>>> the
> >>>>>>>> Hadoop UIs are different. They have fundamental different
sets
> >> of
> >>>>>>> external
> >>>>>>>> inputs that the attackers have control to. Internet
sites have
> >>>> little
> >>>>>>>> control of these inputs. In the case of Gmail / Linkedin,
an
> >>> attack
> >>>>> can
> >>>>>>>> send you a crafted e-mail, or put malicious description
in his
> >> /
> >>>>>>>> her Linkedin profile. The sets of external inputs are
> >>> *restricted*
> >>>> in
> >>>>>>>> Hadoop UIs. The new web UIs take JMX and WebHDFS as
inputs. The
> >>>>>>>> attacker has to launch a XSS attack by:
> >>>>>>>>
> >>>>>>>> * Compromise the jars so that the output of JMX / WebHDFS
have
> >>> the
> >>>>>>>> malicious code.
> >>>>>>>> * Replace the web UIs completely to include the malicious
code.
> >>>>>>>>
> >>>>>>>> In either case *the attacker has to compromise the hadoop
core
> >> or
> >>>> the
> >>>>>>>> namenode*. That means the new web UIs are at least as
secure as
> >>> the
> >>>>>>> hadoop
> >>>>>>>> core, and the namenode machine.
> >>>>>>>>
> >>>>>>>> Second, I argue that using client-side templates are
more
> >> secure
> >>>> than
> >>>>>> the
> >>>>>>>> current JSP-based server-side templates. To defend against
XSS
> >>>>>>>> attacks, both techniques have to filter the external
inputs at
> >>>>> *every*
> >>>>>>>> possible execution paths. Several facts much be taken
into
> >>>>>>>> plays when evaluating the security of both approaches
in
> >>> real-world
> >>>>>>>> environments:
> >>>>>>>>
> >>>>>>>> * The JavaScript libraries used in the new web UIs have
> >> survived
> >>> in
> >>>>>>>> extremely large-scale production tests. jQuery is used
by
> >> Google
> >>>> and
> >>>>>>>> Microsoft, bootstrap is used by Twitter, and dust.js
is used
> >> by
> >>>>>>> Linkedin.
> >>>>>>>> All libraries survived from hundreds of thousands of
> >>>>>>>> attack attempts on a daily basis. I agree that the libraries
> >>> might
> >>>>>> still
> >>>>>>>> be imperfect, but there's no way that we can test the
JSP web
> >>>>>>>> UIs to achieve the same level of assurances given the
amount
> >> of
> >>>>>>> resources
> >>>>>>>> the community has.
> >>>>>>>> * Client-side templates consolidate all filtering logic
in one
> >>>>> central
> >>>>>>>> place. Recall that the goal is to filter all external
inputs at
> >>>> every
> >>>>>>>> execution paths, this is a much more systematic approach
> >>> compared
> >>>> to
> >>>>>> the
> >>>>>>>> server-side templates we have today. It is difficult
(if not
> >>>>>>>> impossible) to do it in a JSP/ASP/PHP application, since
such
> >>>>>> filtering
> >>>>>>>> can be only achieved via ad-hoc approaches ([1] shows
some
> >>>>>>>> empirical data). Also, HDFS-4901 recently describes
a XSS
> >>>>>> vulnerability
> >>>>>>> in
> >>>>>>>> browseDirectory.jsp.
> >>>>>>>>
> >>>>>>>> bq. You'd require proper SSL (not self signed) setup
to avoid
> >> JS
> >>>>>>>> injection
> >>>>>>>>
> >>>>>>>> Commodity browsers enforce Same-Origin Policy to defend
against
> >>>> code
> >>>>>>>> injections. It has nothing to do with what kinds of
SSL
> >>>> certificates
> >>>>>>>> you hold.
> >>>>>>>>
> >>>>>>>> bq.  I also have concerns that we commit these changes
without
> >>>>> matching
> >>>>>>>> unit tests
> >>>>>>>>
> >>>>>>>> The JavaScript code can be automatically tested. The
same code
> >>> can
> >>>> be
> >>>>>> run
> >>>>>>>> by node.js and the test can compared with pre-defined
> >>>>>>>> results. It is also possible to write an adapter to
use Rhino
> >> to
> >>>>>>> accomplish
> >>>>>>>> the same task. We can discuss how to integrate them
into
> >>>>>>>> the maven test routines in a different thread.
> >>>>>>>>
> >>>>>>>> bq. Client side rendering completely breaks the workflows
for
> >> ops
> >>>> who
> >>>>>>> rely
> >>>>>>>> on text based terminal/emacs/vim browsers (no js support)
to
> >>>>>>>> monitor component UI.
> >>>>>>>>
> >>>>>>>> links / elinks (http://elinks.or.cz/) are text-based
web
> >>> browsers
> >>>>> that
> >>>>>>>> support JavaScript.
> >>>>>>>>
> >>>>>>>> bq. The priority/requirements for UI in core Hadoop
should be
> >>>>> security
> >>>>>>> and
> >>>>>>>> correctness, which client side templating cannot address
> >> properly
> >>>>>>>> so far.
> >>>>>>>>
> >>>>>>>> I agree that we should focus on security and correctness.
The
> >>>>>> paragraphs
> >>>>>>>> above explain that how the architecture of the new UIs
> >>>>>>>> makes the UIs more secure in real-world settings compared
to
> >> the
> >>> UI
> >>>>> we
> >>>>>>> have
> >>>>>>>> today.
> >>>>>>>>
> >>>>>>>> References:
> >>>>>>>>
> >>>>>>>> 1. A. Yip et al. Improving Application Security with
Data Flow
> >>>>>>> Assertions.
> >>>>>>>> In SOSP'2009.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <llu@apache.org>
> >>> wrote:
> >>>>>>>>
> >>>>>>>>> Echoing my comments on HDFS-3555:
> >>>>>>>>>
> >>>>>>>>> I have concerns with this client-side js only approach,
which
> >> is
> >>>>> less
> >>>>>>>>> secure than a progressively enhanced hybrid approach
used by
> >>> YARN.
> >>>>> The
> >>>>>>>>> recent gmail XSS fiasco highlights the issue. I
also have
> >>> concerns
> >>>>>> that
> >>>>>>> we
> >>>>>>>>> commit these changes without matching unit tests
– the fact
> >> you
> >>>>> cannot
> >>>>>>>>> effectively unit test these changes should tell
you something
> >>>> about
> >>>>>> this
> >>>>>>>>> approach.
> >>>>>>>>>
> >>>>>>>>> *Requiring* JS means that an admin cannot turn off
js to
> >>>> (partially)
> >>>>>> use
> >>>>>>>>> core Hadoop UI. You'd *require* proper SSL (not
self signed)
> >>> setup
> >>>>> to
> >>>>>>> avoid
> >>>>>>>>> JS injection, even if security of js libraries used
is
> >> perfect,
> >>>>> which
> >>>>>> I
> >>>>>>>>> doubt (search gmail/linkedin XSS). Client side rendering
> >>>> completely
> >>>>>>> breaks
> >>>>>>>>> the workflows for ops who rely on text based
> >> terminal/emacs/vim
> >>>>>> browsers
> >>>>>>>>> (no js support) to monitor component UI.
> >>>>>>>>>
> >>>>>>>>> IMO, JS-only rendering belongs to social networking
sites
> >> and/or
> >>>>> SaaS
> >>>>>>>>> front-ends, where full time UI/security specialists
babysits
> >> UI
> >>>>>>> changes. I
> >>>>>>>>> think eventually most users will use a self servicing
UI in a
> >>> SaaS
> >>>>>>>>> front-end that uses REST/JMX API to get data from
back-end
> >>>>> components,
> >>>>>>>>> besides their own app master/service UI. The
> >>> priority/requirements
> >>>>> for
> >>>>>>> UI
> >>>>>>>>> in core Hadoop should be security and correctness,
which
> >> client
> >>>> side
> >>>>>>>>> templating cannot address properly so far.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> >>> hmai@hortonworks.com
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi all,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Jing Zhao and I recently have reimplemented
the JSP-based
> >> web
> >>>> UIs
> >>>>> in
> >>>>>>>>> HTML 5
> >>>>>>>>>> applications (HDFS-5333). Based on our prelimanary
testing
> >>>> results
> >>>>>> we
> >>>>>>>>>> believe thst the new web UIs of the namenodes
and the
> >> datanode
> >>>> are
> >>>>>>> ready
> >>>>>>>>>> for everyday uses.
> >>>>>>>>>>
> >>>>>>>>>> You're more than welcome to try it out on trunk
by visiting
> >>>>> http://
> >>>>>>>>>> <namenode>/dfshealth.html
> >>>>>>>>>>
> >>>>>>>>>> There are a number of benefits from this transition.
From a
> >>>>>>> developer's
> >>>>>>>>>> prospective, the most notable one is *maintainability*:
> >>>>>>>>>>
> >>>>>>>>>> (1) The abstractions between the UI and the
core server are
> >>>>>>> well-defined,
> >>>>>>>>>> decoupling the UI and the core hadoop servers.
> >>>>>>>>>>
> >>>>>>>>>> (2) It allows us to deprecate the logic in the
JSP pages.
> >> The
> >>>> old
> >>>>>> web
> >>>>>>> UIs
> >>>>>>>>>> have to duplicate the logic in the JSPs. The
logic is often
> >>>>>>> out-of-dated
> >>>>>>>>>> and not well-tested, which leads to broken pages
and
> >> security
> >>>>>>>>>> vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308,
> >>> HDFS-5317
> >>>>> and
> >>>>>>>>>> HDFS-4901). The architecture of the new UIs
prevent these
> >> bugs
> >>>> at
> >>>>>> the
> >>>>>>>>> very
> >>>>>>>>>> beginning.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I propose that deprecate the old, JSP-based
web UIs in 2.3.
> >> I
> >>>>> opened
> >>>>>>>>>> HDFS-5402 to track the relevant discussions.
> >>>>>>>>>>
> >>>>>>>>>> Your feedbacks are highly appreciated.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Sincerely,
> >>>>>>>>>>
> >>>>>>>>>> Haohui
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>>> NOTICE: This message is intended for the use
of the
> >> individual
> >>>> or
> >>>>>>> entity
> >>>>>>>>> to
> >>>>>>>>>> which it is addressed and may contain information
that is
> >>>>>>> confidential,
> >>>>>>>>>> privileged and exempt from disclosure under
applicable law.
> >> If
> >>>> the
> >>>>>>> reader
> >>>>>>>>>> of this message is not the intended recipient,
you are
> >> hereby
> >>>>>> notified
> >>>>>>>>> that
> >>>>>>>>>> any printing, copying, dissemination, distribution,
> >> disclosure
> >>>> or
> >>>>>>>>>> forwarding of this communication is strictly
prohibited. If
> >>> you
> >>>>> have
> >>>>>>>>>> received this communication in error, please
contact the
> >>> sender
> >>>>>>>>> immediately
> >>>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>> NOTICE: This message is intended for the use of the
individual
> >> or
> >>>>>> entity
> >>>>>>> to
> >>>>>>>> which it is addressed and may contain information that
is
> >>>>> confidential,
> >>>>>>>> privileged and exempt from disclosure under applicable
law. If
> >>> the
> >>>>>> reader
> >>>>>>>> of this message is not the intended recipient, you are
hereby
> >>>>> notified
> >>>>>>> that
> >>>>>>>> any printing, copying, dissemination, distribution,
disclosure
> >> or
> >>>>>>>> forwarding of this communication is strictly prohibited.
If you
> >>>> have
> >>>>>>>> received this communication in error, please contact
the sender
> >>>>>>> immediately
> >>>>>>>> and delete it from your system. Thank You.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Alejandro
> >>>>>
> >>>>> --
> >>>>> CONFIDENTIALITY NOTICE
> >>>>> NOTICE: This message is intended for the use of the individual or
> >>> entity
> >>>> to
> >>>>> which it is addressed and may contain information that is
> >> confidential,
> >>>>> privileged and exempt from disclosure under applicable law. If the
> >>> reader
> >>>>> of this message is not the intended recipient, you are hereby
> >> notified
> >>>> that
> >>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>> forwarding of this communication is strictly prohibited. If you
have
> >>>>> received this communication in error, please contact the sender
> >>>> immediately
> >>>>> and delete it from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Alejandro
> >>>
> >>> --
> >>> CONFIDENTIALITY NOTICE
> >>> NOTICE: This message is intended for the use of the individual or
> entity
> >> to
> >>> which it is addressed and may contain information that is confidential,
> >>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>> of this message is not the intended recipient, you are hereby notified
> >> that
> >>> any printing, copying, dissemination, distribution, disclosure or
> >>> forwarding of this communication is strictly prohibited. If you have
> >>> received this communication in error, please contact the sender
> >> immediately
> >>> and delete it from your system. Thank You.
> >>
> >>
> >>
> >> --
> >> Alejandro
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message