hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Abdelnur <t...@cloudera.com>
Subject Re: Replacing the JSP web UIs to HTML 5 applications
Date Tue, 29 Oct 2013 01:01:24 GMT
are you planning to expose things like hdfs browsing and nn/dn logs over jmx?

thx

Alejandro
(phone typing)

On Oct 28, 2013, at 17:48, Haohui Mai <hmai@hortonworks.com> wrote:

> It seems more appealing to me that the UI should JMX directly, because:
> 
> * We're support the JMX in the long term for other management software.
> * The information provided by the JMX API will be mostly identical of the
> JSON API. Today the Web UI covers most of the information provided by JMX.
> The Web UI does some trivial work to extract the information and renders it
> as HTML.
> * We can compatibility and unit tests for free.
> 
> I do agree that the JMX APIs are imperfect and they should be revisited in
> the 3.0 timeframe. However, this is orthogonal of the discussions of
> transitioning from JSP-based Web UI to client-side JavaScript Web UI. The
> architecture of the new Web UI allows easy migration to any JSON-based APIs
> whenever they land in the trunk.
> 
> Thanks,
> Haohui
> 
> 
> On Mon, Oct 28, 2013 at 5:13 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:
> 
>> Isn't using JMX to expose JSON for the web UI misusing JMX?
>> 
>> I would think a more appropriate approach would be having /JMX for
>> monitoring integration and a /JSON end point for the UI data.
>> 
>> Thanks.
>> 
>> 
>> On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hmai@hortonworks.com> wrote:
>> 
>>> Alejandro,
>>> 
>>> If I understand correctly, that is the exact approach that the new web UI
>>> is taking. The new web UI takes the output from JMX and renders them as
>>> HTML at the client side.
>>> 
>>> ~Haohui
>>> 
>>> 
>>> On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>> wrote:
>>> 
>>>> Haohui,
>>>> 
>>>> If you have NN and DNs producing JSON instead HTML, then you can build
>> JS
>>>> based web UIs. Take for example Oozie, Oozie produces JSON, it has a
>>> built
>>>> in JS web ui that consumes JSON and Hue has built an external web UI
>> that
>>>> also consumes JSON. In the case of Hue UI, Oozie didn't have to change
>>>> anything to get that UI and improvements on the Hue UI don't require
>>>> changes in Oozie unless it is to produce additional information.
>>>> 
>>>> hope this clarifies.
>>>> 
>>>> Thx
>>>> 
>>>> 
>>>> On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hmai@hortonworks.com>
>>> wrote:
>>>> 
>>>>> Echo my comments on HDFS-5402:
>>>>> 
>>>>> bq. If we're going to remove the old web UI, I think the new web UI
>> has
>>>>> to have the same level of unit testing. We shouldn't go backwards in
>>>>> terms of unit testing.
>>>>> 
>>>>> I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
>>>>> TestClusterJspHelper. It seems to me that we can merge these tests
>> with
>>>> the
>>>>> unit tests on JMX.
>>>>> 
>>>>> bq. If we are going to
>>>>> remove this capability, we need to add some other command-line tools
>>>>> to get the same functionality. These tools could use REST if we have
>>>>> that, or JMX, but they need to exist before we can consider removing
>>>>> the old UI.
>>>>> 
>>>>> This is a good point. Since all information are available through
>> JMX,
>>>> the
>>>>> easiest way to approach it is to write some scripts using Node.js.
>> The
>>>>> architecture of the new Web UIs is ready for this.
>>>>> 
>>>>> 
>>>>> On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <
>> tucu@cloudera.com
>>>>>> wrote:
>>>>> 
>>>>>> Producing JSON would be great. Agree with Colin that we should
>> leave
>>>> for
>>>>>> now the current JSP based web ui.
>>>>>> 
>>>>>> thx
>>>>>> 
>>>>>> 
>>>>>> On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
>>> cmccabe@alumni.cmu.edu
>>>>>>> wrote:
>>>>>> 
>>>>>>> This is a really interesting project, Haohui.  I think it will
>> make
>>>>>>> our web UI much nicer.
>>>>>>> 
>>>>>>> I have a few concerns about removing the old web UI, however:
>>>>>>> 
>>>>>>> * If we're going to remove the old web UI, I think the new web
UI
>>> has
>>>>>>> to have the same level of unit testing.  We shouldn't go
>> backwards
>>> in
>>>>>>> terms of unit testing.
>>>>>>> 
>>>>>>> * Most of the deployments of elinks and links out there don't
>>> support
>>>>>>> Javascript.  This is just a reality of life when using CentOS
5
>> or
>>> 6,
>>>>>>> which many users are still using.  I have used "links" to
>> diagnose
>>>>>>> problems through the web UI in the past, in systems where access
>> to
>>>>>>> the cluster was available only through telnet.  If we are going
>> to
>>>>>>> remove this capability, we need to add some other command-line
>>> tools
>>>>>>> to get the same functionality.  These tools could use REST if
we
>>> have
>>>>>>> that, or JMX, but they need to exist before we can consider
>>> removing
>>>>>>> the old UI.
>>>>>>> 
>>>>>>> best,
>>>>>>> Colin
>>>>>>> 
>>>>>>> On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <
>> hmai@hortonworks.com>
>>>>>> wrote:
>>>>>>>> Thanks for the reply, Luke. Here I just echo my response
from
>> the
>>>>> jira:
>>>>>>>> 
>>>>>>>> bq. this client-side js only approach, which is less secure
>> than
>>> a
>>>>>>>> progressively enhanced hybrid approach used by YARN. The
recent
>>>> gmail
>>>>>>>> XSS fiasco highlights the issue.
>>>>>>>> 
>>>>>>>> I'm presenting an informal security analysis to compare the
>>>> security
>>>>> of
>>>>>>> the
>>>>>>>> old and the new web UIs.
>>>>>>>> 
>>>>>>>> An attacker launches an XSS attack by injecting malicious
code
>>>> which
>>>>>> are
>>>>>>>> usually HTML or JavaScript fragments into the web page, so
that
>>> the
>>>>>>>> malicious code can have the same privileges of the web page.
>>>>>>>> 
>>>>>>>> First, in the scope of XSS attacks, note that the threat
models
>>> of
>>>>>>>> launching XSS attacks on Internet sites Gmail/Linkedin and
the
>>> one
>>>> of
>>>>>> the
>>>>>>>> Hadoop UIs are different. They have fundamental different
sets
>> of
>>>>>>> external
>>>>>>>> inputs that the attackers have control to. Internet sites
have
>>>> little
>>>>>>>> control of these inputs. In the case of Gmail / Linkedin,
an
>>> attack
>>>>> can
>>>>>>>> send you a crafted e-mail, or put malicious description in
his
>> /
>>>>>>>> her Linkedin profile. The sets of external inputs are
>>> *restricted*
>>>> in
>>>>>>>> Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs.
The
>>>>>>>> attacker has to launch a XSS attack by:
>>>>>>>> 
>>>>>>>> * Compromise the jars so that the output of JMX / WebHDFS
have
>>> the
>>>>>>>> malicious code.
>>>>>>>> * Replace the web UIs completely to include the malicious
code.
>>>>>>>> 
>>>>>>>> In either case *the attacker has to compromise the hadoop
core
>> or
>>>> the
>>>>>>>> namenode*. That means the new web UIs are at least as secure
as
>>> the
>>>>>>> hadoop
>>>>>>>> core, and the namenode machine.
>>>>>>>> 
>>>>>>>> Second, I argue that using client-side templates are more
>> secure
>>>> than
>>>>>> the
>>>>>>>> current JSP-based server-side templates. To defend against
XSS
>>>>>>>> attacks, both techniques have to filter the external inputs
at
>>>>> *every*
>>>>>>>> possible execution paths. Several facts much be taken into
>>>>>>>> plays when evaluating the security of both approaches in
>>> real-world
>>>>>>>> environments:
>>>>>>>> 
>>>>>>>> * The JavaScript libraries used in the new web UIs have
>> survived
>>> in
>>>>>>>> extremely large-scale production tests. jQuery is used by
>> Google
>>>> and
>>>>>>>> Microsoft, bootstrap is used by Twitter, and dust.js is used
>> by
>>>>>>> Linkedin.
>>>>>>>> All libraries survived from hundreds of thousands of
>>>>>>>> attack attempts on a daily basis. I agree that the libraries
>>> might
>>>>>> still
>>>>>>>> be imperfect, but there's no way that we can test the JSP
web
>>>>>>>> UIs to achieve the same level of assurances given the amount
>> of
>>>>>>> resources
>>>>>>>> the community has.
>>>>>>>> * Client-side templates consolidate all filtering logic in
one
>>>>> central
>>>>>>>> place. Recall that the goal is to filter all external inputs
at
>>>> every
>>>>>>>> execution paths, this is a much more systematic approach
>>> compared
>>>> to
>>>>>> the
>>>>>>>> server-side templates we have today. It is difficult (if
not
>>>>>>>> impossible) to do it in a JSP/ASP/PHP application, since
such
>>>>>> filtering
>>>>>>>> can be only achieved via ad-hoc approaches ([1] shows some
>>>>>>>> empirical data). Also, HDFS-4901 recently describes a XSS
>>>>>> vulnerability
>>>>>>> in
>>>>>>>> browseDirectory.jsp.
>>>>>>>> 
>>>>>>>> bq. You'd require proper SSL (not self signed) setup to avoid
>> JS
>>>>>>>> injection
>>>>>>>> 
>>>>>>>> Commodity browsers enforce Same-Origin Policy to defend against
>>>> code
>>>>>>>> injections. It has nothing to do with what kinds of SSL
>>>> certificates
>>>>>>>> you hold.
>>>>>>>> 
>>>>>>>> bq.  I also have concerns that we commit these changes without
>>>>> matching
>>>>>>>> unit tests
>>>>>>>> 
>>>>>>>> The JavaScript code can be automatically tested. The same
code
>>> can
>>>> be
>>>>>> run
>>>>>>>> by node.js and the test can compared with pre-defined
>>>>>>>> results. It is also possible to write an adapter to use Rhino
>> to
>>>>>>> accomplish
>>>>>>>> the same task. We can discuss how to integrate them into
>>>>>>>> the maven test routines in a different thread.
>>>>>>>> 
>>>>>>>> bq. Client side rendering completely breaks the workflows
for
>> ops
>>>> who
>>>>>>> rely
>>>>>>>> on text based terminal/emacs/vim browsers (no js support)
to
>>>>>>>> monitor component UI.
>>>>>>>> 
>>>>>>>> links / elinks (http://elinks.or.cz/) are text-based web
>>> browsers
>>>>> that
>>>>>>>> support JavaScript.
>>>>>>>> 
>>>>>>>> bq. The priority/requirements for UI in core Hadoop should
be
>>>>> security
>>>>>>> and
>>>>>>>> correctness, which client side templating cannot address
>> properly
>>>>>>>> so far.
>>>>>>>> 
>>>>>>>> I agree that we should focus on security and correctness.
The
>>>>>> paragraphs
>>>>>>>> above explain that how the architecture of the new UIs
>>>>>>>> makes the UIs more secure in real-world settings compared
to
>> the
>>> UI
>>>>> we
>>>>>>> have
>>>>>>>> today.
>>>>>>>> 
>>>>>>>> References:
>>>>>>>> 
>>>>>>>> 1. A. Yip et al. Improving Application Security with Data
Flow
>>>>>>> Assertions.
>>>>>>>> In SOSP'2009.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <llu@apache.org>
>>> wrote:
>>>>>>>> 
>>>>>>>>> Echoing my comments on HDFS-3555:
>>>>>>>>> 
>>>>>>>>> I have concerns with this client-side js only approach,
which
>> is
>>>>> less
>>>>>>>>> secure than a progressively enhanced hybrid approach
used by
>>> YARN.
>>>>> The
>>>>>>>>> recent gmail XSS fiasco highlights the issue. I also
have
>>> concerns
>>>>>> that
>>>>>>> we
>>>>>>>>> commit these changes without matching unit tests –
the fact
>> you
>>>>> cannot
>>>>>>>>> effectively unit test these changes should tell you something
>>>> about
>>>>>> this
>>>>>>>>> approach.
>>>>>>>>> 
>>>>>>>>> *Requiring* JS means that an admin cannot turn off js
to
>>>> (partially)
>>>>>> use
>>>>>>>>> core Hadoop UI. You'd *require* proper SSL (not self
signed)
>>> setup
>>>>> to
>>>>>>> avoid
>>>>>>>>> JS injection, even if security of js libraries used is
>> perfect,
>>>>> which
>>>>>> I
>>>>>>>>> doubt (search gmail/linkedin XSS). Client side rendering
>>>> completely
>>>>>>> breaks
>>>>>>>>> the workflows for ops who rely on text based
>> terminal/emacs/vim
>>>>>> browsers
>>>>>>>>> (no js support) to monitor component UI.
>>>>>>>>> 
>>>>>>>>> IMO, JS-only rendering belongs to social networking sites
>> and/or
>>>>> SaaS
>>>>>>>>> front-ends, where full time UI/security specialists babysits
>> UI
>>>>>>> changes. I
>>>>>>>>> think eventually most users will use a self servicing
UI in a
>>> SaaS
>>>>>>>>> front-end that uses REST/JMX API to get data from back-end
>>>>> components,
>>>>>>>>> besides their own app master/service UI. The
>>> priority/requirements
>>>>> for
>>>>>>> UI
>>>>>>>>> in core Hadoop should be security and correctness, which
>> client
>>>> side
>>>>>>>>> templating cannot address properly so far.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
>>> hmai@hortonworks.com
>>>>> 
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Jing Zhao and I recently have reimplemented the JSP-based
>> web
>>>> UIs
>>>>> in
>>>>>>>>> HTML 5
>>>>>>>>>> applications (HDFS-5333). Based on our prelimanary
testing
>>>> results
>>>>>> we
>>>>>>>>>> believe thst the new web UIs of the namenodes and
the
>> datanode
>>>> are
>>>>>>> ready
>>>>>>>>>> for everyday uses.
>>>>>>>>>> 
>>>>>>>>>> You're more than welcome to try it out on trunk by
visiting
>>>>> http://
>>>>>>>>>> <namenode>/dfshealth.html
>>>>>>>>>> 
>>>>>>>>>> There are a number of benefits from this transition.
From a
>>>>>>> developer's
>>>>>>>>>> prospective, the most notable one is *maintainability*:
>>>>>>>>>> 
>>>>>>>>>> (1) The abstractions between the UI and the core
server are
>>>>>>> well-defined,
>>>>>>>>>> decoupling the UI and the core hadoop servers.
>>>>>>>>>> 
>>>>>>>>>> (2) It allows us to deprecate the logic in the JSP
pages.
>> The
>>>> old
>>>>>> web
>>>>>>> UIs
>>>>>>>>>> have to duplicate the logic in the JSPs. The logic
is often
>>>>>>> out-of-dated
>>>>>>>>>> and not well-tested, which leads to broken pages
and
>> security
>>>>>>>>>> vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308,
>>> HDFS-5317
>>>>> and
>>>>>>>>>> HDFS-4901). The architecture of the new UIs prevent
these
>> bugs
>>>> at
>>>>>> the
>>>>>>>>> very
>>>>>>>>>> beginning.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I propose that deprecate the old, JSP-based web UIs
in 2.3.
>> I
>>>>> opened
>>>>>>>>>> HDFS-5402 to track the relevant discussions.
>>>>>>>>>> 
>>>>>>>>>> Your feedbacks are highly appreciated.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Sincerely,
>>>>>>>>>> 
>>>>>>>>>> Haohui
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>> NOTICE: This message is intended for the use of the
>> individual
>>>> or
>>>>>>> entity
>>>>>>>>> to
>>>>>>>>>> which it is addressed and may contain information
that is
>>>>>>> confidential,
>>>>>>>>>> privileged and exempt from disclosure under applicable
law.
>> If
>>>> the
>>>>>>> reader
>>>>>>>>>> of this message is not the intended recipient, you
are
>> hereby
>>>>>> notified
>>>>>>>>> that
>>>>>>>>>> any printing, copying, dissemination, distribution,
>> disclosure
>>>> or
>>>>>>>>>> forwarding of this communication is strictly prohibited.
If
>>> you
>>>>> have
>>>>>>>>>> received this communication in error, please contact
the
>>> sender
>>>>>>>>> immediately
>>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>> 
>>>>>>>> --
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual
>> or
>>>>>> entity
>>>>>>> to
>>>>>>>> which it is addressed and may contain information that is
>>>>> confidential,
>>>>>>>> privileged and exempt from disclosure under applicable law.
If
>>> the
>>>>>> reader
>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>> that
>>>>>>>> any printing, copying, dissemination, distribution, disclosure
>> or
>>>>>>>> forwarding of this communication is strictly prohibited.
If you
>>>> have
>>>>>>>> received this communication in error, please contact the
sender
>>>>>>> immediately
>>>>>>>> and delete it from your system. Thank You.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Alejandro
>>>>> 
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>> entity
>>>> to
>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>>>> of this message is not the intended recipient, you are hereby
>> notified
>>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>> immediately
>>>>> and delete it from your system. Thank You.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alejandro
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>> immediately
>>> and delete it from your system. Thank You.
>> 
>> 
>> 
>> --
>> Alejandro
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.

Mime
View raw message