incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Rohr <rohr.ch...@gmail.com>
Subject Re: Blur Console v2 Design Proposal
Date Tue, 01 Oct 2013 00:10:14 GMT
Michael,

Thanks for the thoughts.  I agree with you that we could solve some of
these issues with some simple servlets in the container.

As for the WebSockets I think its a good idea to help update the client,
however, we may have some browser issues with this.  I know that one of my
user bases doesn't have browser that support web sockets.

Chris


On Mon, Sep 30, 2013 at 3:39 PM, Michael Blumberg <
michael.blumberg@bti360.com> wrote:

> Hey Chris,
>
> Thanks for the response, that helped clear it up. I asked because I had a
> thought regarding a potential solution but wasn't sure how some of the
> missing functionality of v1 worked and if my solution was even viable.
>
> I was thinking that implementing a WebSocketServlet on the Jetty server
> could:
>
>    - Track changes to the Blur queries and store queries < 30 minutes old
>    (in memory or filesystem)
>    - Track changes in the zookeeper status
>    - Send updates to the client via websockets.
>
> The AngularJS client app would need to open a websocket connection to the
> WebSocketServlet.
>
> I think you touched on something like this above when you said you could
> "Add on demand checking in the Jetty server" to track the Zookeeper
> ensemble status.
>
> Anyway, just a thought.
>
> Cheers,
> Michael
>
>
> On Mon, Sep 30, 2013 at 12:39 PM, Chris Rohr <rohr.chris@gmail.com> wrote:
>
> > Michael,
> >
> > Thank you for your question.  Hopefully this answers the bill:
> >
> > Of the features that I listed not being available the following were done
> > by the Java application (back-end) piece in v1:
> >
> > - Zookeeper ensemble status
> >     This feature utilized the Java zookeeper libraries to connect and
> check
> > the status.  I have not found a Javascript Library that can do this
> > (outside of using NodeJS).  There are only 2 ways we can do this going
> > forward if this feature is indeed needed, 1. Add the zookeeper status to
> > the Blur API or 2. Add on demand checking in the Jetty server (not sure
> > what the load would be due to this)
> >
> > - Ability to monitor more than one Blur instance in the same tool
> >     This feature was possible because the Java application had its own
> > configuration and allowed many instances to be setup all at once.  Since
> > this will be started (hopefully) with the start-all.sh and it will be
> using
> > the blur-site.config for its configuration, it will not make sense to
> have
> > this version contain many instances
> >
> > - HDFS information
> >     The Java application did all polling HDFS node and capacity status as
> > well as metric gathering.  The Rails application did have a thrift
> > interface that allowed file browsing, but this I believe can all be used
> in
> > other tools.
> >
> > - Keeping track of more queries than the Blur cache of 2 minutes
> >     The Java application would keep 30 minutes of queries so that we
> could
> > go back and see if something bad had happened.  If we are using Blur
> > exclusively and not storing in a DB now, than we can't maintain the
> queries
> > for a longer period of time, with the exception of allowing the browser
> to
> > keep track until page reload.  So basically you would get 2 minutes on
> > reload and then it would add in new ones as you stayed on the page.
> >
> > Hope this helps,
> > Chris
> >
> >
> > On Mon, Sep 30, 2013 at 10:34 AM, Michael Blumberg <
> > michael.blumberg@bti360.com> wrote:
> >
> > > Hi Chris,
> > >
> > > You mentioned that certain features would not be in v2 for a variety of
> > > reasons one of which was a lack of a server polling component. Which
> > > feature(s) in particular is/are affected by this and is the lack of
> > support
> > > on the server or client side?
> > >
> > > -Michael
> > >
> > >
> > > On Sun, Sep 29, 2013 at 8:33 PM, Chris Rohr <rohr.chris@gmail.com>
> > wrote:
> > >
> > > > All,
> > > >
> > > > As I have begun to start the next version of the console I wanted to
> > give
> > > > everyone my proposal for what should be in it and get any feedback
> that
> > > you
> > > > may have.  I have created a new branch (new-blur-console) and will be
> > > > pushing it up as soon as I can.  Please let me know if you have any
> > > > questions/opinions/suggestions about this proposal.  If I don't hear
> > from
> > > > anyone, I will just assume it looks good and I will move forward.
> > > >
> > > > Thanks in advance,
> > > > Chris
> > > >
> > > > *Blur Console v2 Design Proposal*
> > > >
> > > > *Purpose*
> > > >
> > > > This writeup will help to determine what direction the console
> portion
> > of
> > > > Blur should go.  There has always been a need for a push button, easy
> > > setup
> > > > solution so that dev-ops could get up and running with the console
> > > faster.
> > > > *NOTE: *This is purely a suggestion and all opinions and
> > recommendations
> > > > are welcome.
> > > >
> > > >
> > > > *Server*
> > > >
> > > > Java program that runs an embedded Jetty server.  The purpose of the
> > > Jetty
> > > > server would be to serve up any of the html pages and resources
> needed
> > > for
> > > > the page as well as potentially any JSON based calls.
> > > >
> > > >
> > > > *Interaction with Blur*
> > > >
> > > > The console would utilize the Javascript generated Thrift API of Blur
> > to
> > > > communicate and retrieve information from Blur.
> > > >
> > > >
> > > > *Main Site Framework*
> > > >
> > > > We are looking to use AngularJS to provide an MVC framework in the
> > > browser
> > > > so that we will not have to rely on a front-end component and a
> > back-end
> > > > component to this system.  This should allow this tool to be much
> more
> > > > lightweight.
> > > >
> > > >
> > > > *Features*
> > > >
> > > > The main page of the site would be broken into tabs for major pieces
> of
> > > > functionality.  The list below will outline what each tab would
> > provide:
> > > >
> > > >
> > > >
> > > >    - Dashboard
> > > >       - Controller Node Status (chart)
> > > >       - Shard Node Status by Cluster (chart)
> > > >       - Query Load by Cluster (chart)
> > > >       - Query Performance by Cluster (chart)
> > > >       - Number of Tables (chart)
> > > >       - Long Running Query Warnings
> > > >    - Tables
> > > >       - Listing of Tables by Cluster and by Status
> > > >       - For each table
> > > >          - Shard Layout
> > > >          - Schema
> > > >          - Term Lookup
> > > >       - Actions
> > > >          - Disable Table
> > > >          - Enable Table
> > > >          - Delete Table (with or without deleting underlying index in
> > > HDFS)
> > > >        - Queries
> > > >       - List of Running Queries (last 2 minutes)
> > > >       - For each Query
> > > >          - Display base info about the query (query string, user,
> > status)
> > > >          - Extra info about query (selector information)
> > > >        - Search (live search of Blur)
> > > >
> > > >
> > > > In addition we can have some base user preferences that will be
> stored
> > in
> > > > local storage or cookies in the browser.
> > > >
> > > >
> > > > *Things missing from v1*
> > > >
> > > > The following items will not be in v2 either because they are not in
> > the
> > > > Blur API yet or because of the lack of a server polling component.
> >  These
> > > > are all things that were nice to have but aren’t as critical.  If any
> > of
> > > > these become critical we can figure out how to provide this
> information
> > > to
> > > > the console.
> > > >
> > > >
> > > >
> > > >    - Zookeeper ensemble status
> > > >    - Ability to monitor more than one Blur instance in the same tool
> > > (each
> > > >    instance will have its own Console now)
> > > >    - All HDFS pieces (deferring to other purpose built tools like
> Hue)
> > > >    - User auditing of actions like disabling or deleting a table (we
> > > could
> > > >    do this with the server component)
> > > >    - Keeping track of more queries than the Blur cache of 2 minutes
> (we
> > > >    could do something that until a page refresh we could keep track
> of
> > > > more).
> > > >
> > > >
> > > > *Other Items to consider*
> > > >
> > > > One other big item to consider is how to configure the console.  I’m
> > > > thinking that we could just figure out how to tie into the blur-site
> > > > config.
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message