incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Blumberg <michael.blumb...@bti360.com>
Subject Re: Blur Console v2 Design Proposal
Date Mon, 30 Sep 2013 19:39:51 GMT
Hey Chris,

Thanks for the response, that helped clear it up. I asked because I had a
thought regarding a potential solution but wasn't sure how some of the
missing functionality of v1 worked and if my solution was even viable.

I was thinking that implementing a WebSocketServlet on the Jetty server
could:

   - Track changes to the Blur queries and store queries < 30 minutes old
   (in memory or filesystem)
   - Track changes in the zookeeper status
   - Send updates to the client via websockets.

The AngularJS client app would need to open a websocket connection to the
WebSocketServlet.

I think you touched on something like this above when you said you could
"Add on demand checking in the Jetty server" to track the Zookeeper
ensemble status.

Anyway, just a thought.

Cheers,
Michael


On Mon, Sep 30, 2013 at 12:39 PM, Chris Rohr <rohr.chris@gmail.com> wrote:

> Michael,
>
> Thank you for your question.  Hopefully this answers the bill:
>
> Of the features that I listed not being available the following were done
> by the Java application (back-end) piece in v1:
>
> - Zookeeper ensemble status
>     This feature utilized the Java zookeeper libraries to connect and check
> the status.  I have not found a Javascript Library that can do this
> (outside of using NodeJS).  There are only 2 ways we can do this going
> forward if this feature is indeed needed, 1. Add the zookeeper status to
> the Blur API or 2. Add on demand checking in the Jetty server (not sure
> what the load would be due to this)
>
> - Ability to monitor more than one Blur instance in the same tool
>     This feature was possible because the Java application had its own
> configuration and allowed many instances to be setup all at once.  Since
> this will be started (hopefully) with the start-all.sh and it will be using
> the blur-site.config for its configuration, it will not make sense to have
> this version contain many instances
>
> - HDFS information
>     The Java application did all polling HDFS node and capacity status as
> well as metric gathering.  The Rails application did have a thrift
> interface that allowed file browsing, but this I believe can all be used in
> other tools.
>
> - Keeping track of more queries than the Blur cache of 2 minutes
>     The Java application would keep 30 minutes of queries so that we could
> go back and see if something bad had happened.  If we are using Blur
> exclusively and not storing in a DB now, than we can't maintain the queries
> for a longer period of time, with the exception of allowing the browser to
> keep track until page reload.  So basically you would get 2 minutes on
> reload and then it would add in new ones as you stayed on the page.
>
> Hope this helps,
> Chris
>
>
> On Mon, Sep 30, 2013 at 10:34 AM, Michael Blumberg <
> michael.blumberg@bti360.com> wrote:
>
> > Hi Chris,
> >
> > You mentioned that certain features would not be in v2 for a variety of
> > reasons one of which was a lack of a server polling component. Which
> > feature(s) in particular is/are affected by this and is the lack of
> support
> > on the server or client side?
> >
> > -Michael
> >
> >
> > On Sun, Sep 29, 2013 at 8:33 PM, Chris Rohr <rohr.chris@gmail.com>
> wrote:
> >
> > > All,
> > >
> > > As I have begun to start the next version of the console I wanted to
> give
> > > everyone my proposal for what should be in it and get any feedback that
> > you
> > > may have.  I have created a new branch (new-blur-console) and will be
> > > pushing it up as soon as I can.  Please let me know if you have any
> > > questions/opinions/suggestions about this proposal.  If I don't hear
> from
> > > anyone, I will just assume it looks good and I will move forward.
> > >
> > > Thanks in advance,
> > > Chris
> > >
> > > *Blur Console v2 Design Proposal*
> > >
> > > *Purpose*
> > >
> > > This writeup will help to determine what direction the console portion
> of
> > > Blur should go.  There has always been a need for a push button, easy
> > setup
> > > solution so that dev-ops could get up and running with the console
> > faster.
> > > *NOTE: *This is purely a suggestion and all opinions and
> recommendations
> > > are welcome.
> > >
> > >
> > > *Server*
> > >
> > > Java program that runs an embedded Jetty server.  The purpose of the
> > Jetty
> > > server would be to serve up any of the html pages and resources needed
> > for
> > > the page as well as potentially any JSON based calls.
> > >
> > >
> > > *Interaction with Blur*
> > >
> > > The console would utilize the Javascript generated Thrift API of Blur
> to
> > > communicate and retrieve information from Blur.
> > >
> > >
> > > *Main Site Framework*
> > >
> > > We are looking to use AngularJS to provide an MVC framework in the
> > browser
> > > so that we will not have to rely on a front-end component and a
> back-end
> > > component to this system.  This should allow this tool to be much more
> > > lightweight.
> > >
> > >
> > > *Features*
> > >
> > > The main page of the site would be broken into tabs for major pieces of
> > > functionality.  The list below will outline what each tab would
> provide:
> > >
> > >
> > >
> > >    - Dashboard
> > >       - Controller Node Status (chart)
> > >       - Shard Node Status by Cluster (chart)
> > >       - Query Load by Cluster (chart)
> > >       - Query Performance by Cluster (chart)
> > >       - Number of Tables (chart)
> > >       - Long Running Query Warnings
> > >    - Tables
> > >       - Listing of Tables by Cluster and by Status
> > >       - For each table
> > >          - Shard Layout
> > >          - Schema
> > >          - Term Lookup
> > >       - Actions
> > >          - Disable Table
> > >          - Enable Table
> > >          - Delete Table (with or without deleting underlying index in
> > HDFS)
> > >        - Queries
> > >       - List of Running Queries (last 2 minutes)
> > >       - For each Query
> > >          - Display base info about the query (query string, user,
> status)
> > >          - Extra info about query (selector information)
> > >        - Search (live search of Blur)
> > >
> > >
> > > In addition we can have some base user preferences that will be stored
> in
> > > local storage or cookies in the browser.
> > >
> > >
> > > *Things missing from v1*
> > >
> > > The following items will not be in v2 either because they are not in
> the
> > > Blur API yet or because of the lack of a server polling component.
>  These
> > > are all things that were nice to have but aren’t as critical.  If any
> of
> > > these become critical we can figure out how to provide this information
> > to
> > > the console.
> > >
> > >
> > >
> > >    - Zookeeper ensemble status
> > >    - Ability to monitor more than one Blur instance in the same tool
> > (each
> > >    instance will have its own Console now)
> > >    - All HDFS pieces (deferring to other purpose built tools like Hue)
> > >    - User auditing of actions like disabling or deleting a table (we
> > could
> > >    do this with the server component)
> > >    - Keeping track of more queries than the Blur cache of 2 minutes (we
> > >    could do something that until a page refresh we could keep track of
> > > more).
> > >
> > >
> > > *Other Items to consider*
> > >
> > > One other big item to consider is how to configure the console.  I’m
> > > thinking that we could just figure out how to tie into the blur-site
> > > config.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message