couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From madhuka udantha <madhukaudan...@gmail.com>
Subject Re: [GSOC 2015] COUCHDB-2605 Visualize the CouchDB Cluster
Date Wed, 25 Mar 2015 18:16:33 GMT
Hi,

Sharing mockups UIs again, Since some ML-Users do not have account in
box.com.
I am sharing them via jira.

On Wed, Mar 25, 2015 at 12:20 AM, madhuka udantha <madhukaudantha@gmail.com>
wrote:

> Hi,
>
> After communicating with couchdb community got to know that  there can be
> 100~ 10000 nodes in cluster. So there will 2 layouts in cluster link
> depending on node count in cluster.
>
> My first UI is for simple count of node in cluster (4 nodes in cluster)
>
> Here is mock UIs where it hit more than 5 to 100000 nodes, I am using
> matrix visual.
> Here is initial look of the page. [Page1v1-initial-page]
> https://app.box.com/s/sn84u6wuzya58trssw11ba6fwtmn862f
>
https://issues.apache.org/jira/secure/attachment/12707205/Page1v1-initial-page.png



> Features:
> 1. User can select the attribute eg: Disk usage, http status, data usage,
> alerts. etc.
> 2. Chart / matrix will be draw with legend
>
> [Page2v1-searching-node]
> https://app.box.com/s/i5uomdhirn96co6acbbrvvviifyw95pb
>
https://issues.apache.org/jira/secure/attachment/12707218/Page2v1-searching-node.png
<https://issues.apache.org/jira/secure/attachment/12707218/Page2v1-searching-node.png>


> Who can we search node in there?
> You will have search in top of it. (it allow to search the node), It will
> show it with red color box.
> Or every node have tool tip to verify node, what is you look for.
>
>
> [Page3v1-selecting-node]
> https://app.box.com/s/rsd5ysld597gvra2nkcn7027bpok37jc
>
https://issues.apache.org/jira/secure/attachment/12707219/Page3v1-selecting-node.png
<https://issues.apache.org/jira/secure/attachment/12707219/Page3v1-selecting-node.png>


> When user click on node, node panel will be open RHS.
> It have all the details of the node and operation that you can perform in
> node leve
>
> [Page4v1-searching-db_in_node]
> https://app.box.com/s/jk3ojb0khj192ldnyw0eqs1p5t7l485r
>
https://issues.apache.org/jira/secure/attachment/12707220/Page4v1-searching-db_in_node.png
<https://issues.apache.org/jira/secure/attachment/12707220/Page4v1-searching-db_in_node.png>


> User can search Databases in nodes even
> This searching have advance search query support
>
> [Page5v1-db_view]
> https://app.box.com/s/20hjgtcb7c9ikgfdn7xy8xli4uhorbni
>
https://issues.apache.org/jira/secure/attachment/12707206/Page5v1-db_view.png
<https://issues.apache.org/jira/secure/attachment/12707206/Page5v1-db_view.png>

<https://app.box.com/s/20hjgtcb7c9ikgfdn7xy8xli4uhorbni>
> User can select the database and it will show database details.
>
>
>  feed backs for UIs are welcome.
>
>
> On Tue, Mar 24, 2015 at 4:35 PM, madhuka udantha <madhukaudantha@gmail.com
> > wrote:
>
>> Hi James,
>>
>> Thanks for you feedback. It will help me to improve my proposal and to
>> get better understanding about couchdb cluster
>>
>> Since I am new to couchdb and I did not know much user case that you all
>> have come across. But when you share those ideas. It is help to understand
>> couchdb cluster. All you feedback is important.
>>
>> On Tue, Mar 24, 2015 at 12:07 PM, Mutton, James <jmutton@akamai.com>
>> wrote:
>>
>>> I can share a few quick thoughts (which I admit is rare, but I happen to
>>> have a few minutes and this hit the top of my stack):
>>>
>>> * How do you envision this overflowing when you have hundreds of
>>> databases within a cluster, or maybe 64 nodes in a cluster.
>>>
>> Attached UIs  can only handle very low level nodes (4 to 8) according to
>> layout (we can called it expanded layout).  When I am drawing I did not
>> thought that level of count of nodes and databases. Therefore I will
>> revisit the mockup ui. Here is updated UI
>>
>> Here you can have 1 ~ 1000 node even and each node can contain 100
>> databases.
>> Node are listed and user can pick one of it then it will show it details
>> and databases of it in list.
>>
>> UI for Listing node when nodes count is above 8 in cluster (This is how
>> it look as it have 64 nodes in cluster)
>> https://app.box.com/s/kqg2kadf0xa1zoym9sme22q5m4wbg8i5
>>  [Page10-list-nodes-more-than-8]
>>
>> *Features of the UI:*
>>
>>    - Table header support sorting (ascending and descending) by just
>>    click on it
>>    - User have searching option to search (node/databases by name)
>>    - It support advance search query (popup form ui)
>>    - Notification listed with icon and warning/alert level is shown by
>>    color of icon
>>    - User can perform node operation by clicking hyperlink in it's raw
>>
>>
>> Node contains 100 databases (answer)
>> https://app.box.com/s/ggbzr00orkwacsi4cflpvn9d60q7h1st
>>  [Page11-list-dbs-in-selected-node]
>>
>>
>>
>> *Features of the UI*
>>
>>    - In Database panel contain search that scope to databases of the
>>    selected node
>>    - When user select database it will show database summary details and
>>    operations that can be perform for that database
>>
>>
>>
>>> ** On a related note, drag and drop is cool, but level of difficulty
>>> involved in adding a shard from node-1 increases in direct proportion to
>>> the number of nodes there are.
>>
>>  Yes, we can add button ( in database/node level
>>
>>   I think the “Add Database” button on a new server should be labeled
>>> “Add Shard” and should present the option of selecting what shards to add.
>>> The add button is also not present on the nodes that have shards already so
>>> it would be good to keep it around somehow.
>>>
>> Yes,  It will be label as “Add Shard”. It is possible to show only that
>> is not shard so far.
>>
>>
>> * Speaking then of shards… They don’t appear to be captured here.  How
>>> does one know which shard-set each server has?  Each database has a number
>>> of shards that all have multiple shard copies and it is not a requirement
>>> that all servers have all shards.  None of that is really represented in
>>> these mockups.  When looking at a shard, it would be good to know if it is
>>> 00000000-15555554 or aaaaaaa8-bffffffc, or even both.  Knowing this and
>>> being able to use it to scale/move hotspots in a cluster is pretty
>>> important for operations, once databases hit a significant size.
>>>
>> Thanks for pointing out. shards will add for visualization.
>>
>> Can you share resources/documentations which that help to understand the
>> couchdb clustering?
>> I like to know what are most important operations regard couch
>> clustering? (that you think good to added in here. since I am new for
>> couchdb and some time I have miss them)
>>
>>
>>
>>> * I don’t think doc-count and deleted-count is useful information at the
>>> overview page
>>
>> Thanks, We can remove doc count details from overview page. If user need
>> to read such details he/she can click 'more' so there it will show them.
>>
>>>
>>
>> and runs the risk of requiring a lot of requests to fulfill (see point 1
>>> on hundreds of databases).
>>>
>> Sorry, In here I did not get your point
>>
>> * What does compact: false mean?  That the database is not small? or that
>>> it’s not actively compacting?  And if the latter, wouldn’t that be on a
>>> db-shard basis, not a whole server.
>>>
>> compact running is mapping to 'compact_running'[2]
>>
>> [1] https://wiki.apache.org/couchdb/Compaction
>>
>>
>>> * It would be nice if cluster usage graphing page would show http-stats
>>> as well.  Would give people a snapshot indication of where their load is
>>> hot.
>>>
>> Sure, Different status codes will be notify on cluster usage graph.
>> (Mockups UI will shared tonight)
>>
>> Thanks JamesM for your feed backs
>>
>>>
>>> </JamesM>
>>>
>>>
>>> On Mar 23, 2015, at 20:32, madhuka udantha <madhukaudantha@gmail.com>
>>> wrote:
>>>
>>> > Hi All,
>>> >
>>> > are there any ideas and feedback for above mockup UIs?
>>> >
>>> > Community feedback will be useful in here. :)
>>> >
>>> > On Sun, Mar 22, 2015 at 11:51 PM, madhuka udantha <
>>> madhukaudantha@gmail.com>
>>> > wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> Here are the mockup UIs for Visualizing the CouchDB Clusters. It is
>>> just
>>> >> the initial UIs.
>>> >>
>>> >> * Page 1-cluster.png* [
>>> >> https://app.box.com/s/isjbfop69j2cmz0kg0qar7l5nmupjnfl
>>> >>
>>> >> This Page visualize nodes in the clusters with summary information for
>>> >> each node. (Name, Memory usage..etc..) with a nice web UI interface.
>>> It is
>>> >> interactive (drag and drop supporting over nodes)
>>> >>
>>> >> *Page 2-cluster-notification-tip.png [*
>>> >> https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb
>>> >> <https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb>]
>>> >>
>>> >> Top of each panel of the nodes it contains a notification icon if
>>> there is
>>> >> notification.
>>> >> eg: Memory/disk space availability, Error in node etc...
>>> >> Notification will have a simple drop down structure as it is showing.
>>> >>
>>> >> *Page 3-cluster-warning.png* [
>>> >> https://app.box.com/s/f2ljwxvz2w01f8uepsj4zyarhwbpiz8b]
>>> >>
>>> >> Critical level alert is showen. (Alert level can be set in config and
>>> >> notification center)
>>> >>
>>> >> *Page 4-adding-new-node.png* [
>>> >> https://app.box.com/s/noygbeyze792bh4t93iaoxywdciau057]
>>> >>
>>> >> By clicking "add new node" you can add new nodes for clusters. UI
>>> >> Consistency is maintained
>>> >>
>>> >> *Page 5-adding-new-node-by-drag-drop.png*[
>>> >> https://app.box.com/s/gbfsbwb6zt1jdkijve72mnullplecpj7]
>>> >>
>>> >> User interface is supported for drag-drop of databases in nodes.
>>> >> (replication operation in the BE)
>>> >>
>>> >> *Page 6-selected-db-in-node.png*[
>>> >> https://app.box.com/s/6ojrvpxuvmo9k18dlnf8i4nmibzdx5s9]
>>> >> Each node shows the databases in it. By Clicking on database you can
>>> see
>>> >> detailed view of a particular database. This allows users to
>>> understand
>>> >> memory/data/doc distribution over DBs in particular nodes.
>>> >>
>>> >> *Page 7-custer-mgt-center.png* [
>>> >> https://app.box.com/s/t9s42g3gwvv6bocbdsxrgjimdrh2ty0z]
>>> >> Each node is shown with disk usage under icon and summary text of the
>>> node
>>> >> is also given. It is easy to identify the disk space - free
>>> >> availability. (In here we need some BE services help)
>>> >>
>>> >>
>>> >> *Page 8-custer-mgt-center-infor.png* [
>>> >> https://app.box.com/s/vvo4v9ji79l8japo3nm9f4lfbjhdamkc]
>>> >> User can customize notifications for each nodes by level (warning,
>>> infor,
>>> >> etc..)
>>> >>
>>> >> *Page 9-charting-disk-usages.png*[
>>> >> https://app.box.com/s/fk7eyoyrtg8d3b29xoo0mdho9rr8cmyz]
>>> >> Cluster usage of the system is shown as a bar chart and the
>>> attributes to
>>> >> be shown can be selected according to user's preferences.
>>> >>
>>> >> You ideas are welcome in here.
>>> >> Thanks
>>> >>
>>> >> On Sun, Mar 22, 2015 at 7:34 PM, madhuka udantha <
>>> madhukaudantha@gmail.com
>>> >>> wrote:
>>> >>
>>> >>> Hi All,
>>> >>>
>>> >>> As I'm drafting my proposal for GSoC 2015 I went through some of
the
>>> >>> existing Cluster Management Systems with Visualization. I recognized
>>> some
>>> >>> of the features regarding to Cluster Management HTTP API. (Some
>>> times not
>>> >>> fully documented yet)
>>> >>>     Such us partitioning, re-balance..etc.
>>> >>> I was unable to find them on CouchDB API Documentation(only at the
>>> >>> implemented level but not documented yet? ) but I was able to find
>>> some of
>>> >>> the features such as adding nodes to clusters through dev-mailing
>>> list.
>>> >>> Are some of these features regarding cluster management present
on
>>> the
>>> >>> API. If so where can I find the relevant documentations/resources?
>>> Even
>>> >>> the source code is fine to me.
>>> >>>
>>> >>>
>>> >>> Since I was able to clearly identify those missing features can
I
>>> start a
>>> >>> discussion on a thread on the best feature list for cluster
>>> management on
>>> >>> CouchDB to be added to it on a future release.
>>> >>> My current project could also include these features that are
>>> supported
>>> >>> by the API else I could help on including them.
>>> >>>
>>> >>> [1] http://guide.couchdb.org/draft/clustering.html
>>> >>> Thanks.
>>> >>> BR,
>>> >>>
>>> >>> On Sat, Mar 21, 2015 at 11:40 AM, madhuka udantha <
>>> >>> madhukaudantha@gmail.com> wrote:
>>> >>>
>>> >>>>
>>> >>>>
>>> >>>> On Sat, Mar 21, 2015 at 12:17 AM, Alexander Shorin <
>>> kxepal@gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> On Fri, Mar 20, 2015 at 9:28 PM, madhuka udantha
>>> >>>>> <madhukaudantha@gmail.com> wrote:
>>> >>>>>> I like to know what is *different of '_nodes' and 'node'*?
>>> >>>>>>
>>> >>>>>> http://localhost:15986/nodes
>>> -->{"db_name":"nodes","doc_count":3..}
>>> >>>>>> http://localhost:25986/nodes
>>> -->{"db_name":"nodes","doc_count":1..}
>>> >>>>>> http://localhost:35986/nodes
>>> >>>>> -->{"error":"not_found","reason":"no_db_file"}
>>> >>>>>
>>> >>>>> Where did you get these nodes?
>>> >>>>>
>>> >>>> By '~/test/1/couchdb $ dev/run -n 4' I can added 4 nodes. (default
>>> it is
>>> >>>> 3 nodes)
>>> >>>>
>>> >>>> Here is my console print, Yes now I have four node running.
>>> >>>>
>>> >>>> *max@max-VirtualBox ~/test/1/couchdb $ dev/run -n 4*
>>> >>>> *Setup environment... ok*
>>> >>>> *Ensure CouchDB is built... ok*
>>> >>>> *Prepare configuration files... ok*
>>> >>>> *Start node node1... ok*
>>> >>>> *Start node node2... ok*
>>> >>>> *Start node node3... ok*
>>> >>>> *Start node node4... ok*
>>> >>>> *Ensure all nodes are run... ok*
>>> >>>> *Join nodes into cluster... ok*
>>> >>>> *Developers cluster is set up at http://127.0.0.1:15984
>>> >>>> <http://127.0.0.1:15984>. Time to hack!... *
>>> >>>>
>>> >>>> Each above nodes are running in localhost in port 15986, 25986,
>>> 35986
>>> >>>> and 45986
>>> >>>>
>>> >>>> It seems docs are mainly in 15986 (node1) shown by 'doc_count'
>>> value,
>>> >>>> is there any reason for that or is it happening randomly?
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> ,,,^..^,,,
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Cheers,
>>> >>>> Madhuka Udantha
>>> >>>> http://madhukaudantha.blogspot.com
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Cheers,
>>> >>> Madhuka Udantha
>>> >>> http://madhukaudantha.blogspot.com
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Cheers,
>>> >> Madhuka Udantha
>>> >> http://madhukaudantha.blogspot.com
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Cheers,
>>> > Madhuka Udantha
>>> > http://madhukaudantha.blogspot.com
>>>
>>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message