couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From madhuka udantha <madhukaudan...@gmail.com>
Subject Re: [GSOC 2015] COUCHDB-2605 Visualize the CouchDB Cluster
Date Wed, 25 Mar 2015 12:43:21 GMT
looping Robert Newson

According to Robert Kowalski's suggestion I'm sharing my mockup UI's
regarding to CouchDB Cluster.

Your feedback will be mostly welcomed.

Thanks in Advance.







On Wed, Mar 25, 2015 at 12:20 AM, madhuka udantha <madhukaudantha@gmail.com>
wrote:

> Hi,
>
> After communicating with couchdb community got to know that  there can be
> 100~ 10000 nodes in cluster. So there will 2 layouts in cluster link
> depending on node count in cluster.
>
> My first UI is for simple count of node in cluster (4 nodes in cluster)
>
> Here is mock UIs where it hit more than 5 to 100000 nodes, I am using
> matrix visual.
> Here is initial look of the page. [Page1v1-initial-page]
> https://app.box.com/s/sn84u6wuzya58trssw11ba6fwtmn862f
> Features:
> 1. User can select the attribute eg: Disk usage, http status, data usage,
> alerts. etc.
> 2. Chart / matrix will be draw with legend
>
> [Page2v1-searching-node]
> https://app.box.com/s/i5uomdhirn96co6acbbrvvviifyw95pb
> Who can we search node in there?
> You will have search in top of it. (it allow to search the node), It will
> show it with red color box.
> Or every node have tool tip to verify node, what is you look for.
>
>
> [Page3v1-selecting-node]
> https://app.box.com/s/rsd5ysld597gvra2nkcn7027bpok37jc
> When user click on node, node panel will be open RHS.
> It have all the details of the node and operation that you can perform in
> node leve
>
> [Page4v1-searching-db_in_node]
> https://app.box.com/s/jk3ojb0khj192ldnyw0eqs1p5t7l485r
> User can search Databases in nodes even
> This searching have advance search query support
>
> [Page5v1-db_view]
> https://app.box.com/s/20hjgtcb7c9ikgfdn7xy8xli4uhorbni
> User can select the database and it will show database details.
>
>
>  feed backs for UIs are welcome.
>
>
> On Tue, Mar 24, 2015 at 4:35 PM, madhuka udantha <madhukaudantha@gmail.com
> > wrote:
>
>> Hi James,
>>
>> Thanks for you feedback. It will help me to improve my proposal and to
>> get better understanding about couchdb cluster
>>
>> Since I am new to couchdb and I did not know much user case that you all
>> have come across. But when you share those ideas. It is help to understand
>> couchdb cluster. All you feedback is important.
>>
>> On Tue, Mar 24, 2015 at 12:07 PM, Mutton, James <jmutton@akamai.com>
>> wrote:
>>
>>> I can share a few quick thoughts (which I admit is rare, but I happen to
>>> have a few minutes and this hit the top of my stack):
>>>
>>> * How do you envision this overflowing when you have hundreds of
>>> databases within a cluster, or maybe 64 nodes in a cluster.
>>>
>> Attached UIs  can only handle very low level nodes (4 to 8) according to
>> layout (we can called it expanded layout).  When I am drawing I did not
>> thought that level of count of nodes and databases. Therefore I will
>> revisit the mockup ui. Here is updated UI
>>
>> Here you can have 1 ~ 1000 node even and each node can contain 100
>> databases.
>> Node are listed and user can pick one of it then it will show it details
>> and databases of it in list.
>>
>> UI for Listing node when nodes count is above 8 in cluster (This is how
>> it look as it have 64 nodes in cluster)
>> https://app.box.com/s/kqg2kadf0xa1zoym9sme22q5m4wbg8i5
>>  [Page10-list-nodes-more-than-8]
>>
>> *Features of the UI:*
>>
>>    - Table header support sorting (ascending and descending) by just
>>    click on it
>>    - User have searching option to search (node/databases by name)
>>    - It support advance search query (popup form ui)
>>    - Notification listed with icon and warning/alert level is shown by
>>    color of icon
>>    - User can perform node operation by clicking hyperlink in it's raw
>>
>>
>> Node contains 100 databases (answer)
>> https://app.box.com/s/ggbzr00orkwacsi4cflpvn9d60q7h1st
>>  [Page11-list-dbs-in-selected-node]
>>
>>
>>
>> *Features of the UI*
>>
>>    - In Database panel contain search that scope to databases of the
>>    selected node
>>    - When user select database it will show database summary details and
>>    operations that can be perform for that database
>>
>>
>>
>>> ** On a related note, drag and drop is cool, but level of difficulty
>>> involved in adding a shard from node-1 increases in direct proportion to
>>> the number of nodes there are.
>>
>>  Yes, we can add button ( in database/node level
>>
>>   I think the “Add Database” button on a new server should be labeled
>>> “Add Shard” and should present the option of selecting what shards to add.
>>> The add button is also not present on the nodes that have shards already so
>>> it would be good to keep it around somehow.
>>>
>> Yes,  It will be label as “Add Shard”. It is possible to show only that
>> is not shard so far.
>>
>>
>> * Speaking then of shards… They don’t appear to be captured here.  How
>>> does one know which shard-set each server has?  Each database has a number
>>> of shards that all have multiple shard copies and it is not a requirement
>>> that all servers have all shards.  None of that is really represented in
>>> these mockups.  When looking at a shard, it would be good to know if it is
>>> 00000000-15555554 or aaaaaaa8-bffffffc, or even both.  Knowing this and
>>> being able to use it to scale/move hotspots in a cluster is pretty
>>> important for operations, once databases hit a significant size.
>>>
>> Thanks for pointing out. shards will add for visualization.
>>
>> Can you share resources/documentations which that help to understand the
>> couchdb clustering?
>> I like to know what are most important operations regard couch
>> clustering? (that you think good to added in here. since I am new for
>> couchdb and some time I have miss them)
>>
>>
>>
>>> * I don’t think doc-count and deleted-count is useful information at the
>>> overview page
>>
>> Thanks, We can remove doc count details from overview page. If user need
>> to read such details he/she can click 'more' so there it will show them.
>>
>>>
>>
>> and runs the risk of requiring a lot of requests to fulfill (see point 1
>>> on hundreds of databases).
>>>
>> Sorry, In here I did not get your point
>>
>> * What does compact: false mean?  That the database is not small? or that
>>> it’s not actively compacting?  And if the latter, wouldn’t that be on a
>>> db-shard basis, not a whole server.
>>>
>> compact running is mapping to 'compact_running'[2]
>>
>> [1] https://wiki.apache.org/couchdb/Compaction
>>
>>
>>> * It would be nice if cluster usage graphing page would show http-stats
>>> as well.  Would give people a snapshot indication of where their load is
>>> hot.
>>>
>> Sure, Different status codes will be notify on cluster usage graph.
>> (Mockups UI will shared tonight)
>>
>> Thanks JamesM for your feed backs
>>
>>>
>>> </JamesM>
>>>
>>>
>>> On Mar 23, 2015, at 20:32, madhuka udantha <madhukaudantha@gmail.com>
>>> wrote:
>>>
>>> > Hi All,
>>> >
>>> > are there any ideas and feedback for above mockup UIs?
>>> >
>>> > Community feedback will be useful in here. :)
>>> >
>>> > On Sun, Mar 22, 2015 at 11:51 PM, madhuka udantha <
>>> madhukaudantha@gmail.com>
>>> > wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> Here are the mockup UIs for Visualizing the CouchDB Clusters. It is
>>> just
>>> >> the initial UIs.
>>> >>
>>> >> * Page 1-cluster.png* [
>>> >> https://app.box.com/s/isjbfop69j2cmz0kg0qar7l5nmupjnfl
>>> >>
>>> >> This Page visualize nodes in the clusters with summary information for
>>> >> each node. (Name, Memory usage..etc..) with a nice web UI interface.
>>> It is
>>> >> interactive (drag and drop supporting over nodes)
>>> >>
>>> >> *Page 2-cluster-notification-tip.png [*
>>> >> https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb
>>> >> <https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb>]
>>> >>
>>> >> Top of each panel of the nodes it contains a notification icon if
>>> there is
>>> >> notification.
>>> >> eg: Memory/disk space availability, Error in node etc...
>>> >> Notification will have a simple drop down structure as it is showing.
>>> >>
>>> >> *Page 3-cluster-warning.png* [
>>> >> https://app.box.com/s/f2ljwxvz2w01f8uepsj4zyarhwbpiz8b]
>>> >>
>>> >> Critical level alert is showen. (Alert level can be set in config and
>>> >> notification center)
>>> >>
>>> >> *Page 4-adding-new-node.png* [
>>> >> https://app.box.com/s/noygbeyze792bh4t93iaoxywdciau057]
>>> >>
>>> >> By clicking "add new node" you can add new nodes for clusters. UI
>>> >> Consistency is maintained
>>> >>
>>> >> *Page 5-adding-new-node-by-drag-drop.png*[
>>> >> https://app.box.com/s/gbfsbwb6zt1jdkijve72mnullplecpj7]
>>> >>
>>> >> User interface is supported for drag-drop of databases in nodes.
>>> >> (replication operation in the BE)
>>> >>
>>> >> *Page 6-selected-db-in-node.png*[
>>> >> https://app.box.com/s/6ojrvpxuvmo9k18dlnf8i4nmibzdx5s9]
>>> >> Each node shows the databases in it. By Clicking on database you can
>>> see
>>> >> detailed view of a particular database. This allows users to
>>> understand
>>> >> memory/data/doc distribution over DBs in particular nodes.
>>> >>
>>> >> *Page 7-custer-mgt-center.png* [
>>> >> https://app.box.com/s/t9s42g3gwvv6bocbdsxrgjimdrh2ty0z]
>>> >> Each node is shown with disk usage under icon and summary text of the
>>> node
>>> >> is also given. It is easy to identify the disk space - free
>>> >> availability. (In here we need some BE services help)
>>> >>
>>> >>
>>> >> *Page 8-custer-mgt-center-infor.png* [
>>> >> https://app.box.com/s/vvo4v9ji79l8japo3nm9f4lfbjhdamkc]
>>> >> User can customize notifications for each nodes by level (warning,
>>> infor,
>>> >> etc..)
>>> >>
>>> >> *Page 9-charting-disk-usages.png*[
>>> >> https://app.box.com/s/fk7eyoyrtg8d3b29xoo0mdho9rr8cmyz]
>>> >> Cluster usage of the system is shown as a bar chart and the
>>> attributes to
>>> >> be shown can be selected according to user's preferences.
>>> >>
>>> >> You ideas are welcome in here.
>>> >> Thanks
>>> >>
>>> >> On Sun, Mar 22, 2015 at 7:34 PM, madhuka udantha <
>>> madhukaudantha@gmail.com
>>> >>> wrote:
>>> >>
>>> >>> Hi All,
>>> >>>
>>> >>> As I'm drafting my proposal for GSoC 2015 I went through some of
the
>>> >>> existing Cluster Management Systems with Visualization. I recognized
>>> some
>>> >>> of the features regarding to Cluster Management HTTP API. (Some
>>> times not
>>> >>> fully documented yet)
>>> >>>     Such us partitioning, re-balance..etc.
>>> >>> I was unable to find them on CouchDB API Documentation(only at the
>>> >>> implemented level but not documented yet? ) but I was able to find
>>> some of
>>> >>> the features such as adding nodes to clusters through dev-mailing
>>> list.
>>> >>> Are some of these features regarding cluster management present
on
>>> the
>>> >>> API. If so where can I find the relevant documentations/resources?
>>> Even
>>> >>> the source code is fine to me.
>>> >>>
>>> >>>
>>> >>> Since I was able to clearly identify those missing features can
I
>>> start a
>>> >>> discussion on a thread on the best feature list for cluster
>>> management on
>>> >>> CouchDB to be added to it on a future release.
>>> >>> My current project could also include these features that are
>>> supported
>>> >>> by the API else I could help on including them.
>>> >>>
>>> >>> [1] http://guide.couchdb.org/draft/clustering.html
>>> >>> Thanks.
>>> >>> BR,
>>> >>>
>>> >>> On Sat, Mar 21, 2015 at 11:40 AM, madhuka udantha <
>>> >>> madhukaudantha@gmail.com> wrote:
>>> >>>
>>> >>>>
>>> >>>>
>>> >>>> On Sat, Mar 21, 2015 at 12:17 AM, Alexander Shorin <
>>> kxepal@gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> On Fri, Mar 20, 2015 at 9:28 PM, madhuka udantha
>>> >>>>> <madhukaudantha@gmail.com> wrote:
>>> >>>>>> I like to know what is *different of '_nodes' and 'node'*?
>>> >>>>>>
>>> >>>>>> http://localhost:15986/nodes
>>> -->{"db_name":"nodes","doc_count":3..}
>>> >>>>>> http://localhost:25986/nodes
>>> -->{"db_name":"nodes","doc_count":1..}
>>> >>>>>> http://localhost:35986/nodes
>>> >>>>> -->{"error":"not_found","reason":"no_db_file"}
>>> >>>>>
>>> >>>>> Where did you get these nodes?
>>> >>>>>
>>> >>>> By '~/test/1/couchdb $ dev/run -n 4' I can added 4 nodes. (default
>>> it is
>>> >>>> 3 nodes)
>>> >>>>
>>> >>>> Here is my console print, Yes now I have four node running.
>>> >>>>
>>> >>>> *max@max-VirtualBox ~/test/1/couchdb $ dev/run -n 4*
>>> >>>> *Setup environment... ok*
>>> >>>> *Ensure CouchDB is built... ok*
>>> >>>> *Prepare configuration files... ok*
>>> >>>> *Start node node1... ok*
>>> >>>> *Start node node2... ok*
>>> >>>> *Start node node3... ok*
>>> >>>> *Start node node4... ok*
>>> >>>> *Ensure all nodes are run... ok*
>>> >>>> *Join nodes into cluster... ok*
>>> >>>> *Developers cluster is set up at http://127.0.0.1:15984
>>> >>>> <http://127.0.0.1:15984>. Time to hack!... *
>>> >>>>
>>> >>>> Each above nodes are running in localhost in port 15986, 25986,
>>> 35986
>>> >>>> and 45986
>>> >>>>
>>> >>>> It seems docs are mainly in 15986 (node1) shown by 'doc_count'
>>> value,
>>> >>>> is there any reason for that or is it happening randomly?
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> ,,,^..^,,,
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Cheers,
>>> >>>> Madhuka Udantha
>>> >>>> http://madhukaudantha.blogspot.com
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Cheers,
>>> >>> Madhuka Udantha
>>> >>> http://madhukaudantha.blogspot.com
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Cheers,
>>> >> Madhuka Udantha
>>> >> http://madhukaudantha.blogspot.com
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Cheers,
>>> > Madhuka Udantha
>>> > http://madhukaudantha.blogspot.com
>>>
>>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message