couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From madhuka udantha <madhukaudan...@gmail.com>
Subject Re: [GSOC 2015] COUCHDB-2605 Visualize the CouchDB Cluster
Date Tue, 24 Mar 2015 11:05:05 GMT
Hi James,

Thanks for you feedback. It will help me to improve my proposal and to get
better understanding about couchdb cluster

Since I am new to couchdb and I did not know much user case that you all
have come across. But when you share those ideas. It is help to understand
couchdb cluster. All you feedback is important.

On Tue, Mar 24, 2015 at 12:07 PM, Mutton, James <jmutton@akamai.com> wrote:

> I can share a few quick thoughts (which I admit is rare, but I happen to
> have a few minutes and this hit the top of my stack):
>
> * How do you envision this overflowing when you have hundreds of databases
> within a cluster, or maybe 64 nodes in a cluster.
>
Attached UIs  can only handle very low level nodes (4 to 8) according to
layout (we can called it expanded layout).  When I am drawing I did not
thought that level of count of nodes and databases. Therefore I will
revisit the mockup ui. Here is updated UI

Here you can have 1 ~ 1000 node even and each node can contain 100
databases.
Node are listed and user can pick one of it then it will show it details
and databases of it in list.

UI for Listing node when nodes count is above 8 in cluster (This is how it
look as it have 64 nodes in cluster)
https://app.box.com/s/kqg2kadf0xa1zoym9sme22q5m4wbg8i5
 [Page10-list-nodes-more-than-8]

*Features of the UI:*

   - Table header support sorting (ascending and descending) by just click
   on it
   - User have searching option to search (node/databases by name)
   - It support advance search query (popup form ui)
   - Notification listed with icon and warning/alert level is shown by
   color of icon
   - User can perform node operation by clicking hyperlink in it's raw


Node contains 100 databases (answer)
https://app.box.com/s/ggbzr00orkwacsi4cflpvn9d60q7h1st
 [Page11-list-dbs-in-selected-node]



*Features of the UI*

   - In Database panel contain search that scope to databases of the
   selected node
   - When user select database it will show database summary details and
   operations that can be perform for that database



> ** On a related note, drag and drop is cool, but level of difficulty
> involved in adding a shard from node-1 increases in direct proportion to
> the number of nodes there are.

 Yes, we can add button ( in database/node level

  I think the “Add Database” button on a new server should be labeled “Add
> Shard” and should present the option of selecting what shards to add.  The
> add button is also not present on the nodes that have shards already so it
> would be good to keep it around somehow.
>
Yes,  It will be label as “Add Shard”. It is possible to show only that is
not shard so far.


* Speaking then of shards… They don’t appear to be captured here.  How does
> one know which shard-set each server has?  Each database has a number of
> shards that all have multiple shard copies and it is not a requirement that
> all servers have all shards.  None of that is really represented in these
> mockups.  When looking at a shard, it would be good to know if it is
> 00000000-15555554 or aaaaaaa8-bffffffc, or even both.  Knowing this and
> being able to use it to scale/move hotspots in a cluster is pretty
> important for operations, once databases hit a significant size.
>
Thanks for pointing out. shards will add for visualization.

Can you share resources/documentations which that help to understand the
couchdb clustering?
I like to know what are most important operations regard couch clustering?
(that you think good to added in here. since I am new for couchdb and some
time I have miss them)



> * I don’t think doc-count and deleted-count is useful information at the
> overview page

Thanks, We can remove doc count details from overview page. If user need to
read such details he/she can click 'more' so there it will show them.

>

and runs the risk of requiring a lot of requests to fulfill (see point 1 on
> hundreds of databases).
>
Sorry, In here I did not get your point

* What does compact: false mean?  That the database is not small? or that
> it’s not actively compacting?  And if the latter, wouldn’t that be on a
> db-shard basis, not a whole server.
>
compact running is mapping to 'compact_running'[2]

[1] https://wiki.apache.org/couchdb/Compaction


> * It would be nice if cluster usage graphing page would show http-stats as
> well.  Would give people a snapshot indication of where their load is hot.
>
Sure, Different status codes will be notify on cluster usage graph.
(Mockups UI will shared tonight)

Thanks JamesM for your feed backs

>
> </JamesM>
>
>
> On Mar 23, 2015, at 20:32, madhuka udantha <madhukaudantha@gmail.com>
> wrote:
>
> > Hi All,
> >
> > are there any ideas and feedback for above mockup UIs?
> >
> > Community feedback will be useful in here. :)
> >
> > On Sun, Mar 22, 2015 at 11:51 PM, madhuka udantha <
> madhukaudantha@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> Here are the mockup UIs for Visualizing the CouchDB Clusters. It is just
> >> the initial UIs.
> >>
> >> * Page 1-cluster.png* [
> >> https://app.box.com/s/isjbfop69j2cmz0kg0qar7l5nmupjnfl
> >>
> >> This Page visualize nodes in the clusters with summary information for
> >> each node. (Name, Memory usage..etc..) with a nice web UI interface. It
> is
> >> interactive (drag and drop supporting over nodes)
> >>
> >> *Page 2-cluster-notification-tip.png [*
> >> https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb
> >> <https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb>]
> >>
> >> Top of each panel of the nodes it contains a notification icon if there
> is
> >> notification.
> >> eg: Memory/disk space availability, Error in node etc...
> >> Notification will have a simple drop down structure as it is showing.
> >>
> >> *Page 3-cluster-warning.png* [
> >> https://app.box.com/s/f2ljwxvz2w01f8uepsj4zyarhwbpiz8b]
> >>
> >> Critical level alert is showen. (Alert level can be set in config and
> >> notification center)
> >>
> >> *Page 4-adding-new-node.png* [
> >> https://app.box.com/s/noygbeyze792bh4t93iaoxywdciau057]
> >>
> >> By clicking "add new node" you can add new nodes for clusters. UI
> >> Consistency is maintained
> >>
> >> *Page 5-adding-new-node-by-drag-drop.png*[
> >> https://app.box.com/s/gbfsbwb6zt1jdkijve72mnullplecpj7]
> >>
> >> User interface is supported for drag-drop of databases in nodes.
> >> (replication operation in the BE)
> >>
> >> *Page 6-selected-db-in-node.png*[
> >> https://app.box.com/s/6ojrvpxuvmo9k18dlnf8i4nmibzdx5s9]
> >> Each node shows the databases in it. By Clicking on database you can see
> >> detailed view of a particular database. This allows users to understand
> >> memory/data/doc distribution over DBs in particular nodes.
> >>
> >> *Page 7-custer-mgt-center.png* [
> >> https://app.box.com/s/t9s42g3gwvv6bocbdsxrgjimdrh2ty0z]
> >> Each node is shown with disk usage under icon and summary text of the
> node
> >> is also given. It is easy to identify the disk space - free
> >> availability. (In here we need some BE services help)
> >>
> >>
> >> *Page 8-custer-mgt-center-infor.png* [
> >> https://app.box.com/s/vvo4v9ji79l8japo3nm9f4lfbjhdamkc]
> >> User can customize notifications for each nodes by level (warning,
> infor,
> >> etc..)
> >>
> >> *Page 9-charting-disk-usages.png*[
> >> https://app.box.com/s/fk7eyoyrtg8d3b29xoo0mdho9rr8cmyz]
> >> Cluster usage of the system is shown as a bar chart and the attributes
> to
> >> be shown can be selected according to user's preferences.
> >>
> >> You ideas are welcome in here.
> >> Thanks
> >>
> >> On Sun, Mar 22, 2015 at 7:34 PM, madhuka udantha <
> madhukaudantha@gmail.com
> >>> wrote:
> >>
> >>> Hi All,
> >>>
> >>> As I'm drafting my proposal for GSoC 2015 I went through some of the
> >>> existing Cluster Management Systems with Visualization. I recognized
> some
> >>> of the features regarding to Cluster Management HTTP API. (Some times
> not
> >>> fully documented yet)
> >>>     Such us partitioning, re-balance..etc.
> >>> I was unable to find them on CouchDB API Documentation(only at the
> >>> implemented level but not documented yet? ) but I was able to find
> some of
> >>> the features such as adding nodes to clusters through dev-mailing list.
> >>> Are some of these features regarding cluster management present on the
> >>> API. If so where can I find the relevant documentations/resources? Even
> >>> the source code is fine to me.
> >>>
> >>>
> >>> Since I was able to clearly identify those missing features can I
> start a
> >>> discussion on a thread on the best feature list for cluster management
> on
> >>> CouchDB to be added to it on a future release.
> >>> My current project could also include these features that are supported
> >>> by the API else I could help on including them.
> >>>
> >>> [1] http://guide.couchdb.org/draft/clustering.html
> >>> Thanks.
> >>> BR,
> >>>
> >>> On Sat, Mar 21, 2015 at 11:40 AM, madhuka udantha <
> >>> madhukaudantha@gmail.com> wrote:
> >>>
> >>>>
> >>>>
> >>>> On Sat, Mar 21, 2015 at 12:17 AM, Alexander Shorin <kxepal@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> On Fri, Mar 20, 2015 at 9:28 PM, madhuka udantha
> >>>>> <madhukaudantha@gmail.com> wrote:
> >>>>>> I like to know what is *different of '_nodes' and 'node'*?
> >>>>>>
> >>>>>> http://localhost:15986/nodes -->{"db_name":"nodes","doc_count":3..}
> >>>>>> http://localhost:25986/nodes -->{"db_name":"nodes","doc_count":1..}
> >>>>>> http://localhost:35986/nodes
> >>>>> -->{"error":"not_found","reason":"no_db_file"}
> >>>>>
> >>>>> Where did you get these nodes?
> >>>>>
> >>>> By '~/test/1/couchdb $ dev/run -n 4' I can added 4 nodes. (default it
> is
> >>>> 3 nodes)
> >>>>
> >>>> Here is my console print, Yes now I have four node running.
> >>>>
> >>>> *max@max-VirtualBox ~/test/1/couchdb $ dev/run -n 4*
> >>>> *Setup environment... ok*
> >>>> *Ensure CouchDB is built... ok*
> >>>> *Prepare configuration files... ok*
> >>>> *Start node node1... ok*
> >>>> *Start node node2... ok*
> >>>> *Start node node3... ok*
> >>>> *Start node node4... ok*
> >>>> *Ensure all nodes are run... ok*
> >>>> *Join nodes into cluster... ok*
> >>>> *Developers cluster is set up at http://127.0.0.1:15984
> >>>> <http://127.0.0.1:15984>. Time to hack!... *
> >>>>
> >>>> Each above nodes are running in localhost in port 15986, 25986, 35986
> >>>> and 45986
> >>>>
> >>>> It seems docs are mainly in 15986 (node1) shown by 'doc_count' value,
> >>>> is there any reason for that or is it happening randomly?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>
> >>>>> --
> >>>>> ,,,^..^,,,
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Cheers,
> >>>> Madhuka Udantha
> >>>> http://madhukaudantha.blogspot.com
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Cheers,
> >>> Madhuka Udantha
> >>> http://madhukaudantha.blogspot.com
> >>>
> >>
> >>
> >>
> >> --
> >> Cheers,
> >> Madhuka Udantha
> >> http://madhukaudantha.blogspot.com
> >>
> >
> >
> >
> > --
> > Cheers,
> > Madhuka Udantha
> > http://madhukaudantha.blogspot.com
>
>


-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message