couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From madhuka udantha <madhukaudan...@gmail.com>
Subject Re: [GSOC 2015] COUCHDB-2605 Visualize the CouchDB Cluster
Date Wed, 25 Mar 2015 18:06:39 GMT
Hi All,

Since this about couchDB cluster as per requested by Robert Kowalski I just
had an IRC chat with Robert Newson who is an expert on Clusters in CouchDB.
I am sharing this in order to other new comes (GSOC applicants) also
benefit from this

I was pointed to the following resources.
http://docs.cloudant.com/guides.html#cap-theorem
https://cloudant.com/blog/dynamo-and-couchdb-clusters/
http://bigcouch.cloudant.com/api
https://github.com/cloudant/bigcouch

Also I was pointed to  'GET /_membership'

That's the endpoint that tells you how many nodes are in a cluster.

10,000 nodes is far, far higher than any cluster we've ever seen.
hundreds would be a more reasonable maximum expectation
and dozens much more common

Bigcouch or couchdb are the 2 variants for cluster and stand-alone systems
respectively.

couchDB perview 2.0 is the later version of bigcouch and cloudant was also
highlighted in the chat which refers to the same system.

Robert Newson, If I have missed any important point please feel free to
share.

On Wed, Mar 25, 2015 at 6:13 PM, madhuka udantha <madhukaudantha@gmail.com>
wrote:

> looping Robert Newson
>
> According to Robert Kowalski's suggestion I'm sharing my mockup UI's
> regarding to CouchDB Cluster.
>
> Your feedback will be mostly welcomed.
>
> Thanks in Advance.
>
>
>
>
>
>
>
> On Wed, Mar 25, 2015 at 12:20 AM, madhuka udantha <
> madhukaudantha@gmail.com> wrote:
>
>> Hi,
>>
>> After communicating with couchdb community got to know that  there can be
>> 100~ 10000 nodes in cluster. So there will 2 layouts in cluster link
>> depending on node count in cluster.
>>
>> My first UI is for simple count of node in cluster (4 nodes in cluster)
>>
>> Here is mock UIs where it hit more than 5 to 100000 nodes, I am using
>> matrix visual.
>> Here is initial look of the page. [Page1v1-initial-page]
>> https://app.box.com/s/sn84u6wuzya58trssw11ba6fwtmn862f
>> Features:
>> 1. User can select the attribute eg: Disk usage, http status, data usage,
>> alerts. etc.
>> 2. Chart / matrix will be draw with legend
>>
>> [Page2v1-searching-node]
>> https://app.box.com/s/i5uomdhirn96co6acbbrvvviifyw95pb
>> Who can we search node in there?
>> You will have search in top of it. (it allow to search the node), It will
>> show it with red color box.
>> Or every node have tool tip to verify node, what is you look for.
>>
>>
>> [Page3v1-selecting-node]
>> https://app.box.com/s/rsd5ysld597gvra2nkcn7027bpok37jc
>> When user click on node, node panel will be open RHS.
>> It have all the details of the node and operation that you can perform in
>> node leve
>>
>> [Page4v1-searching-db_in_node]
>> https://app.box.com/s/jk3ojb0khj192ldnyw0eqs1p5t7l485r
>> User can search Databases in nodes even
>> This searching have advance search query support
>>
>> [Page5v1-db_view]
>> https://app.box.com/s/20hjgtcb7c9ikgfdn7xy8xli4uhorbni
>> User can select the database and it will show database details.
>>
>>
>>  feed backs for UIs are welcome.
>>
>>
>> On Tue, Mar 24, 2015 at 4:35 PM, madhuka udantha <
>> madhukaudantha@gmail.com> wrote:
>>
>>> Hi James,
>>>
>>> Thanks for you feedback. It will help me to improve my proposal and to
>>> get better understanding about couchdb cluster
>>>
>>> Since I am new to couchdb and I did not know much user case that you all
>>> have come across. But when you share those ideas. It is help to understand
>>> couchdb cluster. All you feedback is important.
>>>
>>> On Tue, Mar 24, 2015 at 12:07 PM, Mutton, James <jmutton@akamai.com>
>>> wrote:
>>>
>>>> I can share a few quick thoughts (which I admit is rare, but I happen
>>>> to have a few minutes and this hit the top of my stack):
>>>>
>>>> * How do you envision this overflowing when you have hundreds of
>>>> databases within a cluster, or maybe 64 nodes in a cluster.
>>>>
>>> Attached UIs  can only handle very low level nodes (4 to 8) according to
>>> layout (we can called it expanded layout).  When I am drawing I did not
>>> thought that level of count of nodes and databases. Therefore I will
>>> revisit the mockup ui. Here is updated UI
>>>
>>> Here you can have 1 ~ 1000 node even and each node can contain 100
>>> databases.
>>> Node are listed and user can pick one of it then it will show it details
>>> and databases of it in list.
>>>
>>> UI for Listing node when nodes count is above 8 in cluster (This is how
>>> it look as it have 64 nodes in cluster)
>>> https://app.box.com/s/kqg2kadf0xa1zoym9sme22q5m4wbg8i5
>>>  [Page10-list-nodes-more-than-8]
>>>
>>> *Features of the UI:*
>>>
>>>    - Table header support sorting (ascending and descending) by just
>>>    click on it
>>>    - User have searching option to search (node/databases by name)
>>>    - It support advance search query (popup form ui)
>>>    - Notification listed with icon and warning/alert level is shown by
>>>    color of icon
>>>    - User can perform node operation by clicking hyperlink in it's raw
>>>
>>>
>>> Node contains 100 databases (answer)
>>> https://app.box.com/s/ggbzr00orkwacsi4cflpvn9d60q7h1st
>>>  [Page11-list-dbs-in-selected-node]
>>>
>>>
>>>
>>> *Features of the UI*
>>>
>>>    - In Database panel contain search that scope to databases of the
>>>    selected node
>>>    - When user select database it will show database summary details
>>>    and operations that can be perform for that database
>>>
>>>
>>>
>>>> ** On a related note, drag and drop is cool, but level of difficulty
>>>> involved in adding a shard from node-1 increases in direct proportion to
>>>> the number of nodes there are.
>>>
>>>  Yes, we can add button ( in database/node level
>>>
>>>   I think the “Add Database” button on a new server should be labeled
>>>> “Add Shard” and should present the option of selecting what shards to
add.
>>>> The add button is also not present on the nodes that have shards already
so
>>>> it would be good to keep it around somehow.
>>>>
>>> Yes,  It will be label as “Add Shard”. It is possible to show only that
>>> is not shard so far.
>>>
>>>
>>> * Speaking then of shards… They don’t appear to be captured here.  How
>>>> does one know which shard-set each server has?  Each database has a number
>>>> of shards that all have multiple shard copies and it is not a requirement
>>>> that all servers have all shards.  None of that is really represented in
>>>> these mockups.  When looking at a shard, it would be good to know if it is
>>>> 00000000-15555554 or aaaaaaa8-bffffffc, or even both.  Knowing this and
>>>> being able to use it to scale/move hotspots in a cluster is pretty
>>>> important for operations, once databases hit a significant size.
>>>>
>>> Thanks for pointing out. shards will add for visualization.
>>>
>>> Can you share resources/documentations which that help to understand the
>>> couchdb clustering?
>>> I like to know what are most important operations regard couch
>>> clustering? (that you think good to added in here. since I am new for
>>> couchdb and some time I have miss them)
>>>
>>>
>>>
>>>> * I don’t think doc-count and deleted-count is useful information at
>>>> the overview page
>>>
>>> Thanks, We can remove doc count details from overview page. If user need
>>> to read such details he/she can click 'more' so there it will show them.
>>>
>>>>
>>>
>>> and runs the risk of requiring a lot of requests to fulfill (see point 1
>>>> on hundreds of databases).
>>>>
>>> Sorry, In here I did not get your point
>>>
>>> * What does compact: false mean?  That the database is not small? or
>>>> that it’s not actively compacting?  And if the latter, wouldn’t that
be on
>>>> a db-shard basis, not a whole server.
>>>>
>>> compact running is mapping to 'compact_running'[2]
>>>
>>> [1] https://wiki.apache.org/couchdb/Compaction
>>>
>>>
>>>> * It would be nice if cluster usage graphing page would show http-stats
>>>> as well.  Would give people a snapshot indication of where their load is
>>>> hot.
>>>>
>>> Sure, Different status codes will be notify on cluster usage graph.
>>> (Mockups UI will shared tonight)
>>>
>>> Thanks JamesM for your feed backs
>>>
>>>>
>>>> </JamesM>
>>>>
>>>>
>>>> On Mar 23, 2015, at 20:32, madhuka udantha <madhukaudantha@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi All,
>>>> >
>>>> > are there any ideas and feedback for above mockup UIs?
>>>> >
>>>> > Community feedback will be useful in here. :)
>>>> >
>>>> > On Sun, Mar 22, 2015 at 11:51 PM, madhuka udantha <
>>>> madhukaudantha@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> Hi,
>>>> >>
>>>> >> Here are the mockup UIs for Visualizing the CouchDB Clusters. It
is
>>>> just
>>>> >> the initial UIs.
>>>> >>
>>>> >> * Page 1-cluster.png* [
>>>> >> https://app.box.com/s/isjbfop69j2cmz0kg0qar7l5nmupjnfl
>>>> >>
>>>> >> This Page visualize nodes in the clusters with summary information
>>>> for
>>>> >> each node. (Name, Memory usage..etc..) with a nice web UI interface.
>>>> It is
>>>> >> interactive (drag and drop supporting over nodes)
>>>> >>
>>>> >> *Page 2-cluster-notification-tip.png [*
>>>> >> https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb
>>>> >> <https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb>]
>>>> >>
>>>> >> Top of each panel of the nodes it contains a notification icon if
>>>> there is
>>>> >> notification.
>>>> >> eg: Memory/disk space availability, Error in node etc...
>>>> >> Notification will have a simple drop down structure as it is showing.
>>>> >>
>>>> >> *Page 3-cluster-warning.png* [
>>>> >> https://app.box.com/s/f2ljwxvz2w01f8uepsj4zyarhwbpiz8b]
>>>> >>
>>>> >> Critical level alert is showen. (Alert level can be set in config
and
>>>> >> notification center)
>>>> >>
>>>> >> *Page 4-adding-new-node.png* [
>>>> >> https://app.box.com/s/noygbeyze792bh4t93iaoxywdciau057]
>>>> >>
>>>> >> By clicking "add new node" you can add new nodes for clusters. UI
>>>> >> Consistency is maintained
>>>> >>
>>>> >> *Page 5-adding-new-node-by-drag-drop.png*[
>>>> >> https://app.box.com/s/gbfsbwb6zt1jdkijve72mnullplecpj7]
>>>> >>
>>>> >> User interface is supported for drag-drop of databases in nodes.
>>>> >> (replication operation in the BE)
>>>> >>
>>>> >> *Page 6-selected-db-in-node.png*[
>>>> >> https://app.box.com/s/6ojrvpxuvmo9k18dlnf8i4nmibzdx5s9]
>>>> >> Each node shows the databases in it. By Clicking on database you
can
>>>> see
>>>> >> detailed view of a particular database. This allows users to
>>>> understand
>>>> >> memory/data/doc distribution over DBs in particular nodes.
>>>> >>
>>>> >> *Page 7-custer-mgt-center.png* [
>>>> >> https://app.box.com/s/t9s42g3gwvv6bocbdsxrgjimdrh2ty0z]
>>>> >> Each node is shown with disk usage under icon and summary text of
>>>> the node
>>>> >> is also given. It is easy to identify the disk space - free
>>>> >> availability. (In here we need some BE services help)
>>>> >>
>>>> >>
>>>> >> *Page 8-custer-mgt-center-infor.png* [
>>>> >> https://app.box.com/s/vvo4v9ji79l8japo3nm9f4lfbjhdamkc]
>>>> >> User can customize notifications for each nodes by level (warning,
>>>> infor,
>>>> >> etc..)
>>>> >>
>>>> >> *Page 9-charting-disk-usages.png*[
>>>> >> https://app.box.com/s/fk7eyoyrtg8d3b29xoo0mdho9rr8cmyz]
>>>> >> Cluster usage of the system is shown as a bar chart and the
>>>> attributes to
>>>> >> be shown can be selected according to user's preferences.
>>>> >>
>>>> >> You ideas are welcome in here.
>>>> >> Thanks
>>>> >>
>>>> >> On Sun, Mar 22, 2015 at 7:34 PM, madhuka udantha <
>>>> madhukaudantha@gmail.com
>>>> >>> wrote:
>>>> >>
>>>> >>> Hi All,
>>>> >>>
>>>> >>> As I'm drafting my proposal for GSoC 2015 I went through some
of the
>>>> >>> existing Cluster Management Systems with Visualization. I
>>>> recognized some
>>>> >>> of the features regarding to Cluster Management HTTP API. (Some
>>>> times not
>>>> >>> fully documented yet)
>>>> >>>     Such us partitioning, re-balance..etc.
>>>> >>> I was unable to find them on CouchDB API Documentation(only
at the
>>>> >>> implemented level but not documented yet? ) but I was able to
find
>>>> some of
>>>> >>> the features such as adding nodes to clusters through dev-mailing
>>>> list.
>>>> >>> Are some of these features regarding cluster management present
on
>>>> the
>>>> >>> API. If so where can I find the relevant documentations/resources?
>>>> Even
>>>> >>> the source code is fine to me.
>>>> >>>
>>>> >>>
>>>> >>> Since I was able to clearly identify those missing features
can I
>>>> start a
>>>> >>> discussion on a thread on the best feature list for cluster
>>>> management on
>>>> >>> CouchDB to be added to it on a future release.
>>>> >>> My current project could also include these features that are
>>>> supported
>>>> >>> by the API else I could help on including them.
>>>> >>>
>>>> >>> [1] http://guide.couchdb.org/draft/clustering.html
>>>> >>> Thanks.
>>>> >>> BR,
>>>> >>>
>>>> >>> On Sat, Mar 21, 2015 at 11:40 AM, madhuka udantha <
>>>> >>> madhukaudantha@gmail.com> wrote:
>>>> >>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On Sat, Mar 21, 2015 at 12:17 AM, Alexander Shorin <
>>>> kxepal@gmail.com>
>>>> >>>> wrote:
>>>> >>>>
>>>> >>>>> On Fri, Mar 20, 2015 at 9:28 PM, madhuka udantha
>>>> >>>>> <madhukaudantha@gmail.com> wrote:
>>>> >>>>>> I like to know what is *different of '_nodes' and
'node'*?
>>>> >>>>>>
>>>> >>>>>> http://localhost:15986/nodes
>>>> -->{"db_name":"nodes","doc_count":3..}
>>>> >>>>>> http://localhost:25986/nodes
>>>> -->{"db_name":"nodes","doc_count":1..}
>>>> >>>>>> http://localhost:35986/nodes
>>>> >>>>> -->{"error":"not_found","reason":"no_db_file"}
>>>> >>>>>
>>>> >>>>> Where did you get these nodes?
>>>> >>>>>
>>>> >>>> By '~/test/1/couchdb $ dev/run -n 4' I can added 4 nodes.
(default
>>>> it is
>>>> >>>> 3 nodes)
>>>> >>>>
>>>> >>>> Here is my console print, Yes now I have four node running.
>>>> >>>>
>>>> >>>> *max@max-VirtualBox ~/test/1/couchdb $ dev/run -n 4*
>>>> >>>> *Setup environment... ok*
>>>> >>>> *Ensure CouchDB is built... ok*
>>>> >>>> *Prepare configuration files... ok*
>>>> >>>> *Start node node1... ok*
>>>> >>>> *Start node node2... ok*
>>>> >>>> *Start node node3... ok*
>>>> >>>> *Start node node4... ok*
>>>> >>>> *Ensure all nodes are run... ok*
>>>> >>>> *Join nodes into cluster... ok*
>>>> >>>> *Developers cluster is set up at http://127.0.0.1:15984
>>>> >>>> <http://127.0.0.1:15984>. Time to hack!... *
>>>> >>>>
>>>> >>>> Each above nodes are running in localhost in port 15986,
25986,
>>>> 35986
>>>> >>>> and 45986
>>>> >>>>
>>>> >>>> It seems docs are mainly in 15986 (node1) shown by 'doc_count'
>>>> value,
>>>> >>>> is there any reason for that or is it happening randomly?
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> ,,,^..^,,,
>>>> >>>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> --
>>>> >>>> Cheers,
>>>> >>>> Madhuka Udantha
>>>> >>>> http://madhukaudantha.blogspot.com
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Cheers,
>>>> >>> Madhuka Udantha
>>>> >>> http://madhukaudantha.blogspot.com
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Cheers,
>>>> >> Madhuka Udantha
>>>> >> http://madhukaudantha.blogspot.com
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Cheers,
>>>> > Madhuka Udantha
>>>> > http://madhukaudantha.blogspot.com
>>>>
>>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Madhuka Udantha
>>> http://madhukaudantha.blogspot.com
>>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message