couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Kowalski <...@kowalski.gd>
Subject Re: [GSOC 2015] COUCHDB-2605 Visualize the CouchDB Cluster
Date Fri, 27 Mar 2015 13:43:45 GMT
yes thank you!

On Fri, Mar 27, 2015 at 7:02 AM, madhuka udantha
<madhukaudantha@gmail.com> wrote:
> Hi,
>
> I'm sharing a sample mxn Matrix Graph[1] so you all can get an idea of what
> my proposed solution is. This can represent high number of nodes with out
> any issue and  at a glance you can get the Summarized View of the Cluster.
> Accordingly my mxn Matrix would represent nodes in the Cluster.
> (Mockup UIs are attached to JIRA)
>
> Does this clarify the issue Robert?
>
> Your feedback / ideas mostly welcomed.
>
> [1] http://bost.ocks.org/mike/miserables/
>
> On Wed, Mar 25, 2015 at 11:46 PM, madhuka udantha <madhukaudantha@gmail.com>
> wrote:
>>
>> Hi,
>>
>> Sharing mockups UIs again, Since some ML-Users do not have account in
>> box.com.
>> I am sharing them via jira.
>>
>> On Wed, Mar 25, 2015 at 12:20 AM, madhuka udantha
>> <madhukaudantha@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> After communicating with couchdb community got to know that  there can be
>>> 100~ 10000 nodes in cluster. So there will 2 layouts in cluster link
>>> depending on node count in cluster.
>>>
>>> My first UI is for simple count of node in cluster (4 nodes in cluster)
>>>
>>> Here is mock UIs where it hit more than 5 to 100000 nodes, I am using
>>> matrix visual.
>>> Here is initial look of the page. [Page1v1-initial-page]
>>> https://app.box.com/s/sn84u6wuzya58trssw11ba6fwtmn862f
>>
>>
>> https://issues.apache.org/jira/secure/attachment/12707205/Page1v1-initial-page.png
>>
>>>
>>> Features:
>>> 1. User can select the attribute eg: Disk usage, http status, data usage,
>>> alerts. etc.
>>> 2. Chart / matrix will be draw with legend
>>>
>>> [Page2v1-searching-node]
>>> https://app.box.com/s/i5uomdhirn96co6acbbrvvviifyw95pb
>>
>>
>> https://issues.apache.org/jira/secure/attachment/12707218/Page2v1-searching-node.png
>>
>>>
>>> Who can we search node in there?
>>> You will have search in top of it. (it allow to search the node), It will
>>> show it with red color box.
>>> Or every node have tool tip to verify node, what is you look for.
>>>
>>>
>>> [Page3v1-selecting-node]
>>> https://app.box.com/s/rsd5ysld597gvra2nkcn7027bpok37jc
>>
>>
>> https://issues.apache.org/jira/secure/attachment/12707219/Page3v1-selecting-node.png
>>
>>>
>>> When user click on node, node panel will be open RHS.
>>> It have all the details of the node and operation that you can perform in
>>> node leve
>>>
>>> [Page4v1-searching-db_in_node]
>>> https://app.box.com/s/jk3ojb0khj192ldnyw0eqs1p5t7l485r
>>
>>
>> https://issues.apache.org/jira/secure/attachment/12707220/Page4v1-searching-db_in_node.png
>>
>>>
>>> User can search Databases in nodes even
>>> This searching have advance search query support
>>>
>>> [Page5v1-db_view]
>>> https://app.box.com/s/20hjgtcb7c9ikgfdn7xy8xli4uhorbni
>>
>>
>> https://issues.apache.org/jira/secure/attachment/12707206/Page5v1-db_view.png
>>
>>> User can select the database and it will show database details.
>>>
>>>
>>>  feed backs for UIs are welcome.
>>>
>>>
>>> On Tue, Mar 24, 2015 at 4:35 PM, madhuka udantha
>>> <madhukaudantha@gmail.com> wrote:
>>>>
>>>> Hi James,
>>>>
>>>> Thanks for you feedback. It will help me to improve my proposal and to
>>>> get better understanding about couchdb cluster
>>>>
>>>> Since I am new to couchdb and I did not know much user case that you all
>>>> have come across. But when you share those ideas. It is help to understand
>>>> couchdb cluster. All you feedback is important.
>>>>
>>>> On Tue, Mar 24, 2015 at 12:07 PM, Mutton, James <jmutton@akamai.com>
>>>> wrote:
>>>>>
>>>>> I can share a few quick thoughts (which I admit is rare, but I happen
>>>>> to have a few minutes and this hit the top of my stack):
>>>>>
>>>>> * How do you envision this overflowing when you have hundreds of
>>>>> databases within a cluster, or maybe 64 nodes in a cluster.
>>>>
>>>> Attached UIs  can only handle very low level nodes (4 to 8) according to
>>>> layout (we can called it expanded layout).  When I am drawing I did not
>>>> thought that level of count of nodes and databases. Therefore I will revisit
>>>> the mockup ui. Here is updated UI
>>>>
>>>> Here you can have 1 ~ 1000 node even and each node can contain 100
>>>> databases.
>>>> Node are listed and user can pick one of it then it will show it details
>>>> and databases of it in list.
>>>>
>>>> UI for Listing node when nodes count is above 8 in cluster (This is how
>>>> it look as it have 64 nodes in cluster)
>>>> https://app.box.com/s/kqg2kadf0xa1zoym9sme22q5m4wbg8i5
>>>> [Page10-list-nodes-more-than-8]
>>>>
>>>> Features of the UI:
>>>>
>>>> Table header support sorting (ascending and descending) by just click on
>>>> it
>>>> User have searching option to search (node/databases by name)
>>>> It support advance search query (popup form ui)
>>>> Notification listed with icon and warning/alert level is shown by color
>>>> of icon
>>>> User can perform node operation by clicking hyperlink in it's raw
>>>>
>>>>
>>>> Node contains 100 databases (answer)
>>>> https://app.box.com/s/ggbzr00orkwacsi4cflpvn9d60q7h1st
>>>> [Page11-list-dbs-in-selected-node]
>>>>
>>>>
>>>>
>>>> Features of the UI
>>>>
>>>> In Database panel contain search that scope to databases of the selected
>>>> node
>>>> When user select database it will show database summary details and
>>>> operations that can be perform for that database
>>>>
>>>>
>>>>>
>>>>> ** On a related note, drag and drop is cool, but level of difficulty
>>>>> involved in adding a shard from node-1 increases in direct proportion
to the
>>>>> number of nodes there are.
>>>>
>>>>  Yes, we can add button ( in database/node level
>>>>
>>>>>   I think the “Add Database” button on a new server should be labeled
>>>>> “Add Shard” and should present the option of selecting what shards
to add.
>>>>> The add button is also not present on the nodes that have shards already
so
>>>>> it would be good to keep it around somehow.
>>>>
>>>> Yes,  It will be label as “Add Shard”. It is possible to show only that
>>>> is not shard so far.
>>>>
>>>>
>>>>> * Speaking then of shards… They don’t appear to be captured here.
 How
>>>>> does one know which shard-set each server has?  Each database has a number
>>>>> of shards that all have multiple shard copies and it is not a requirement
>>>>> that all servers have all shards.  None of that is really represented
in
>>>>> these mockups.  When looking at a shard, it would be good to know if
it is
>>>>> 00000000-15555554 or aaaaaaa8-bffffffc, or even both.  Knowing this and
>>>>> being able to use it to scale/move hotspots in a cluster is pretty important
>>>>> for operations, once databases hit a significant size.
>>>>
>>>> Thanks for pointing out. shards will add for visualization.
>>>>
>>>> Can you share resources/documentations which that help to understand the
>>>> couchdb clustering?
>>>> I like to know what are most important operations regard couch
>>>> clustering? (that you think good to added in here. since I am new for
>>>> couchdb and some time I have miss them)
>>>>
>>>>
>>>>>
>>>>> * I don’t think doc-count and deleted-count is useful information at
>>>>> the overview page
>>>>
>>>> Thanks, We can remove doc count details from overview page. If user need
>>>> to read such details he/she can click 'more' so there it will show them.
>>>>>
>>>>>
>>>>>
>>>>> and runs the risk of requiring a lot of requests to fulfill (see point
>>>>> 1 on hundreds of databases).
>>>>
>>>> Sorry, In here I did not get your point
>>>>
>>>>> * What does compact: false mean?  That the database is not small? or
>>>>> that it’s not actively compacting?  And if the latter, wouldn’t that
be on a
>>>>> db-shard basis, not a whole server.
>>>>
>>>> compact running is mapping to 'compact_running'[2]
>>>>
>>>> [1] https://wiki.apache.org/couchdb/Compaction
>>>>
>>>>>
>>>>> * It would be nice if cluster usage graphing page would show http-stats
>>>>> as well.  Would give people a snapshot indication of where their load
is
>>>>> hot.
>>>>
>>>> Sure, Different status codes will be notify on cluster usage graph.
>>>> (Mockups UI will shared tonight)
>>>>
>>>> Thanks JamesM for your feed backs
>>>>>
>>>>>
>>>>> </JamesM>
>>>>>
>>>>>
>>>>> On Mar 23, 2015, at 20:32, madhuka udantha <madhukaudantha@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi All,
>>>>> >
>>>>> > are there any ideas and feedback for above mockup UIs?
>>>>> >
>>>>> > Community feedback will be useful in here. :)
>>>>> >
>>>>> > On Sun, Mar 22, 2015 at 11:51 PM, madhuka udantha
>>>>> > <madhukaudantha@gmail.com>
>>>>> > wrote:
>>>>> >
>>>>> >> Hi,
>>>>> >>
>>>>> >> Here are the mockup UIs for Visualizing the CouchDB Clusters.
It is
>>>>> >> just
>>>>> >> the initial UIs.
>>>>> >>
>>>>> >> * Page 1-cluster.png* [
>>>>> >> https://app.box.com/s/isjbfop69j2cmz0kg0qar7l5nmupjnfl
>>>>> >>
>>>>> >> This Page visualize nodes in the clusters with summary information
>>>>> >> for
>>>>> >> each node. (Name, Memory usage..etc..) with a nice web UI interface.
>>>>> >> It is
>>>>> >> interactive (drag and drop supporting over nodes)
>>>>> >>
>>>>> >> *Page 2-cluster-notification-tip.png [*
>>>>> >> https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb
>>>>> >> <https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb>]
>>>>> >>
>>>>> >> Top of each panel of the nodes it contains a notification icon
if
>>>>> >> there is
>>>>> >> notification.
>>>>> >> eg: Memory/disk space availability, Error in node etc...
>>>>> >> Notification will have a simple drop down structure as it is
>>>>> >> showing.
>>>>> >>
>>>>> >> *Page 3-cluster-warning.png* [
>>>>> >> https://app.box.com/s/f2ljwxvz2w01f8uepsj4zyarhwbpiz8b]
>>>>> >>
>>>>> >> Critical level alert is showen. (Alert level can be set in config
>>>>> >> and
>>>>> >> notification center)
>>>>> >>
>>>>> >> *Page 4-adding-new-node.png* [
>>>>> >> https://app.box.com/s/noygbeyze792bh4t93iaoxywdciau057]
>>>>> >>
>>>>> >> By clicking "add new node" you can add new nodes for clusters.
UI
>>>>> >> Consistency is maintained
>>>>> >>
>>>>> >> *Page 5-adding-new-node-by-drag-drop.png*[
>>>>> >> https://app.box.com/s/gbfsbwb6zt1jdkijve72mnullplecpj7]
>>>>> >>
>>>>> >> User interface is supported for drag-drop of databases in nodes.
>>>>> >> (replication operation in the BE)
>>>>> >>
>>>>> >> *Page 6-selected-db-in-node.png*[
>>>>> >> https://app.box.com/s/6ojrvpxuvmo9k18dlnf8i4nmibzdx5s9]
>>>>> >> Each node shows the databases in it. By Clicking on database
you can
>>>>> >> see
>>>>> >> detailed view of a particular database. This allows users to
>>>>> >> understand
>>>>> >> memory/data/doc distribution over DBs in particular nodes.
>>>>> >>
>>>>> >> *Page 7-custer-mgt-center.png* [
>>>>> >> https://app.box.com/s/t9s42g3gwvv6bocbdsxrgjimdrh2ty0z]
>>>>> >> Each node is shown with disk usage under icon and summary text
of
>>>>> >> the node
>>>>> >> is also given. It is easy to identify the disk space - free
>>>>> >> availability. (In here we need some BE services help)
>>>>> >>
>>>>> >>
>>>>> >> *Page 8-custer-mgt-center-infor.png* [
>>>>> >> https://app.box.com/s/vvo4v9ji79l8japo3nm9f4lfbjhdamkc]
>>>>> >> User can customize notifications for each nodes by level (warning,
>>>>> >> infor,
>>>>> >> etc..)
>>>>> >>
>>>>> >> *Page 9-charting-disk-usages.png*[
>>>>> >> https://app.box.com/s/fk7eyoyrtg8d3b29xoo0mdho9rr8cmyz]
>>>>> >> Cluster usage of the system is shown as a bar chart and the
>>>>> >> attributes to
>>>>> >> be shown can be selected according to user's preferences.
>>>>> >>
>>>>> >> You ideas are welcome in here.
>>>>> >> Thanks
>>>>> >>
>>>>> >> On Sun, Mar 22, 2015 at 7:34 PM, madhuka udantha
>>>>> >> <madhukaudantha@gmail.com
>>>>> >>> wrote:
>>>>> >>
>>>>> >>> Hi All,
>>>>> >>>
>>>>> >>> As I'm drafting my proposal for GSoC 2015 I went through
some of
>>>>> >>> the
>>>>> >>> existing Cluster Management Systems with Visualization.
I
>>>>> >>> recognized some
>>>>> >>> of the features regarding to Cluster Management HTTP API.
(Some
>>>>> >>> times not
>>>>> >>> fully documented yet)
>>>>> >>>     Such us partitioning, re-balance..etc.
>>>>> >>> I was unable to find them on CouchDB API Documentation(only
at the
>>>>> >>> implemented level but not documented yet? ) but I was able
to find
>>>>> >>> some of
>>>>> >>> the features such as adding nodes to clusters through dev-mailing
>>>>> >>> list.
>>>>> >>> Are some of these features regarding cluster management
present on
>>>>> >>> the
>>>>> >>> API. If so where can I find the relevant documentations/resources?
>>>>> >>> Even
>>>>> >>> the source code is fine to me.
>>>>> >>>
>>>>> >>>
>>>>> >>> Since I was able to clearly identify those missing features
can I
>>>>> >>> start a
>>>>> >>> discussion on a thread on the best feature list for cluster
>>>>> >>> management on
>>>>> >>> CouchDB to be added to it on a future release.
>>>>> >>> My current project could also include these features that
are
>>>>> >>> supported
>>>>> >>> by the API else I could help on including them.
>>>>> >>>
>>>>> >>> [1] http://guide.couchdb.org/draft/clustering.html
>>>>> >>> Thanks.
>>>>> >>> BR,
>>>>> >>>
>>>>> >>> On Sat, Mar 21, 2015 at 11:40 AM, madhuka udantha <
>>>>> >>> madhukaudantha@gmail.com> wrote:
>>>>> >>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Sat, Mar 21, 2015 at 12:17 AM, Alexander Shorin
>>>>> >>>> <kxepal@gmail.com>
>>>>> >>>> wrote:
>>>>> >>>>
>>>>> >>>>> On Fri, Mar 20, 2015 at 9:28 PM, madhuka udantha
>>>>> >>>>> <madhukaudantha@gmail.com> wrote:
>>>>> >>>>>> I like to know what is *different of '_nodes'
and 'node'*?
>>>>> >>>>>>
>>>>> >>>>>> http://localhost:15986/nodes
>>>>> >>>>>> -->{"db_name":"nodes","doc_count":3..}
>>>>> >>>>>> http://localhost:25986/nodes
>>>>> >>>>>> -->{"db_name":"nodes","doc_count":1..}
>>>>> >>>>>> http://localhost:35986/nodes
>>>>> >>>>> -->{"error":"not_found","reason":"no_db_file"}
>>>>> >>>>>
>>>>> >>>>> Where did you get these nodes?
>>>>> >>>>>
>>>>> >>>> By '~/test/1/couchdb $ dev/run -n 4' I can added 4 nodes.
(default
>>>>> >>>> it is
>>>>> >>>> 3 nodes)
>>>>> >>>>
>>>>> >>>> Here is my console print, Yes now I have four node running.
>>>>> >>>>
>>>>> >>>> *max@max-VirtualBox ~/test/1/couchdb $ dev/run -n 4*
>>>>> >>>> *Setup environment... ok*
>>>>> >>>> *Ensure CouchDB is built... ok*
>>>>> >>>> *Prepare configuration files... ok*
>>>>> >>>> *Start node node1... ok*
>>>>> >>>> *Start node node2... ok*
>>>>> >>>> *Start node node3... ok*
>>>>> >>>> *Start node node4... ok*
>>>>> >>>> *Ensure all nodes are run... ok*
>>>>> >>>> *Join nodes into cluster... ok*
>>>>> >>>> *Developers cluster is set up at http://127.0.0.1:15984
>>>>> >>>> <http://127.0.0.1:15984>. Time to hack!... *
>>>>> >>>>
>>>>> >>>> Each above nodes are running in localhost in port 15986,
25986,
>>>>> >>>> 35986
>>>>> >>>> and 45986
>>>>> >>>>
>>>>> >>>> It seems docs are mainly in 15986 (node1) shown by 'doc_count'
>>>>> >>>> value,
>>>>> >>>> is there any reason for that or is it happening randomly?
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>>
>>>>> >>>>> --
>>>>> >>>>> ,,,^..^,,,
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> Cheers,
>>>>> >>>> Madhuka Udantha
>>>>> >>>> http://madhukaudantha.blogspot.com
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> Cheers,
>>>>> >>> Madhuka Udantha
>>>>> >>> http://madhukaudantha.blogspot.com
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Cheers,
>>>>> >> Madhuka Udantha
>>>>> >> http://madhukaudantha.blogspot.com
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Cheers,
>>>>> > Madhuka Udantha
>>>>> > http://madhukaudantha.blogspot.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Madhuka Udantha
>>>> http://madhukaudantha.blogspot.com
>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Madhuka Udantha
>>> http://madhukaudantha.blogspot.com
>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com

Mime
View raw message