couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Metson <>
Subject Re: Is CouchDB the right tool for this?
Date Wed, 28 Dec 2011 22:29:21 GMT
Hey Joe,
   I've thought about writing something like this off and on for a while, and I think couch
is a good fit (depending on what granularity of data you need to store and associated life
tike, frequency and volumes - there are better options for some regions of phase space). It's
pretty easy to hook up a logger to write json into couch as well as log to files (I have some
demo python code somewhere if you're interested) and visualising live data in couch with something
like d3 is a piece of cake.
   One thing you might need to consider is having tiered levels of detail, so that machines
have very detailed info about themselves but only propagate up a subset to a rack/region server,
which in turn propagates up a subset to some master server. Drilling down through the servers
is easy assuming you access from inside the network or have some kind of bastion/proxy/VPN.
The replication between the servers can be done using couch's replicator (killer feature)
but you might need to be clever about how you filter the data; we've found filter functions
to be slow enough to cause problems (though it'll be very dependent on what you filter and
the rate that data is inserted). 
  I'd be interested in collaborating on something if that's possible. 

On 24 Dec 2011, at 20:54, Joe Hillenbrand <> wrote:

> I'm building an server monitoring application.
> The basic idea is that there will be several nodes that will monitor
> metrics about themselves and store them as JSON docs locally. Then
> there will be a single host server that will collect these metrics for
> the use of a web application.
> CouchDB seems like a good options for this because each node could
> have their own CouchDB database and then eventually consolidate this
> data from all the nodes to the host server where it could be queried.
> If the host fails, then the nodes can just keep storing their data
> locally until it is ready to be received by the host.
> I have a faint recollection from when I first learned CouchDB that it
> had these kind of replication mechanics built in (something kind of
> like UbuntuOne), but when I look over guide and documentation, I don't
> find anything like this. Am I mistaken? Is there a better option?
> Thanks,
> -Joe

View raw message