couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Anderson <jch...@apache.org>
Subject Re: Objective criteria
Date Thu, 14 Jan 2010 04:46:15 GMT
On Wed, Jan 13, 2010 at 7:21 PM, Roger Binns <rogerb@rogerbinns.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Damien Katz wrote:
>>> That's fine, the issue is that bugs saying "It's too slow" is always true for
someone.
>
> I did give specific numbers - ie 10 million documents, 2GB of JSON data etc
> and the amount of time taken as well as space.  I doubt you'd find anyone
> that considers 4 hours or 27GB to be reasonable numbers for that :-)
>
>> Many people find the view indexing performance just fine, many do not.
>
> True.  For example one of my other projects currently has 100 documents and
> I have no issue any part of CouchDB for that.
>
> What isn't clear is statement of reasonable expectations - should I be able
> to handle 10 million documents in CouchDB?  Will it ever handle that?  Do
> you as the project leader care about that?  Everything has a sweet spot and
> I am not asking you to make 10 million documents be encompassed by the sweet
> spot, but clearly if you never intend for CouchDB to handle that much data
> then I need to go elsewhere.
>
>>> Since CouchDB makes no performance or size guarantees,
>
> How about publishing some?  Not guarantees but rather some expectations.
> For example if someone has 1GB of JSON data in 1 million documents what
> would be an expectation of size? The bugs can then be about substantial
> divergences from that.
>
>> Unless you have specific bug to fix, or enhancement to make, don't use JIRA.
>
> The issues you closed listed specific enhancements (pipelining, multiple
> instances, different file format etc).  I do acknowledge that I didn't
> supply code but I can't do everything :-)  All my personal projects are open
> source - it isn't like I am trying to take and never give.
>
>>> Also, if you want something without view performance problems but similar to
CouchDB, you should look at MongoDB.
>
> I did research the alternatives I could find.  CouchDB is the only solution
> that was designed for replication (and hence offline working, occasional
> disconnection, any topology for replication etc).  CouchDB is also the only
> one that allows for indices/views on data that is "calculated" rather than
> just extracting a particular value statically from the docs.  (That can be
> worked around by calculated values and shoving them into the docs but is
> less elegant.)
>
> Other than that MongoDB seemed to be the nicest.  But I really want CouchDB
> to take over the world.  The concepts are right.  The replication point of
> view is right etc.  It not handling millions of documents in a reasonable
> amount of space and time is not right IMHO but I still don't know what the
> project opinion is.

I think it's obvious that we want to handle data sets that large. I
know your data is on the large side and CouchDB doesn't auto-cluster
for you, but you aren't that large as far as many of the installations
I've seen running smoothly (they all implement some form of
partitioning).

When the Cloudant clustering is ready for trunk, that will make a
significant difference (in that it will make it easier for users to
tune large Couches than it is now.)

Even "oversharding" with the Cloudant cluster or CouchDB-Lounge is a
good way to use CPUs and lower the impact of compaction / view
compaction.

I'm glad you feel the way you do about the project. If you help us, we
can handle your use case. It's definitely on the roadmap.

That doesn't mean we're going to replace the view engine (it's so easy
to make plugins anyway) but we're definitely eyeing view performance
as a bottleneck (and have a more than 5x speedup in the last 6
months). There's more low-hanging performance fruit, and Cloudant's
clustering on the way.

Looking forward to seeing those benchmarks!

Chris


>
> Roger
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAktOjaUACgkQmOOfHg372QQBcACgmi/Rn5jtsiFvJZy0ksC6F6BU
> 8fMAn225w2RxsTKY8M/cg/29YSxc6mek
> =4qrM
> -----END PGP SIGNATURE-----
>
>



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Mime
View raw message