couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cesar Delgado <beett...@ymail.com>
Subject Re: large attachments/huge databases ?`
Date Thu, 13 May 2010 19:52:47 GMT
CK,

My $0.02 on the storage of the videos is not to use CouchDB for that.  Use Couch to store
metadata on the files.  Stuff like filesystem path, server holding the file, date, time-stamp,
video info, etc.  The actual files are better stored in a filesystem somewhere else.  It's
kind of the idea of Facebook's Haystack project (http://www.facebook.com/note.php?note_id=76191543919).
 That's Facebook's system to store photos.

The problem with storing large attachments in any database is you're creating a filesystem
on a filesystem. You ask Couch for a file, Couch has to go the filesystem to get the files,
and then you go back and forth between Couch and the filesystem getting information that then
gets sent back to you.  Removing the middle-man, Couch in this case, should give you better
performance and a smaller DB.  As Couch, "where should I find the file" and the your application
just goes out and gets the file directly.

There used to be a great article explaining why putting large files in a database (the article
was about MySQL) was a bad idea but I can't seen to find it anymore.

Anyway, good luck!

-Cesar  



________________________________
From: c.Kleinhuis <ck@digitalgott.de>
To: user@couchdb.apache.org
Sent: Thu, May 13, 2010 9:35:24 AM
Subject: large attachments/huge databases ?`

i need to convience my project manager ;)

-he read that indexing is significantly higher than e.g. mysql -
my answer was that indexing is not affecting performance because it is a
one time action ....

-another point is general performance of about e.g. 200.000 documents in a single
database ... how is disk usage when maintaining versioning of each document ?
-can the versioning be deactivated or deleted ?!

and finally it is about handling huge movie files ( production 720p files ) which
must be handled somehow, what happens if an upload fails, or should a proxy like
php be used to receive those files, and just store a reference in the couchdb ?

thx in advance
ck
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message