I'm running a trunk build of couchdb, thought that counted as 0.11+ will try with c-l 0-4 maint, thanks. --Søren On Tuesday 08 December 2009 00:11:47 Sebastian Cohnen wrote: > hey søren, > > you want to have a look at c-l 0.4: > http://github.com/rnewson/couchdb-lucene/tree/0.4-maint (see the readme > there for [update_notification]). c-l 0.5 (current master, unstable/dev) > uses the _changes-API to "see" changes to documents. > > > Sebastian > > On 07.12.2009, at 23:59, Søren Hilmer wrote: > > Hi > > > > I took the latest couchdb-lucene for a spin, but I believe something is > > missing in how this is to be configured. > > Shouldn't there be a setting in the [update_notification] section? > > I cannot see how couchdb can notify the couchdb-lucene server otherwise, > > or am I missing something? > > > > Great improvement that it is now a standalone server and not spawned by > > couch! > > > > --Søren > > -- > > Søren Hilmer, M.Sc., M.Crypt. > > wideTrail Phone: +45 25481225 > > Pilevænget 41 Email: sh@widetrail.dk > > DK-8961 Allingåbro Web: www.widetrail.dk -- Søren Hilmer, M.Sc., M.Crypt. wideTrail Phone: +45 25481225 Pilevænget 41 Email: sh@widetrail.dk DK-8961 Allingåbro Web: www.widetrail.dk
hey søren, you want to have a look at c-l 0.4: http://github.com/rnewson/couchdb-lucene/tree/0.4-maint (see the readme there for [update_notification]). c-l 0.5 (current master, unstable/dev) uses the _changes-API to "see" changes to documents. Sebastian On 07.12.2009, at 23:59, Søren Hilmer wrote: > Hi > > I took the latest couchdb-lucene for a spin, but I believe something is > missing in how this is to be configured. > Shouldn't there be a setting in the [update_notification] section? > I cannot see how couchdb can notify the couchdb-lucene server otherwise, or am > I missing something? > > Great improvement that it is now a standalone server and not spawned by couch! > > --Søren > -- > Søren Hilmer, M.Sc., M.Crypt. > wideTrail Phone: +45 25481225 > Pilevænget 41 Email: sh@widetrail.dk > DK-8961 Allingåbro Web: www.widetrail.dk
Hi I took the latest couchdb-lucene for a spin, but I believe something is missing in how this is to be configured. Shouldn't there be a setting in the [update_notification] section? I cannot see how couchdb can notify the couchdb-lucene server otherwise, or am I missing something? Great improvement that it is now a standalone server and not spawned by couch! --Søren -- Søren Hilmer, M.Sc., M.Crypt. wideTrail Phone: +45 25481225 Pilevænget 41 Email: sh@widetrail.dk DK-8961 Allingåbro Web: www.widetrail.dk
On Mon, Dec 7, 2009 at 2:34 AM, Brian Candler <B.Candler@pobox.com> wrote: > I am thinking about storing some derived data which is associated with key > ranges of a view. (Example: an image which provides a graphical summary of a > key range). > > I would like to determine when it's time to regenerate an image, that is, > when the underlying view has changed within that range. > > One thought I had was if I could make a reduce function which was some sort > of checksum of the key/value pairs. Then I could just do a reduce query > across the key range, and see if the reduce value has changed. It would be > like an etag for the range. > > Unfortunately, I can't just do something simple like an md5sum across the > range, because couchdb implements a tree of reduces and re-reduces, and may > decide to restructure this tree. I'd like a checksum which is invariant > across all possible reduce trees for the same data. > > Something simple would be to XOR all the keys and values together, but > sometimes this would not detect changes which happen to XOR to the same > data. > > Perhaps I should md5 each (key,value) pair, and then XOR all those together > in the reduce function. > > Since my docs have updated timestamps, maybe I should just take the max() of > the updated timestamp for each doc, together with a count of the docs (so as > to be able to detect deletions) This is a great question. If there's a generic way to do this, and it is cheap enough, it could be generalized to handle view etags. Your row count + max timestamp trick seems sensible to me, but obviously is not generalizable. Presumably you could avoid hashing the keys and values by leaning on the document._rev. However, that just pushes the problem back a step. What we need for a general solution is a commutative and associated checksum function, which would be a funny beast indeed. Chris > > I just wondered if anyone had already made an elegant solution for this? Or > some completely different way of determining whether a view has changed > between a given startkey and endkey? > > Thanks, > > Brian. > -- Chris Anderson http://jchrisa.net http://couch.io
I am thinking about storing some derived data which is associated with key ranges of a view. (Example: an image which provides a graphical summary of a key range). I would like to determine when it's time to regenerate an image, that is, when the underlying view has changed within that range. One thought I had was if I could make a reduce function which was some sort of checksum of the key/value pairs. Then I could just do a reduce query across the key range, and see if the reduce value has changed. It would be like an etag for the range. Unfortunately, I can't just do something simple like an md5sum across the range, because couchdb implements a tree of reduces and re-reduces, and may decide to restructure this tree. I'd like a checksum which is invariant across all possible reduce trees for the same data. Something simple would be to XOR all the keys and values together, but sometimes this would not detect changes which happen to XOR to the same data. Perhaps I should md5 each (key,value) pair, and then XOR all those together in the reduce function. Since my docs have updated timestamps, maybe I should just take the max() of the updated timestamp for each doc, together with a count of the docs (so as to be able to detect deletions) I just wondered if anyone had already made an elegant solution for this? Or some completely different way of determining whether a view has changed between a given startkey and endkey? Thanks, Brian.
On Sun, Dec 6, 2009 at 2:48 PM, Matthew M. Gamble <mgamble@mgamble.ca> wrote: > Thanks Roger! > > If I go down the "one user, one DB" path is there a limit to the number of databases I can create? Any performance issues with having 100k+ unique databases? Couch has been tested and works fine with millions of databases. If you need to spread them across directories, create them with a "/" in the name (properly escaped in the url, of course) and couch will create subdirectories to hold the files. > > > ------Original Message------ > From: Roger Binns > To: user@couchdb.apache.org > ReplyTo: user@couchdb.apache.org > Subject: Re: Design Question > Sent: Dec 6, 2009 5:25 PM > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Matthew Gamble wrote: >> I'm sorry if a question like this has been asked >> before, but I couldn't find anything in the archives that was quite the same >> as what I'm trying to do. > > There indeed was a similar discussion. One of the final conclusions was > that it would be best to have each user plus their data stored in a separate > DB. The reason for this is that it makes the security rules easier to > define and also makes more sense for replication. (Presumably you want to > eventually allow the users to run a personal CouchDB with the same database.) > >> attachment but that made it very hard to search the data as it required the > > May be helpful http://wiki.apache.org/couchdb/Full_text_search > >> Or is it best to put each user as their own database and store the address >> books / VCards as documents? > > That would be most natural. In theory you can write an update handler that > parses the vCard into fields and makes them keys of the doc. If you do this > then the full text search should work really easily. > > Roger > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iEYEARECAAYFAkscL2oACgkQmOOfHg372QSAwACdGdOXBMw9W7a5jLAeXPHGGz27 > Za4An3KylQT6Z788a09Nr1MOH2jiXeFU > =AJpY > -----END PGP SIGNATURE----- > > > Sent from my BlackBerry device on the Rogers Wireless Network -- Chris Anderson http://jchrisa.net http://couch.io
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Matthew M. Gamble wrote: > If I go down the "one user, one DB" path is there a limit to the number of databases I can create? Any performance issues with having 100k+ unique databases? The database names are constrained (so they are unambiguous on the filesystem - eg only lowercase ascii but you could probably base64 encode the names if needed). You would have problems if the underlying filesystem is crummy - each database corresponds to a file so the filesystem should handle that number of files. Consequently using a NTFS filesystem while running a virus scanner would be a bad idea :-) Modern Linux filesystems are fine. You will likely also have to increase some tunables in the script starting CouchDB such as the number of open file handles. However only you can decide what you consider acceptable performance and what is representative data. You should be able to construct an appropriate benchmark using your favourite scripting language with CouchDB bindings in a few hours (or less!). Do please report back the numbers you get. http://books.couchdb.org/relax/reference/high-performance This will help for ulimit setting: https://issues.apache.org/jira/browse/COUCHDB-590 Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkscRFQACgkQmOOfHg372QTn3QCdGcmVNf3Mq2H+rIkFrZvxLyaK fTsAoIr6UyXb+l4jrkyzzyfPHEaUry/C =BCAe -----END PGP SIGNATURE-----
Thanks Roger! If I go down the "one user, one DB" path is there a limit to the number of databases I can create? Any performance issues with having 100k+ unique databases? ------Original Message------ From: Roger Binns To: user@couchdb.apache.org ReplyTo: user@couchdb.apache.org Subject: Re: Design Question Sent: Dec 6, 2009 5:25 PM -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Matthew Gamble wrote: > I'm sorry if a question like this has been asked > before, but I couldn't find anything in the archives that was quite the same > as what I'm trying to do. There indeed was a similar discussion. One of the final conclusions was that it would be best to have each user plus their data stored in a separate DB. The reason for this is that it makes the security rules easier to define and also makes more sense for replication. (Presumably you want to eventually allow the users to run a personal CouchDB with the same database.) > attachment but that made it very hard to search the data as it required the May be helpful http://wiki.apache.org/couchdb/Full_text_search > Or is it best to put each user as their own database and store the address > books / VCards as documents? That would be most natural. In theory you can write an update handler that parses the vCard into fields and makes them keys of the doc. If you do this then the full text search should work really easily. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkscL2oACgkQmOOfHg372QSAwACdGdOXBMw9W7a5jLAeXPHGGz27 Za4An3KylQT6Z788a09Nr1MOH2jiXeFU =AJpY -----END PGP SIGNATURE----- Sent from my BlackBerry device on the Rogers Wireless Network
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Matthew Gamble wrote: > I'm sorry if a question like this has been asked > before, but I couldn't find anything in the archives that was quite the same > as what I'm trying to do. There indeed was a similar discussion. One of the final conclusions was that it would be best to have each user plus their data stored in a separate DB. The reason for this is that it makes the security rules easier to define and also makes more sense for replication. (Presumably you want to eventually allow the users to run a personal CouchDB with the same database.) > attachment but that made it very hard to search the data as it required the May be helpful http://wiki.apache.org/couchdb/Full_text_search > Or is it best to put each user as their own database and store the address > books / VCards as documents? That would be most natural. In theory you can write an update handler that parses the vCard into fields and makes them keys of the doc. If you do this then the full text search should work really easily. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkscL2oACgkQmOOfHg372QSAwACdGdOXBMw9W7a5jLAeXPHGGz27 Za4An3KylQT6Z788a09Nr1MOH2jiXeFU =AJpY -----END PGP SIGNATURE-----
I'm been trying to wrap my head around how to layout my application in couchDB for the past few days and I can't seem to figure out which is the best way of doing it. I'm sorry if a question like this has been asked before, but I couldn't find anything in the archives that was quite the same as what I'm trying to do. In a nutshell, I'd like to store addresses (VCard like data) for users. My first thought was a single document per-user with each VCard as an attachment but that made it very hard to search the data as it required the client side to download and parse everything for a search. My next idea was a document for each VCard and a key (user_id, for example) to denote which user the card belongs to. This is quick to access with a view, and might be the right way to do this? Or is it best to put each user as their own database and store the address books / VCards as documents? To give you an idea of the scale, I'm looking at 100-200k unique users with (up to) 5000 VCards each. I really like the idea of using CouchDB to store the address book / vcard data as it lets me only enter the data that's actual used. Thanks in advance for any advice - like I said, I'm still trying to wrap my head around the way CouchDB works.
Hi. While my couchdb application is very quiet most of the time, sometimes there is a burst of activity: tens or hundreds of documents are created and updated in few seconds. There are many users logged on the system, but each burst is relevant only for one of them. When this bursts happen, I want to suggest the user's browser to reload his views. Clients are connected to a queue via cometd, messages can be pushed to them. An approach may be to write a _changes listener that sends a message to a client when it thinks best, but there's a catch: to know which client to notify, the proces would either have to download the documents or I'd have to create one listener for each user, which would mean one http connection per user. Not that http connections are a bad thing, but I'm still shy with too many of them around... Another approach would be to execute a view that counts the documents for each user and sends a message to the ones for which the count has changed since last execution. I'm sure I'm not seeing better solutions. Suggestions anyone? Thanks! Notes: Connecting each client to the _changes API is not welcome because each client is already using one connection with a comet server. Also, there are different classes of documents associated with different views, and I'd end up with multiple connections to differently filtered _changes. Of course the processes that create and update the documents know which user owns the document. I could have them send a message when due, but I was hoping to keep generators and consumers as decoupled as possible, with couchdb in the middle. -- :Matteo Caprari matteo.caprari@gmail.com
On Sun, Dec 6, 2009 at 1:25 AM, Kamyar Navidan <kamyar.n@gmail.com> wrote: > Thank you for your response. So in general [x1, x2, ... , xn-1] > gets precedence over [x1, x2, ... , xn-1, xn] where x1...xn-1 is identical. > Am I getting this right? > Yep. Shorter array sorts first if it's a prefix of a longer one.
Thank you for your response. So in general [x1, x2, ... , xn-1]
gets precedence over [x1, x2, ... , xn-1, xn] where x1...xn-1 is identical.
Am I getting this right?
On Sun, Dec 6, 2009 at 9:01 AM, Chris Anderson <jchris@apache.org> wrote:
> On Sat, Dec 5, 2009 at 8:55 PM, Kamyar Navidan <kamyar.n@gmail.com> wrote:
> > Reading through Couchdb: The Definitive Guide I came across this section
> in
> > chapter 06 :
> >
> > In the blog app, we use group_level reduce queries to compute the count
> of
> >> comments both on a per-post and total basis, achieved by querying the
> same
> >> view index with different methods. With some array keys, and assuming
> each
> >> key has the value 1:
> >
> >
> >> ["a","b","c"]
> >
> > ["a","b","e"]
> >
> > ["a","c","m"]
> >
> > ["b","a","c"]
> >
> > ["b","a","g"]
> >
> >
> >> The reduce view:
> >
> >
> >> function(keys, values, rereduce) {
> >
> > return sum(values)
> >
> > }
> >
> >
> >> returns the total number of rows between the start and end key. So with
> >> startkey=["a","b"]&endkey=["b"] (which includes the first 3 of the above
> >> keys) the result would equal 3.
> >
> >
> > Earlier in chapter it is stated that:
> >
> > The startkey and and endkey parameters specify an inclusive range on
> which
> >> we can search.
> >
>
> There is an inclusive_endkey=true http query parameter you can use.
>
> >
> > So how come the fourth and fifth rows that satisfy endkey=["b"] condition
> > are not included in results? I guess I am missing something here. Can
> > somebody please clarify this for me.
> >
>
> In this case, only the keys:
>
> ["a","b","c"]
> ["a","b","e"]
> ["a","c","m"]
>
> are included in the range.
>
> the key ["b","a","c"] comes after ["b"] so it is not included. ["b"]
> itself would match.
>
> Chris
>
>
>
>
>
> --
> Chris Anderson
> http://jchrisa.net
> http://couch.io
>
On Sat, Dec 5, 2009 at 8:55 PM, Kamyar Navidan <kamyar.n@gmail.com> wrote:
> Reading through Couchdb: The Definitive Guide I came across this section in
> chapter 06 :
>
> In the blog app, we use group_level reduce queries to compute the count of
>> comments both on a per-post and total basis, achieved by querying the same
>> view index with different methods. With some array keys, and assuming each
>> key has the value 1:
>
>
>> ["a","b","c"]
>
> ["a","b","e"]
>
> ["a","c","m"]
>
> ["b","a","c"]
>
> ["b","a","g"]
>
>
>> The reduce view:
>
>
>> function(keys, values, rereduce) {
>
> return sum(values)
>
> }
>
>
>> returns the total number of rows between the start and end key. So with
>> startkey=["a","b"]&endkey=["b"] (which includes the first 3 of the above
>> keys) the result would equal 3.
>
>
> Earlier in chapter it is stated that:
>
> The startkey and and endkey parameters specify an inclusive range on which
>> we can search.
>
There is an inclusive_endkey=true http query parameter you can use.
>
> So how come the fourth and fifth rows that satisfy endkey=["b"] condition
> are not included in results? I guess I am missing something here. Can
> somebody please clarify this for me.
>
In this case, only the keys:
["a","b","c"]
["a","b","e"]
["a","c","m"]
are included in the range.
the key ["b","a","c"] comes after ["b"] so it is not included. ["b"]
itself would match.
Chris
--
Chris Anderson
http://jchrisa.net
http://couch.io
Reading through Couchdb: The Definitive Guide I came across this section in
chapter 06 :
In the blog app, we use group_level reduce queries to compute the count of
> comments both on a per-post and total basis, achieved by querying the same
> view index with different methods. With some array keys, and assuming each
> key has the value 1:
> ["a","b","c"]
["a","b","e"]
["a","c","m"]
["b","a","c"]
["b","a","g"]
> The reduce view:
> function(keys, values, rereduce) {
return sum(values)
}
> returns the total number of rows between the start and end key. So with
> startkey=["a","b"]&endkey=["b"] (which includes the first 3 of the above
> keys) the result would equal 3.
Earlier in chapter it is stated that:
The startkey and and endkey parameters specify an inclusive range on which
> we can search.
So how come the fourth and fifth rows that satisfy endkey=["b"] condition
are not included in results? I guess I am missing something here. Can
somebody please clarify this for me.
May be CouchDBX didn't start properly the first time or something, as when I
closed it down and restarted it, its top two frames were displayed, when
previously they were blank (previously only the messages in the bottom pane
were displayed, there was no CouchDB logo, etc).
After that, everything worked fine (especially once I opened my eyes and saw
it was _all_dbs and not _all_dbs_).
Thanks for your help.
On Sun, Dec 6, 2009 at 5:50 AM, Chris Anderson <jchris@apache.org> wrote:
> On Sat, Dec 5, 2009 at 2:48 AM, Steve Anderson <s.anderson.au@gmail.com>
> wrote:
> > I am working my way through the online version of the O'Reilly book
> > "CouchDB: The Definitive Guide" by Anderson, Lehnardt and Slater.
> >
> > I am on Chapter 3 "Getting Started". I've downloaded and started
> CouchDBX,
> > but am getting errors, or at least different responses, when I run the
> first
> > few commands in chapter 3.
> >
> > For example:
> >
> > silverlight:~ stevea$ curl http://127.0.0.1:5984/
> > {"couchdb":"Welcome","version":"0.10.0"}
> >
> > silverlight:~ stevea$ curl -X GET http://127.0.0.1:5984/_all_dbs_
> > {"error":"error","reason":"illegal_database_name"}
>
> this should be:
>
> curl -X GET http://127.0.0.1:5984/_all_dbs
>
> hopefully that will help.
>
>
> >
> > silverlight:~ stevea$ curl -X PUT http://127.0.0.1:5984/baseball
> > curl: (52) Empty reply from server
> >
>
> this should respond with {"ok":true}
>
> if it isn't than maybe there's a problem with your CouchDB. Have you
> tried running the test suite from Firefox? It is available at
> http://127.0.0.0:5984/_utils/couch_tests.html
>
> > Any ideas?
> >
> > Thanks
> >
>
>
>
> --
> Chris Anderson
> http://jchrisa.net
> http://couch.io
>
--
www.submission.org
On Sat, Dec 5, 2009 at 2:55 AM, Hatem Ben Yacoub <hatemben@gmail.com> wrote: > Someone can update the roadmap, or update me about filtered > replication, It's on the 0.10 roadmap, mentioned briefly in the > O'Reilly book, and we are at 0.10.1 and I can't find more details > about it. > The current status is that there is an API for filtering the _changes feed. The replicator in 0.10 consumes the changes feed, so the next step is getting the replicator to use the filter API. There is work in progress on this, so it should be fully ready to go in 0.11. Chris -- Chris Anderson http://jchrisa.net http://couch.io
On Sat, Dec 5, 2009 at 2:48 AM, Steve Anderson <s.anderson.au@gmail.com> wrote:
> I am working my way through the online version of the O'Reilly book
> "CouchDB: The Definitive Guide" by Anderson, Lehnardt and Slater.
>
> I am on Chapter 3 "Getting Started". I've downloaded and started CouchDBX,
> but am getting errors, or at least different responses, when I run the first
> few commands in chapter 3.
>
> For example:
>
> silverlight:~ stevea$ curl http://127.0.0.1:5984/
> {"couchdb":"Welcome","version":"0.10.0"}
>
> silverlight:~ stevea$ curl -X GET http://127.0.0.1:5984/_all_dbs_
> {"error":"error","reason":"illegal_database_name"}
this should be:
curl -X GET http://127.0.0.1:5984/_all_dbs
hopefully that will help.
>
> silverlight:~ stevea$ curl -X PUT http://127.0.0.1:5984/baseball
> curl: (52) Empty reply from server
>
this should respond with {"ok":true}
if it isn't than maybe there's a problem with your CouchDB. Have you
tried running the test suite from Firefox? It is available at
http://127.0.0.0:5984/_utils/couch_tests.html
> Any ideas?
>
> Thanks
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Someone can update the roadmap, or update me about filtered replication, It's on the 0.10 roadmap, mentioned briefly in the O'Reilly book, and we are at 0.10.1 and I can't find more details about it. Thanks
I am working my way through the online version of the O'Reilly book
"CouchDB: The Definitive Guide" by Anderson, Lehnardt and Slater.
I am on Chapter 3 "Getting Started". I've downloaded and started CouchDBX,
but am getting errors, or at least different responses, when I run the first
few commands in chapter 3.
For example:
silverlight:~ stevea$ curl http://127.0.0.1:5984/
{"couchdb":"Welcome","version":"0.10.0"}
silverlight:~ stevea$ curl -X GET http://127.0.0.1:5984/_all_dbs_
{"error":"error","reason":"illegal_database_name"}
silverlight:~ stevea$ curl -X PUT http://127.0.0.1:5984/baseball
curl: (52) Empty reply from server
Any ideas?
Thanks
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Vlad GURDIGA wrote: > Did anyone try E4X? I mention it in the wiki page in the first paragraph as useful for XML based formats. (The book also refers to using it for generating Atom). http://wiki.apache.org/couchdb/Generating%20HTML%20from%20Javascript%20shows%20and%20lists I've added Google's Closure to the list as it has an interesting approach of ahead of time compilation to Javascript code. (The goal isn't to list every possible engine but rather a small handful that would be useful to the kind of people consulting the wiki page for recommendations, and that are known to work right now.) Thanks to Nathan I found the attachment to EJS is because it also includes helper functions that generate the HTML wrapping tags for you. I've mentioned that in the "Best Practises" section. (Its code for doing this is in a separate file confusing named views.js and can be used with the other engines that do not do HTML escaping, with minor tweaking.) I've tried to get EJS to work. The exact same code and template applied on a fresh Fedora 12 install results in a single space being returned (as opposed to an regex error on Ubuntu 9.10). Changing the template didn't change what was returned. An older EJS version had the same regex error problem on Ubuntu. I've yet to hear from anyone what the bugs in Resig's micro-templating actually are, other than the obvious one that it isn't supported and has no way of reporting bugs! Finally I think it is bad idea for the CouchDB book to be using Resig's micro-templating. (My being uncomfortable with it is why I started this thread and wiki page.) In addition to the claims of bugs and its lack of support, not HTML escaping by default sets a bad precedent. Developers new to CouchDB are likely to follow the book's example and then end up with HTML output that is prone to XSS attacks or just plain invalid. As CouchDB gets more popular, I'd hate for there to be a perception that apps using it generally tend to be broken or are good places to hunt for XSS attacks. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksaHLcACgkQmOOfHg372QSmuACg232OJt3gbnjQCj+UbpX5/D5J R7IAn3uzNcsBhkcp1aT29v92CDQ5hOmC =86F+ -----END PGP SIGNATURE-----
Thanks everybody for all the feedback. Having so many PNGs is, in itself, a flaw in the current system so there's a good chance this may just go away as we move forward. It's good to know about these potential problems ahead of time though. Dave On Fri, Dec 4, 2009 at 4:34 AM, Paul Davis <paul.joseph.davis@gmail.com> wrote: >>> One of the CouchDB config files defines the maximum document size as 4GB. >>> Does that include attachments? >>> >> >> No, but it may include attachment stubs (although I'm guessing it doesn't). >> > > That's the mochiweb request body size number isn't it? So it would be > "maximum size of any request to couchdb." > > Paul Davis >
Did anyone try E4X? I'm not sure it counts here, but it is kinda working* for me so far... * http://github.com/gurdiga/cozy On Thu, Dec 3, 2009 at 11:13 PM, Roger Binns <rogerb@rogerbinns.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Can anyone recommend Javascript templating engines that can be used within > the CouchDB server side for implementing shows and lists? Â The Sofa weblog > app (also described in the book) uses a nice small one from John Resig. > > Book section: > Â http://books.couchdb.org/relax/design-documents/shows#Using%20Templates > > Engine: > Â http://ejohn.org/blog/javascript-micro-templating > > I am looking for something a little more complex such as handling > conditionals and automatically HTML escaping all values unless told > otherwise for specific ones (ie make XSS harder to happen accidentally). > > I can find several engines that will only run in the browser, or depend on a > DOM being present, or load the templates from a URL, or require a complete > HTML document to be present in the template (lists build row by row). Â These > are all deal breakers. Some are many kilobytes of code and require other > libraries such as jQuery. Â Will this much code be a problem for show/list usage? > > My plan is for the output to be somewhat plain but functional - ie usable on > a mobile phone. Â Then client side Javascript runs and adds functionality, > increases information detail, improves formatting etc. Â An example of this > in sofa is that the CouchDB side formatting provides dates in output which > client side Javascript then reformats and the client side also knows what > the local timezone is. Â Similarly client side javascript can add in icons, > previews of next items, tooltips etc. Â This all points to a simpler CouchDB > server side templating engine. > > I'll summarise responses in the wiki. > > Roger > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iEYEARECAAYFAksYKeIACgkQmOOfHg372QTh2ACeK4yirwjIqvlEaXx+ki6tegLM > /PcAn0SVnuFHFhkl0TWwUDYoWIXH6oAy > =jc5m > -----END PGP SIGNATURE----- >
Note that Rails has also changed to escaping by default: http://weblog.rubyonrails.org/2009/10/12/what-s-new-in-edge-rails On Fri, Dec 4, 2009 at 4:02 PM, Roger Binns <rogerb@rogerbinns.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > >> http://github.com/yssk22/crayon >> >> This library enables you to write as followings: >> >> <%= h(var) %> >> <%= text_field(doc, "path-to-field") %> > > embeddedjs includes a views.js file that adds something similar. > > My concern about escaping is over simple values. For example if someone > specifies something like this in a template. > > <%= title %> > > If the value is not HTML escaped by default then it is a potential source of > XSS attacks. In the vast majority of cases values should be HTML escaped. > A separate mechanism can then be used to stop escaping (for example Mustache > uses different tags and the Python Genshi templating system wraps the value > in a different class). > > Roger > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iEYEARECAAYFAksZhv4ACgkQmOOfHg372QSUVgCeNqUJn02nvDkmDElF0z6dOwix > kaUAoI1C3us4P07CuBAy//OLa/pmI4pt > =72DE > -----END PGP SIGNATURE----- >
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > http://github.com/yssk22/crayon > > This library enables you to write as followings: > > <%= h(var) %> > <%= text_field(doc, "path-to-field") %> embeddedjs includes a views.js file that adds something similar. My concern about escaping is over simple values. For example if someone specifies something like this in a template. <%= title %> If the value is not HTML escaped by default then it is a potential source of XSS attacks. In the vast majority of cases values should be HTML escaped. A separate mechanism can then be used to stop escaping (for example Mustache uses different tags and the Python Genshi templating system wraps the value in a different class). Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksZhv4ACgkQmOOfHg372QSUVgCeNqUJn02nvDkmDElF0z6dOwix kaUAoI1C3us4P07CuBAy//OLa/pmI4pt =72DE -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Benoit Chesneau wrote: > Well json resig code is buggy and give less features than embededjs. What are the bugs in the Resig implementation? (I'll mention them in the wiki.) Your comment also highlights the biggest problem with it - there is no bug tracker or development/test/release process behind it, as it is more a proof of concept. > Anyway if you don't want logic in templates I'm averse to having Javascript in the templates. You can see the evolution of templating in other scripting languages. > maybe underscore Underscore is in the same family as Resig and embeddedjs - Javascript code in the markup. > though I really don't like having things starting > with a "_" which is against some common usages. I guess we have JQuery to blame for popularising the punctuation character approach > Also have a look on closure which may be interesting too. - From reading the docs it looks like it should work. Having a compilation to Javascript is nice for debugging since you'll find out about syntax errors at compile time rather than run time. I'll add this one to the wiki once I get time to install Java etc. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksZhF8ACgkQmOOfHg372QT0cACfSXeQNvEaEUWAw3thyZbDT+ur ZGgAnjpYiIxOaEVlW6RJ04bwpQvxgRRu =Muw5 -----END PGP SIGNATURE-----
Hi all, someone can update the roadmap, or update me about filtered replication, It's on the 0.10 roadmap, mentioned briefly in the book, and we are at 0.10.1 and I can't find more details. Thanks -- Eng. Hatem Ben Yacoub Senior Systems Architect & IT Consultant http://hatem.tel
yeah, this looks better :)
On 04.12.2009, at 17:08, Zachary Zolton wrote:
> This seems to work for attachments:
>
> function(doc) {
> if (doc['_attachments']) {
> var attachementSize = 0;
> for each (var stub in doc['_attachments']) {
> attachementSize += stub.length;
> }
> emit(null, attachementSize);
> }
> }
>
>
> On Fri, Dec 4, 2009 at 9:46 AM, Sebastian Cohnen
> <sebastiancohnen@googlemail.com> wrote:
>> this still does not include the documents attachments...
>>
>> On 04.12.2009, at 16:33, Zachary Zolton wrote:
>>
>>> You mean something like this map function?
>>>
>>> function(doc) {
>>> function toJson() {
>>> // code...
>>> }
>>>
>>> emit(doc.someAttr, length(toJson(doc)));
>>> }
>>>
>>>
>>> On Fri, Dec 4, 2009 at 3:46 AM, Andreas Pavlogiannis
>>> <paulogiann.couchdb@gmail.com> wrote:
>>>> That might work for a single document, but I 'd rather calculate the size
>>>> inside a map function so that I can then use a reduce to sum up.
>>>>
>>>> 2009/12/4 Sebastian Cohnen <sebastiancohnen@googlemail.com>
>>>>
>>>>> what about doing a HEAD request and look for Content-Length?
>>>>>
>>>>> {~} I curl -I http://localhost:5984/test/DOCUMENT-ID
>>>>> HTTP/1.1 <http://localhost:5984/test/DOCUMENT-ID%0AHTTP/1.1> 200
OK
>>>>> Server: CouchDB/0.11.0b60a6b3e7-git (Erlang OTP/R13B)
>>>>> Etag: "1-1a6c2a80b8615b2399ff5ba66d18534d"
>>>>> Date: Fri, 04 Dec 2009 09:38:34 GMT
>>>>> Content-Type: text/plain;charset=utf-8
>>>>> Content-Length: 169
>>>>> Cache-Control: must-revalidate
>>>>>
>>>>>
>>>>> On 04.12.2009, at 10:17, Andreas Pavlogiannis wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Is there a uniform way to obtain a document's size (attachments'
size
>>>>> included) ?
>>>>>>
>>>>>> Thanks, Andreas
>>>>>
>>>>>
>>>>
>>
>>
This seems to work for attachments:
function(doc) {
if (doc['_attachments']) {
var attachementSize = 0;
for each (var stub in doc['_attachments']) {
attachementSize += stub.length;
}
emit(null, attachementSize);
}
}
On Fri, Dec 4, 2009 at 9:46 AM, Sebastian Cohnen
<sebastiancohnen@googlemail.com> wrote:
> this still does not include the documents attachments...
>
> On 04.12.2009, at 16:33, Zachary Zolton wrote:
>
>> You mean something like this map function?
>>
>> function(doc) {
>> function toJson() {
>> // code...
>> }
>>
>> emit(doc.someAttr, length(toJson(doc)));
>> }
>>
>>
>> On Fri, Dec 4, 2009 at 3:46 AM, Andreas Pavlogiannis
>> <paulogiann.couchdb@gmail.com> wrote:
>>> That might work for a single document, but I 'd rather calculate the size
>>> inside a map function so that I can then use a reduce to sum up.
>>>
>>> 2009/12/4 Sebastian Cohnen <sebastiancohnen@googlemail.com>
>>>
>>>> what about doing a HEAD request and look for Content-Length?
>>>>
>>>> {~} I curl -I http://localhost:5984/test/DOCUMENT-ID
>>>> HTTP/1.1 <http://localhost:5984/test/DOCUMENT-ID%0AHTTP/1.1> 200 OK
>>>> Server: CouchDB/0.11.0b60a6b3e7-git (Erlang OTP/R13B)
>>>> Etag: "1-1a6c2a80b8615b2399ff5ba66d18534d"
>>>> Date: Fri, 04 Dec 2009 09:38:34 GMT
>>>> Content-Type: text/plain;charset=utf-8
>>>> Content-Length: 169
>>>> Cache-Control: must-revalidate
>>>>
>>>>
>>>> On 04.12.2009, at 10:17, Andreas Pavlogiannis wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Is there a uniform way to obtain a document's size (attachments' size
>>>> included) ?
>>>>>
>>>>> Thanks, Andreas
>>>>
>>>>
>>>
>
>
this still does not include the documents attachments...
On 04.12.2009, at 16:33, Zachary Zolton wrote:
> You mean something like this map function?
>
> function(doc) {
> function toJson() {
> // code...
> }
>
> emit(doc.someAttr, length(toJson(doc)));
> }
>
>
> On Fri, Dec 4, 2009 at 3:46 AM, Andreas Pavlogiannis
> <paulogiann.couchdb@gmail.com> wrote:
>> That might work for a single document, but I 'd rather calculate the size
>> inside a map function so that I can then use a reduce to sum up.
>>
>> 2009/12/4 Sebastian Cohnen <sebastiancohnen@googlemail.com>
>>
>>> what about doing a HEAD request and look for Content-Length?
>>>
>>> {~} I curl -I http://localhost:5984/test/DOCUMENT-ID
>>> HTTP/1.1 <http://localhost:5984/test/DOCUMENT-ID%0AHTTP/1.1> 200 OK
>>> Server: CouchDB/0.11.0b60a6b3e7-git (Erlang OTP/R13B)
>>> Etag: "1-1a6c2a80b8615b2399ff5ba66d18534d"
>>> Date: Fri, 04 Dec 2009 09:38:34 GMT
>>> Content-Type: text/plain;charset=utf-8
>>> Content-Length: 169
>>> Cache-Control: must-revalidate
>>>
>>>
>>> On 04.12.2009, at 10:17, Andreas Pavlogiannis wrote:
>>>
>>>> Hello,
>>>>
>>>> Is there a uniform way to obtain a document's size (attachments' size
>>> included) ?
>>>>
>>>> Thanks, Andreas
>>>
>>>
>>
You mean something like this map function?
function(doc) {
function toJson() {
// code...
}
emit(doc.someAttr, length(toJson(doc)));
}
On Fri, Dec 4, 2009 at 3:46 AM, Andreas Pavlogiannis
<paulogiann.couchdb@gmail.com> wrote:
> That might work for a single document, but I 'd rather calculate the size
> inside a map function so that I can then use a reduce to sum up.
>
> 2009/12/4 Sebastian Cohnen <sebastiancohnen@googlemail.com>
>
>> what about doing a HEAD request and look for Content-Length?
>>
>> {~} I curl -I http://localhost:5984/test/DOCUMENT-ID
>> HTTP/1.1 <http://localhost:5984/test/DOCUMENT-ID%0AHTTP/1.1> 200 OK
>> Server: CouchDB/0.11.0b60a6b3e7-git (Erlang OTP/R13B)
>> Etag: "1-1a6c2a80b8615b2399ff5ba66d18534d"
>> Date: Fri, 04 Dec 2009 09:38:34 GMT
>> Content-Type: text/plain;charset=utf-8
>> Content-Length: 169
>> Cache-Control: must-revalidate
>>
>>
>> On 04.12.2009, at 10:17, Andreas Pavlogiannis wrote:
>>
>> > Hello,
>> >
>> > Is there a uniform way to obtain a document's size (attachments' size
>> included) ?
>> >
>> > Thanks, Andreas
>>
>>
>
I second what Benoit said. EJS works great. I use it a lot. It does way more than Resig's simple templating. On Fri, Dec 4, 2009 at 8:38 AM, Yohei Sasaki <yssk22@gmail.com> wrote: > Hi, > > I'm from Rails to Couch. I use embeddedjs, too, because it is similar to > ERB. > > >Also no idea if it does html escaping of values - examining > >code leads me to believe it does not. > > I think so, and I started to develop the library like ActionView. > http://github.com/yssk22/crayon > > This library enables you to write as followings: > > <%= h(var) %> > <%= text_field(doc, "path-to-field") %> > > Thanks. > > -- > Yohei Sasaki > http://www.yssk22.info/ > > On 2009/12/04, at 17:30, Roger Binns wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > Benoit Chesneau wrote: > >> embeddedjs works well here... which crash did you have ? > > > > It is probably versioning or something. The error text is: > > > > JavaScript function raised error: TypeError: regex.exec is not a > function > > > > I'm using CouchDB 0.10.0 on Ubuntu 9.10. Embeddedjs doesn't look too > > promising anyway. It is just Javascript embedded in text, but John > Resig's > > code can do that too (and with way less code). I do like the error > > reporting (line number and details) although that seems unlikely to work > > server side. Also no idea if it does html escaping of values - examining > > code leads me to believe it does not. > > > > Roger > > -----BEGIN PGP SIGNATURE----- > > Version: GnuPG v1.4.9 (GNU/Linux) > > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > > > iEYEARECAAYFAksYyK0ACgkQmOOfHg372QQ82gCgr/9Gol0FLyDDGRcHyJW9qvjV > > zcUAn3kvWoBf9nc+E0TJLmyJUP95LcCo > > =/lQ6 > > -----END PGP SIGNATURE----- > >
Hi, I'm from Rails to Couch. I use embeddedjs, too, because it is similar to ERB. >Also no idea if it does html escaping of values - examining >code leads me to believe it does not. I think so, and I started to develop the library like ActionView. http://github.com/yssk22/crayon This library enables you to write as followings: <%= h(var) %> <%= text_field(doc, "path-to-field") %> Thanks. -- Yohei Sasaki http://www.yssk22.info/ On 2009/12/04, at 17:30, Roger Binns wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Benoit Chesneau wrote: >> embeddedjs works well here... which crash did you have ? > > It is probably versioning or something. The error text is: > > JavaScript function raised error: TypeError: regex.exec is not a function > > I'm using CouchDB 0.10.0 on Ubuntu 9.10. Embeddedjs doesn't look too > promising anyway. It is just Javascript embedded in text, but John Resig's > code can do that too (and with way less code). I do like the error > reporting (line number and details) although that seems unlikely to work > server side. Also no idea if it does html escaping of values - examining > code leads me to believe it does not. > > Roger > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iEYEARECAAYFAksYyK0ACgkQmOOfHg372QQ82gCgr/9Gol0FLyDDGRcHyJW9qvjV > zcUAn3kvWoBf9nc+E0TJLmyJUP95LcCo > =/lQ6 > -----END PGP SIGNATURE-----
On Fri, Dec 4, 2009 at 7:29 AM, ujay68 <ujay68@gmail.com> wrote: > Thanks for answering. I now found this issue which looks like the same topic: > > https://issues.apache.org/jira/browse/COUCHDB-86 > > Any chance of re-opening that one? Not having compaction would make > CouchDB essentially unavailable on Windows, which would be very > unfortunate ideed. (I'm 100% pro Linux, but many customers and project > managers aren't, and convincing them of switching to a NoSQL solution > will be tricky enough even without convincing them to leave Windows at > the same time ... ;-) > > Jay > The proper fix to this is probably going to be intertwined with [1]. Its not a trivial change so its going to require someone to step up and figure out that code. Paul Davis [1] https://issues.apache.org/jira/browse/COUCHDB-67
Thanks for answering. I now found this issue which looks like the same topic: https://issues.apache.org/jira/browse/COUCHDB-86 Any chance of re-opening that one? Not having compaction would make CouchDB essentially unavailable on Windows, which would be very unfortunate ideed. (I'm 100% pro Linux, but many customers and project managers aren't, and convincing them of switching to a NoSQL solution will be tricky enough even without convincing them to leave Windows at the same time ... ;-) Jay
On Fri, Dec 4, 2009 at 6:01 AM, Nils Breunese <N.Breunese@vpro.nl> wrote: > Hello, > > We are currently running CouchDB 0.9.1. We would like to upgrade to version > 0.10 for its JSONP support, but our hosting provider was having problems > getting the testsuite to pass, so we haven't upgraded our production setup > yet. On our test servers 0.10 is working fine and passing all tests. > > Now our hosting provider claims to have found the problem. Translated from > Dutch their report comes down to something like this: > > ---- > CouchDB can't handle having its data directory on NFS. > > It seems CouchDB tries to do this: > 1 mkdir test > 2 start process X with fd under test/foo > 3 do the work > 4 rm test/foo > 5 rmdir test > 6 kill process X > > When test/ is on NFS: > In step 4 an open file is being removed. NFS doesn't really support this and > the workaround is that the client implementation does > mv file .nfsXXXXXXX > > However, the 'rmdir test' in step 5 then cannot succeed anymore, since the > directory is not empty. This eventually causes CouchDB to crash. > > In earlier versions of CouchDB the open/rm/rmdir/kill order may have been > different, for instance open/rm/kill/rmdir. > ---- > > Could this indeed be the problem? Should I file a Jira ticket on this? For > now we might have to ask them to change to local storage? > > Nils Breunese. > I'm not overly familiar with the file handling semantics of NFS other than I know it does weird things like you mention. It does sound like the hosting provider has narrowed it down to the way we do the unix style unlink rename semantics. Its still early so I could be missing an easy answer. It is a bit odd that the upgrade to 0.10.x started causing this though. I don't remember anything in that part of the code changing. There have been a couple proposals on changing some of the file handling semantics to support windows as well as remove the restrictions on database names. Perhaps if someone gets motivated to add that this would go away. HTH, Paul Davis
Hello, We are currently running CouchDB 0.9.1. We would like to upgrade to version 0.10 for its JSONP support, but our hosting provider was having problems getting the testsuite to pass, so we haven't upgraded our production setup yet. On our test servers 0.10 is working fine and passing all tests. Now our hosting provider claims to have found the problem. Translated from Dutch their report comes down to something like this: ---- CouchDB can't handle having its data directory on NFS. It seems CouchDB tries to do this: 1 mkdir test 2 start process X with fd under test/foo 3 do the work 4 rm test/foo 5 rmdir test 6 kill process X When test/ is on NFS: In step 4 an open file is being removed. NFS doesn't really support this and the workaround is that the client implementation does mv file .nfsXXXXXXX However, the 'rmdir test' in step 5 then cannot succeed anymore, since the directory is not empty. This eventually causes CouchDB to crash. In earlier versions of CouchDB the open/rm/rmdir/kill order may have been different, for instance open/rm/kill/rmdir. ---- Could this indeed be the problem? Should I file a Jira ticket on this? For now we might have to ask them to change to local storage? Nils Breunese.
That might work for a single document, but I 'd rather calculate the size
inside a map function so that I can then use a reduce to sum up.
2009/12/4 Sebastian Cohnen <sebastiancohnen@googlemail.com>
> what about doing a HEAD request and look for Content-Length?
>
> {~} I curl -I http://localhost:5984/test/DOCUMENT-ID
> HTTP/1.1 <http://localhost:5984/test/DOCUMENT-ID%0AHTTP/1.1> 200 OK
> Server: CouchDB/0.11.0b60a6b3e7-git (Erlang OTP/R13B)
> Etag: "1-1a6c2a80b8615b2399ff5ba66d18534d"
> Date: Fri, 04 Dec 2009 09:38:34 GMT
> Content-Type: text/plain;charset=utf-8
> Content-Length: 169
> Cache-Control: must-revalidate
>
>
> On 04.12.2009, at 10:17, Andreas Pavlogiannis wrote:
>
> > Hello,
> >
> > Is there a uniform way to obtain a document's size (attachments' size
> included) ?
> >
> > Thanks, Andreas
>
>
what about doing a HEAD request and look for Content-Length?
{~} I curl -I http://localhost:5984/test/DOCUMENT-ID
HTTP/1.1 200 OK
Server: CouchDB/0.11.0b60a6b3e7-git (Erlang OTP/R13B)
Etag: "1-1a6c2a80b8615b2399ff5ba66d18534d"
Date: Fri, 04 Dec 2009 09:38:34 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 169
Cache-Control: must-revalidate
On 04.12.2009, at 10:17, Andreas Pavlogiannis wrote:
> Hello,
>
> Is there a uniform way to obtain a document's size (attachments' size included) ?
>
> Thanks, Andreas
>> One of the CouchDB config files defines the maximum document size as 4GB. >> Does that include attachments? >> > > No, but it may include attachment stubs (although I'm guessing it doesn't). > That's the mochiweb request body size number isn't it? So it would be "maximum size of any request to couchdb." Paul Davis