couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Davis" <paul.joseph.da...@gmail.com>
Subject Re: Closing Files
Date Sun, 23 Nov 2008 08:12:40 GMT
Totally forgot to paste the link after I explicitly pushed code.

http://github.com/davisp/couchdb/tree/update-false

Purge works fine for me.

Some things to note about the code:

I finally made a couch_os_process that can be used for the cases where
we have external processes like the action2 external stuff.

The overall restructuring looks like this:

1. couch_view sets up couch_view_group gen_servers that perform the
actual update.
2. couch_view_group 'owns' a single view_updater process that
basically iterates over enum_docs_since_seq and throws them at
couch_query_server
3. couch_query_server cast's incoming docs to N couch_os_processes
round robin style.
4. Using a closure, the couch_os_process's send the key/value output
back to couch_view_group which buffers and flushes to disk.
5. When couch_view_group recieves as many results as view_updater sent
out, it terminates sending the final group to any process that
registered to get a copy of it.

Also, at the moment I think this implementation might be slower than
the original, but I have a sneaking suspicion that I'm calling fsync
quite a few times that I shouldn't be.

update-false isn't actually in yet, but it'll fit nicely into
couch_view:get_map_view with a call to get the current version of the
Group record stored in the gen_server state.

Also, I realized it would be fairly trivial to add a wait=N to wait at
most N seconds in case people think that's a good idea. I'm on the
fence.

And it's time for bed before my brain explodes.

On Sun, Nov 23, 2008 at 2:51 AM, Chris Anderson <jchris@apache.org> wrote:
> On Sat, Nov 22, 2008 at 11:44 PM, Paul Davis
> <paul.joseph.davis@gmail.com> wrote:
>> I start getting EMFILE errors indicating i've
>> exhausted file handles. I cannot for the life of me piece together how
>> file handles are cleaned up *after* the view has been completely
>> written to the client.
>>
>
> I had the same error, but it would show up when attempting to view
> after a purge (which would never return to the client), so I bet it
> was a different cause. (Mine kept spawning new view updates, over and
> over, due to an off-by-one error when the seq-id decremented)
>
> Is your version of code up on a github branch?
>
> Chris
>
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>

Mime
View raw message