river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Creswell <dan.cresw...@gmail.com>
Subject Re: Space/outrigger suggestions (remote iterator vs. collection)
Date Wed, 22 Dec 2010 20:27:58 GMT
On 22 December 2010 18:57, <jgrahn@simulexinc.com> wrote:

> Regarding the complications pointed out for returning a remote iterator
> from takeMultiple:
> --
> (1) One can size the batch to make best balance network bandwidth and
> latency.
> --
> That's currently done by combination of server-side takeMultipleLimit and
> client-side maxEntries.   If we use a remote iterator, I assume

I'm not sure what you're referring to when you say "server-side
takeMultipleLimit". To the best of my knowledge there's no such thing
currently spec'd/required.

> we'd include takeMultipleBatchSize and retain the client-side maxEntries.
> So I don't see this as being substantially different.
> --
> (2) One can limit the time a collection of exclusive locks are held under a
> transaction by virtue of the timeout.
> --
> Hmm, why would this not be the case under a remote iterator?   I would
> think that the correct behavior would be to release locks after a timeout
> expires regardless of whether the return type was an iterator or collection.

Mmmm, it's not really about correct behaviour, it's about spec'd behaviour.
When you say correct I'm guessing you mean expected or maybe ideal?

As spec'd a remote iterator doesn't have a timeout. It has a lease,
sometimes but not always. You may of course extend the lease (more
roundtrips, more performance cost). However, leases can be bounded down by
the space implementation so you don't actually get the timeout you'd like
and would have a number of round-trips to extend the lease to get the
timeout you want. So you can use the lease as a "poor man's timeout" but
it's not the sharp tool you'd perhaps prefer.

All of that makes for some ugly code just to try and do a "simple" batch

Batch take's timeout as spec'd doesn't actually apply unless there are no
entry's initially present to take. Thus any sane implementation does as many
takes as it can as soon as it's invoked and returns them, thus freeing the
locks fast. If there are no matches, it'll wait but probably only for a
single entry, again to avoid the lock/search-queue problem contents gives

> --
> (3) Batching in this way allows multiple clients to remove and process
> entrys in a more scalable fashion than with a (unbounded or no entry limit)
> remote iterator.
> --
> Users would still be free to make multiple calls with small values for
> maxEntries if they so chose.   They would also gain the ability to make an
> unbounded request, which is currently lacking, outside of repeated calls.

Unbounded requests build up big long lists of taken entries all locked which
inhibits search times for other clients. This is because they will find
matches, then test the locks and find they're potentially taken and must
continue the search. Unless of course each client is using a unique
template. Further you've got something of a fairness problem. Should an
active matchset be filled as a priority over a single take with an identical
template or not? In the interests of liveness, you might say no however the
penalty for that is your matchset holds locks on entrys for longer and
continues to slow searching.

Note that passing a null transaction as you would for a read contents to
reduce the cost of all this locking doesn't work because one must grab and
maintain a take lock for all entries until the matchset is closed down
regardless of transaction type.

And finally, because of the leasing bits, book-keeping and such (and no
means to control the size of batches as spec'd) using a contents type
construct for small batch takes is very costly.

> --
> (4) [gleaned from text] More bookkeeping is necessary.
> --
> Certainly.   Also, we'd have to work out the precise semantics that the
> iterator operates under and make them clear in the documentation.

I wholeheartedly agree with a qualification:

One of the long debates we had when spec'ing particularly contents was how
tight to constrain things. One big fear was that we might as a result of
very explicit spec rule out a reasonably useful implementation (in
particular clustered spaces). To some extent the ifExists methods when done
properly make a performant cluster difficult, tightening some of these other
specs will potentially make things even harder.

> --
> (5) [gleaned from text] A remote iterator would certainly be less
> performant than a straight batch take.
> --
> This is the biggest concern, I think.   As such, I'd be interested in
> seeing performance runs, to back up the intuition.   Then, at least, we'd
> know precisely what trade-off we're talking about.

See above for my commentary on the effects of long running matchsets
asserting take locks. The current batch take is designed so as to eradicate
that as an issue. Interestingly, spec'ing batch take more tightly than it is
now could drive an implementation to contents style approach with all it's

To some extent, I'd claim there's no need for the tests or intuition:

(1) Batch take is a single round-trip. Contents, maybe not, unless you have
means to control the batch size and make sure that fits the maxMatchSize.
Note that the space implementation may choose to control batch sizes so as
to optimise/balance resource usage. Putting control of that in the hands of
the user denies the implementation such an option. In any case, larger
limits equals more batches equals more round-trips so at best you'll get
linear degradation.

(2) Batch take doesn't hold huge chains of locks for extended periods of
time as currently spec'd. Contents does/would. That affects search times
when clients are concurrently "challenging" for similar entries as is often
found in compute farms and such. Again, as above, that'll be at best linear
degradation based on the length of the lock chains.

What the tests might tell you is how bad some particular implementation
approach degrades but that won't be uniform across all implementations which
limits it's value somewhat IMHO.

I think my deepest reservation is that most users I deal with want all
operations to be as quick as possible and have no problems with the
resultant API asymmetry. They also prefer to not worry about managing batch
sizes in contents calls because it's a detail they don't want to have to
account for. Lastly, bulk take is a popular method and adding additional
boilerplate like leasing to get something that looks like contents probably
won't appeal. Contents tends to be used for admin and debugging (some do use
it to get database-style cursors and such, yikes) which are less common than
the need for batch takes and thus the lease paraphernalia is tolerable.

Maybe the test we should do first is to ask our users what they think about
the APIs, naming and such....maybe you guys already did that and I haven't
read enough of the archives to know in which case, my bad.

> The test would need to cover both small batches and large, both in
> multiples of the batch-size/takeMultipleLimit and for numbers off of those
> multiples, with transactions and without.
> jamesG
> -----Original Message-----
> From: "Dan Creswell" <dan.creswell@gmail.com>
> Sent: Wednesday, December 22, 2010 5:23am
> To: river-dev@incubator.apache.org
> Subject: Re: Space/outrigger suggestions (remote iterator vs. collection)
> Hey,
> So the below means you are indeed following my explanation so to your
> question:
> Yes, you could use a remote iterator style of thing but for take it's quite
> a heavyweight construct especially once you have transactions in the way.
> The core implementation itself is very similar to contents and would have
> for the most part similar performance. However, it'd certainly be less
> performant than a straight batch take.
> More of a concern though is the impact on other clients of the space
> implementation: by virtue of lots of book-keeping, the most exclusive locks
> on entry's and long running transactions that inflict delays on other
> clients leading to poor scaling. Contents by virtue of it's read nature is
> a
> little less painful performance wise and for a lot of applications you'd
> pass no transaction which reduces performance pain further.
> So I'd say that batch take is probably a better tradeoff than a take/remote
> iterator combo because:
> (1) One can size the batch to make best balance network bandwidth and
> latency.
> (2) One can limit the time a collection of exclusive locks are held under a
> transaction by virtue of the timeout.
> (3) Batching in this way allows multiple clients to remove and process
> entrys in a more scalable fashion than with a (unbounded or no entry limit)
> remote iterator.
> In essence one puts the control squarely with the user so's they can get
> what they want albeit at the price of some API asymmetry as you correctly
> point out.
> As an implementer, I could reduce my codebase a little if we did takes with
> a remote iterator but being completely honest, not by enough that I'd
> support a spec change for that reason alone.
> HTH,
> Dan.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message