Now that we've established that the TTL passed into the server-create call is for reaping idle connections and not individual operation timeouts, I want to ask about timing out individual operations.
This helps a lot. I think 600 seconds seems like a fine idle-reap timeout.I need to investigate why some lookups take a second or more. Maybe there's a mutex contention on my end somewhere.Thanks!-JoshOn Thu, Sep 27, 2012 at 2:08 PM, Jeff Trawick <firstname.lastname@example.org> wrote:
On Thu, Sep 27, 2012 at 1:55 PM, Joshua Marantz <email@example.com> wrote:no, subversion
> That one call-site is HTTP_24/src/modules/cache/mod_socache_memcache.c,
> right? That was where I stole my args from.
ttl only affects connections which are not currently used; it does not
> As the TCP/IP layer is a lower level abstraction than bathe apr_memcache
> interface, I'm still not clear on exactly what that means. Does a value of
> 600 mean that a single multiget must complete in 600 microseconds otherwise
> it fails with APR_TIMEUP?
control I/O timeouts
create on demand
> That might explain the behavior I saw.
> I've now jacked that up by x1e6 to 600 seconds and I don't see timeouts,
> but I'm hoping someone can bridge the gap between the socket-level
> explanation and the apr_memcache API call.
> I was assuming that apr_memcache created the TCP/IP connection when I called
> apr_memcache_server_create, and there even 600 seconds seems too short. Is
> the functionality more like it will create connections on-demand and leave
> them running for N microseconds, re-using the connection for multiple
> requests until TTL microseconds have elapsed since creation?
reuse existing idle connections when possible
when performing maintenance on the idle connections, clean up any
which were idle for N microseconds
If a connection is always reused before it is idle for N microseconds,
it will live as long as memcached allows.
It is. The ttl is interpreted by the reslist layer, which won't touch
> If that's the case then I guess that every 10 minutes one of my cache
> lookups may have high latency to re-establish the connection, is that right?
> I've been histogramming this under load and seeing some long tail requests
> with very high latency. My median latency is only 143us which is great.
> My 90%, 95% and 99% are all around 5ms, which is fine as well. But I've got
> a fairly significant number of long-tail lookups that take hundreds of ms or
> even seconds to finish, and one crazy theory is that this is all reconnect
> It would be nice if the TTL were interpreted as a maximum idle time before
> the connection is reaped, rather than stuttering response-time on a very
> active channel.
objects until they're returned to the list.
> This testing is all using a single memcached running on localhost.
> On Thu, Sep 27, 2012 at 11:24 AM, Jeff Trawick <firstname.lastname@example.org> wrote:
>> On Thu, Sep 27, 2012 at 11:15 AM, Joshua Marantz <email@example.com>
>> > On Thu, Sep 27, 2012 at 10:58 AM, Ben Noordhuis <firstname.lastname@example.org>
>> > wrote:
>> >> If dlsym() is called with the special handle NULL, it is interpreted
>> >> as
>> >> a
>> >> reference to the executable or shared object from which the call is
>> >> being
>> >> made. Thus a shared object can reference its own symbols.
>> >> And that's how it works on Linux, Solaris, NetBSD and probably OpenBSD
>> >> as
>> >> well.
>> > Cool, thanks.
>> >> > Do you have a feel for the exact meaning of that TTL parameter to
>> >> > apr_memcache_server_create?
>> >> You mean what units it uses? Microseconds (at least, in 2.4).
>> > Actually what I meant was what that value is used for in the library.
>> > The
>> > phrase "time to live of client connection" confuses me. Does it really
>> > mean
>> > "the maximum number of seconds apr_memcache is willing to wait for a
>> > single
>> > operation? Or does it mean *both*, implying that a fresh TCP/IP
>> > connection
>> > is made for every new operation, but will stay alive for only a certain
>> > number of seconds.
>> TCP/IP connections, once created, will be retained for the specified
>> (ttl) number of seconds. They'll be created when needed.
>> The socket connect timeout is hard-coded to 1 second, and there's no
>> timeout for I/O.
>> > It is a little disturbing from a module-developer perspective to have
>> > the
>> > meaning of that parameter change by a factor of 1M between versions.
>> > Would
>> > it be better to revert the recent change and instead change the doc to
>> > match
>> > the current behavior?
>> The doc was already changed to match the behavior, but I missed that.
>> The caller I know of used the wrong unit, and I'll submit a patch to
>> fix that in the caller, as well as revert my screw-up from yesterday.
>> > -Josh
>> Born in Roswell... married an alien...
Born in Roswell... married an alien...