river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gregg Wonderly <gr...@wonderly.org>
Subject Re: ThreadPool calling Thread.setName
Date Sun, 28 Apr 2013 03:50:03 GMT
Yes, the important thing to understand, as shown in the text below, is that "setName" is rewritten
so that the "array" value is assigned to the "visible" class member before the "System.arrayCopy"
is performed.  This can result in different failure modes if setName is racing with getName.
 One of the failure modes is that getName sees "bytes" with a value of "zero" in the "name".
 I am not sure about the code in getName().  Given the fact that this has not already been
a huge issue in the wild, to the point that it was fixed, it's probably not a giant deal.

But, my ranting in the postings below are just an illustration of how frustrated I am with
how we've (well Peter anyway) had to deal with working though racy code which is not simply
showing signs of racy read/write, but instead "random" behaviors that are caused by instruction
reordering that make certain things "visible" before or after the code says they should happen.
 The JLS rules for "ordering" only apply for a single thread.  They only also apply for "concurrency",
when "Happens Before" relationships exist in the "code". Threads executing concurrently in
the same code, are seeing an "ordering" implied at the "moment" of that "happens before" only.

Thread's setName/getName will be fixed by making the class variable "volatile", and "setName"
synchronized, so that two threads can't call setName nearly simultaneously and perhaps execute
the code in this order:

T1: name = new char[str.length()];
T2: name = new char[str.length()];
T1 System.arrayCopy( str.getBytes(), 0, name, 0, str.length() );
T2 System.arrayCopy( str.getBytes(), 0, name, 0, str.length() );

which could create an ArrayIndexOutOfBounds exception, or worse, memory corruption depending
on the real ordering of the underlying machine level instructions.

We all love Java because it's a safe language…well..it used to be...

Gregg Wonderly

On Apr 27, 2013, at 6:19 PM, Peter Firmstone <peter.firmstone@zeus.net.au> wrote:

> Discussion on the concurrency interest list touched briefly on the consequences of different
threads calling Thread.setName(), under these conditions it appears legal for hotspot to potentially
reorder the operations in a way that causes errors.
> 
> We can reason in ThreadPool's case, there is a high enough probability that Thread.setName()
is only ever called from one thread [client code may call Thread.currentThread().setName("Bob's
thread")], so it's unlikely that mutations from separate threads will occur, while it remains
highly likely that reads from separate threads will occur.
> 
> But we are starting to encroach into territory where angels fear to tread, requiring
genius reasoning powers.  For 99% of cases I'd like to stick to simple rules that avoid reasoning
about consequences of data races.  This will make it easier for developers to reason about
the behaviour of River.  Throw in the asynchronous nature of the network and we've already
got a high enough bar for developers to jump over.
> 
> I don't want to be in the position of having to justify replacing race conditions with
properly synchronized code because I can't prove a failure occurs.
> 
> So far, 99% of concurrency fixes haven't affected semantics, but instead ensure predictability
of original intent (based on my interpretation and test results).
> 
> This case falls into that other 1%, so yes I'll be changing it back to allow the data
race, but I believe this is an exceptional case because there is no other way to perform that
operation safely, the likelihood of error remains low and I believe there is a good chance
it'll be fixed in the next version of Java.
> 
> Peter.
> 
> Peter wrote:
> > Based on your reasoning and because all tests are passing reliably without failure
now, it should be safe to uncomment it.
> >
> > Peter.
> >
> >
> > ----- Original message -----
> >> I also don't see the particular concern with toString(). It uses
> >> getName(), so it does all its work with a particular char[]. At the
> >> worst, that char[] will either be an old one, or the current one but
> >> with some out-of-date elements, resulting in an unhelpful value.
> >>
> >> Patricia
> >>
> >>
> >> On 4/27/2013 5:55 AM, Patricia Shanahan wrote:
> >>> I'll read the paper. However, C and C++ tend to have nastier semantics
> >>> for data races than Java, which lacks anything quite as drastic as the C
> >>> definition of "undefined behavior".
> >>>
> >>> My reasoning for the relative safety of this particular race is based on
> >>> one of the more basic rules, atomicity of reference read or write,
> >>> combined with reading the relevant code. Regardless of data races, the
> >>> result of accessing "name" will always be a pointer to a char[]. We have
> >>> a significant chance of the char[] containing useful, helpful data in
> >>> the old version of the code. The probability of it containing anything
> >>> useful is zero in the new version.
> >>>
> >>> Patricia
> >>>
> >>>
> >>> On 4/27/2013 3:51 AM, Peter Firmstone wrote:
> >>>> Ok found the link, the paper's called:
> >>>> Position paper: Nondeterminism is unavoidable, but data races are
> >>>> pure evil
> >>>>
> >>>> http://www.hpl.hp.com/techreports/2012/HPL-2012-218.pdf
> >>>>
> >>>>
> >>>> Wheather Thread's name field is an issue or not depends on the code
that
> >>>> uses it (Object.toString() and Thread.getName() methods do) and what
> >>>> that code does with the information.
> >>>>
> >>>> Outrigger uses it in lifecycle logs, although it doesn't appear to be
> >>>> called on any threads from ThreadPool.  Instead Outrigger calls
> >>>> getName() on threads it references from fields in OutriggerServerImpl.
> >>>> These fields have since been changed to final, so if the name is set
> >>>> during construction and not changed after publication, it's thread safe,
> >>>> in addition these threads aren't started until after
> >>>> OutriggerServerImpl's constructor returns.
> >>>>
> >>>> The concurrency issues I most recently experienced were related to
> >>>> Outrigger, I couldn't determine the cause of these failures, perhaps
I
> >>>> lack sufficient ability to reason deeply about these issues, my fix
for
> >>>> the problem was akin to carpet bombing; fix every concurrency problem
> >>>> and that appears to have paid off.
> >>>>
> >>>> It's very difficult to determine if toString() may be called on
> >>>> ThreadPool threads.  I reasoned it could be and that was sufficient
> >>>> justification to not use setName().
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Peter.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 27/04/2013 1:20 AM, Patricia Shanahan wrote:
> >>>>> I've looked at the source code. The field "name" that is shared
> >>>>> between threads doing getName or setName on a given Thread is a
> >>>>> reference. Writes and reads of references are always atomic.
> >>>>>
> >>>>> The worst that could happen is that the change to the name does
not
> >>>>> propagate to all threads that might display information about the
> >>>>> thread. The proposed fix ensures that worst case outcome happens
all
> >>>>> the time.
> >>>>>
> >>>>> Patricia
> >>>>>
> >>>>> On 4/26/2013 5:44 AM, Greg Trasuk wrote:
> >>>>>> I'm curious, as I don't see any indication in the Javadocs that
> >>>>>> setName() isn't thread safe.  Is there another reference that
calls
> >>>>>> that
> >>>>>> out?  And what would be the failure mode, apart from a mangled
> >>>>>> string in
> >>>>>> a log output?
> >>>>>>
> >>>>>> Personally, if the potential failure mode wasn't onerous, I'd
opt for
> >>>>>> more descriptive logging.  Comprehensibility is everything when
you're
> >>>>>> troubleshooting.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Greg.
> >>>>>>
> >>>>>> On Fri, 2013-04-26 at 05:48, Peter Firmstone wrote:
> >>>>>>> Hope you don't mind, I've removed the call to Thread.setName
in
> >>>>>>> com.sun.jini.ThreadPool
> >>>>>>>
> >>>>>>> As a result threads will be less descriptive, unfortunately
setName
> >>>>>>> isn't thread safe, it's final and cannot be overridden.
> >>>>>>> Thread.getName
> >>>>>>> is only thread safe if a Thread's name isn't changed after
> >>>>>>> publication.
> >>>>>>>
> >>>>>>> ThreadPool was the only instance of Thread.setName in River.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Peter.
> >
> >
> 
> From: concurrency-interest-request@cs.oswego.edu
> Subject: Concurrency-interest Digest, Vol 98, Issue 24
> Date: March 15, 2013 9:39:26 PM CDT
> To: concurrency-interest@cs.oswego.edu
> Reply-To: concurrency-interest@cs.oswego.edu
> 
> 
> Send Concurrency-interest mailing list submissions to
> 	concurrency-interest@cs.oswego.edu
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://cs.oswego.edu/mailman/listinfo/concurrency-interest
> or, via email, send a message with subject or body 'help' to
> 	concurrency-interest-request@cs.oswego.edu
> 
> You can reach the person managing the list at
> 	concurrency-interest-owner@cs.oswego.edu
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Concurrency-interest digest..."
> 
> 
> Today's Topics:
> 
>   1. Re: Thread safety of Thread.getName() (Nathan Reynolds)
>   2. Re: Thread safety of Thread.getName() (Gregg Wonderly)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Fri, 15 Mar 2013 19:22:11 -0700
> From: Nathan Reynolds <nathan.reynolds@oracle.com>
> To: "Boehm, Hans" <hans.boehm@hp.com>
> Cc: Gregg Wonderly <gergg@cox.net>,
> 	"concurrency-interest@cs.oswego.edu"
> 	<concurrency-interest@cs.oswego.edu>
> Subject: Re: [concurrency-interest] Thread safety of Thread.getName()
> Message-ID: <5143D753.9080509@oracle.com>
> Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
> 
> Its called Java Path Finder. http://babelfish.arc.nasa.gov/trac/jpf/
> 
> Nathan Reynolds 
> <http://psr.us.oracle.com/wiki/index.php/User:Nathan_Reynolds> | 
> Architect | 602.333.9091
> Oracle PSR Engineering <http://psr.us.oracle.com/> | Server Technology
> On 3/15/2013 4:49 PM, Boehm, Hans wrote:
>> 
>> It seems to me that the right general solution consists of:
>> 
>> -Annotations for intentional, and extremely carefully considered, data 
>> races.
>> 
>> -A data race detector that can be used on a regular basis, and ignores 
>> the above data races.
>> 
>> -A general understanding that unannotated data races are bugs.
>> 
>> AFAIK, all of this is fundamentally possible.
>> 
>> Hans
>> 
>> *From:*concurrency-interest-bounces@cs.oswego.edu 
>> [mailto:concurrency-interest-bounces@cs.oswego.edu] *On Behalf Of 
>> *Gregg Wonderly
>> *Sent:* Friday, March 15, 2013 2:53 PM
>> *To:* Nathan Reynolds
>> *Cc:* concurrency-interest@cs.oswego.edu
>> *Subject:* Re: [concurrency-interest] Thread safety of Thread.getName()
>> 
>> On Mar 15, 2013, at 4:40 PM, Nathan Reynolds 
>> <nathan.reynolds@oracle.com <mailto:nathan.reynolds@oracle.com>> wrote:
>> 
>> 
>> 
>> System.arraycopy has its own internal checks to make sure the copy 
>> doesn't go beyond the ends of the array.  I highly doubt JIT will be 
>> able to move instructions around in such a way as to cause 
>> System.arraycopy instructions to reload the "this.name" reference 
>> after checking the array bounds.
>> 
>> Okay, but is there actually any guarantee?  Since that still leaves 
>> the "ArrayIndexOutOfBoundsException" as a possibility, I'd suggest 
>> that there needs to be something done to "fix" this API.  It still 
>> seems like we need an annotation that actually turns on "intra-thread" 
>> optimizations.  In the end, inter-thread optimizations should be all 
>> that is done by default, because we do not have any "guarantees" in 
>> the language constructs or the compilers operation, that would allow 
>> it to guarantee that a method is not used inter-thread.
>> 
>> @MultiThreaded or some kind of annotation needs to come into play at 
>> some point.  I just can't imagine that we will be able to continue to 
>> "guess" about this and "hope" and "laugh when it fails" or "fix it 
>> later" for too much longer.  The hardware is in our faces now, and the 
>> tools and Java in particular just doesn't provide a developer with any 
>> indication that there code is not thread-safe.  I think it would be 
>> much, much more productive to let developers make that designation, 
>> literally, so that Javadoc and introspection and all kinds of "real" 
>> observations regarding safe concurrency can actually happen.
>> 
>> Gregg
>> 
>> 
>> 
>> Nathan Reynolds 
>> <http://psr.us.oracle.com/wiki/index.php/User:Nathan_Reynolds> | 
>> Architect | 602.333.9091
>> Oracle PSR Engineering <http://psr.us.oracle.com/> | Server Technology
>> 
>> On 3/15/2013 1:44 PM, Zhong Yu wrote:
>> 
>>    On Fri, Mar 15, 2013 at 3:22 PM, Gregg Wonderly<gregg@cytetech.com>  <mailto:gregg@cytetech.com>
 wrote:
>> 
>>        Nathan's example below is my issue...
>> 
>> 
>> 
>> 
>> 
>>        On 3/15/2013 12:29 AM, David Holmes wrote:
>> 
>>            Gregg,
>> 
>> 
>> 
>>            Gregg Wonderly writes:
>> 
>> 
>> 
>>                On 3/14/2013 6:14 PM, Nathan Reynolds wrote:
>> 
>>                    I've inlined the name.toCharArray() and rearranged a few
>> 
>>                operations that JIT
>> 
>>                    could do.  If another thread reads this.name between its
>> 
>>                creation and the end of
>> 
>>                    arraycopy() then the thread will see a partially filled char[].
>> 
>> 
>> 
>>                           public final void setName(String name) {
>> 
>>                               this.name = new char[name.length()];
>> 
>>                               System.arraycopy(..., 0, result, 0, name.length());
>> 
>> 
>> 
>>        If two different threads call setName concurrently, then if the code was
>> 
>>        transformed as shown above, then it is possible to see
>> 
>> 
>> 
>>        T1:           this.name = new char["some longer string".length()];
>> 
>>        T2:           this.name = new char["a short string".length()];
>> 
>>        T1:           System.arraycopy( "some longer string", 0,
>> 
>>                                 this.name, 0, "some longer string".length());
>> 
>> 
>> 
>>        Which will result in an exception at best and memory corruption at worse
>> 
>>        depending on how this.name is referenced.
>> 
>>    I think you are right, the compiler must not issue a load that did not
>> 
>>    exist in the source code (the compiler *is* aware that some other
>> 
>>    thread may be changing the variable).
>> 
>> 
>> 
>>    The following transformation should be legal
>> 
>> 
>> 
>>            public final void setName(String name) {
>> 
>>                 char[]  tmp = new char[name.length()]
>> 
>>                 this.name = tmp;
>> 
>>                 System.arraycopy(name.value, 0, tmp, 0, name.length());
>> 
>> 
>> 
>>    Zhong Yu
>> 
>> 
>> 
>>        Gregg Wonderly
>> 
>> 
>> 
>>        _______________________________________________
>> 
>>        Concurrency-interest mailing list
>> 
>>        Concurrency-interest@cs.oswego.edu  <mailto:Concurrency-interest@cs.oswego.edu>
>> 
>>        http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>> 
>>    _______________________________________________
>> 
>>    Concurrency-interest mailing list
>> 
>>    Concurrency-interest@cs.oswego.edu  <mailto:Concurrency-interest@cs.oswego.edu>
>> 
>>    http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Concurrency-interest mailing list
>> Concurrency-interest@cs.oswego.edu 
>> <mailto:Concurrency-interest@cs.oswego.edu>
>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>> 
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://cs.oswego.edu/pipermail/concurrency-interest/attachments/20130315/4b795c5c/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 2
> Date: Fri, 15 Mar 2013 21:39:07 -0500
> From: Gregg Wonderly <gregg@cytetech.com>
> To: dholmes@ieee.org
> Cc: Gregg Wonderly <gergg@cox.net>, concurrency-interest@cs.oswego.edu
> Subject: Re: [concurrency-interest] Thread safety of Thread.getName()
> Message-ID: <5143DB4B.5070505@cytetech.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> 
> On 3/15/2013 6:27 PM, David Holmes wrote:
>> Gregg,
>> 
>> The sky is not falling, the end of the world is not nigh. There are
>> thousands of non-thread-safe methods in the JDK libraries, this is just an
>> example. This might be a surprising example, but I expect there was little
>> consideration given to dynamic thread re-naming. You may ineed get an
>> exception because of the possible races - so be it.
> 
> I am not sure of how many lines of code you have written in applications running 
> in the wild David, and I don't want to diminish anything about what you know 
> and/or have experienced.  I can tell you for sure, that I am constantly 
> frustrated by these kinds of APIs with neither specified nor correct (in some 
> potential cases) concurrency and having people involved in the development of 
> Java tell me, the user and customer, that everything is just fine.
> 
> I really want my software to function correctly and run unfettered, forever, 
> without any "odd" behavior.  My software is involved in applications which need 
> correct results each and every time.  I can not live with any chance that 
> something will sometimes be wrong or may not complete for unexplainable or 
> correctable reasons.
> 
> I just can not understand how the possibility of "wrong", or "incomplete" or 
> "unpredictable" results can feel acceptable.
> 
> To mediate this issue in my software, for some time now, I've lived by the rule, 
> all variables must be declared either volatile or final.  I've mentioned that 
> here before, and I recall that I was sort of pushed back on, saying that was not 
> really necessary.  It's cases like this where an API permits something to happen 
> that is incorrect for its implementation of concurrent access that really 
> highlight the reasons for me.
> 
> Yes, volatile doesn't result in 'non-race' execution.  But, it does keep the 
> compiler for doing things that this reordering might do which would cause 
> software to produce unexpected results in the form of a RuntimeException.  I'm 
> not up for surrounding every statement with a try{}catch.
> 
> Only when performance becomes a consideration, do I start to revisit the 
> volatile declarations.  I've not really had to do very much removing of 
> volatile.  For simple values, I'd either have synchronized setter/getter or 
> volatile already anyway.
> 
>> We can file a bug and make them synchronized or whaetver.
> 
> I know this seems like an over-the-top reaction, but it's just so frustrating to 
> keep having to deal with things like this in my software, and everytime I turn 
> around, there's another something like this that highlights something that I've 
> overlooked or had not expected to possibly be an issue.  My trust in the JDK 
> libraries (except for java.util.concurrent, which Doug has been flawlessly open 
> and considerate of developers needs in my opinion) is waning...
> 
> Maybe it's time to open src.zip and start writing bug reports.  I just really 
> don't think I'll enjoy doing that, but I suppose no one would...
> 
> Gregg Wonderly
> 
>> David
>> 
>>> -----Original Message-----
>>> From: concurrency-interest-bounces@cs.oswego.edu
>>> [mailto:concurrency-interest-bounces@cs.oswego.edu]On Behalf Of Gregg
>>> Wonderly
>>> Sent: Saturday, 16 March 2013 7:46 AM
>>> To: Zhong Yu
>>> Cc: concurrency-interest@cs.oswego.edu; dholmes@ieee.org;
>>> gregg.wonderly@pobox.com
>>> Subject: Re: [concurrency-interest] Thread safety of Thread.getName()
>>> 
>>> 
>>> Yes, that transformation is legal, but it is also broken in the
>>> current design of that class.  Because Thread.setName() is
>>> callable, and because there is no other visible happensBefore
>>> with Thread.getName, the compiler can make the decision to
>>> designate these methods as intra-thread use only.
>>> 
>>> That is the problem.  Even if I subsequently introduce a "happens
>>> before" by synchronizing or locking or something else, I have to
>>> know what the implementation is, know how it is broken and only
>>> then can I actually expect to achieve safe use of the provided API.
>>> 
>>> I assert that this is completely and totally non-productive for
>>> software developers.  The JDK and many other large Java
>>> applications/libraries are broken in subtle ways, such as the
>>> missing "volatile" here.   The compilers rewrite of such code
>>> into non-thread safe segments when the developer(s) might look at
>>> their code and say "that's racy, but safe, because I'll just use
>>> "synchronized" around my calls, is changing the outcome that
>>> breaks SC.  Yes SC is not supposed to be guaranteed in this case,
>>> but do JVM and hotspot developers really feel like they are doing
>>> great and wonderful things, when this kind of stuff is haunting
>>> everybody everywhere, because even the JDK has not been "fixed"?
>>> 
>>> Gregg Wonderly
>>> 
>>> On Mar 15, 2013, at 3:44 PM, Zhong Yu <zhong.j.yu@gmail.com> wrote:
>>> 
>>>> On Fri, Mar 15, 2013 at 3:22 PM, Gregg Wonderly
>>> <gregg@cytetech.com> wrote:
>>>>> Nathan's example below is my issue...
>>>>> 
>>>>> 
>>>>> On 3/15/2013 12:29 AM, David Holmes wrote:
>>>>>> 
>>>>>> Gregg,
>>>>>> 
>>>>>> Gregg Wonderly writes:
>>>>>>> 
>>>>>>> 
>>>>>>> On 3/14/2013 6:14 PM, Nathan Reynolds wrote:
>>>>>>>> 
>>>>>>>> I've inlined the name.toCharArray() and rearranged a few
>>>>>>> 
>>>>>>> operations that JIT
>>>>>>>> 
>>>>>>>> could do.  If another thread reads this.name between its
>>>>>>> 
>>>>>>> creation and the end of
>>>>>>>> 
>>>>>>>> arraycopy() then the thread will see a partially filled char[].
>>>>>>>> 
>>>>>>>>      public final void setName(String name) {
>>>>>>>>          this.name = new char[name.length()];
>>>>>>>>          System.arraycopy(..., 0, result, 0, name.length());
>>>>> 
>>>>> 
>>>>> If two different threads call setName concurrently, then if
>>> the code was
>>>>> transformed as shown above, then it is possible to see
>>>>> 
>>>>> T1:           this.name = new char["some longer string".length()];
>>>>> T2:           this.name = new char["a short string".length()];
>>>>> T1:           System.arraycopy( "some longer string", 0,
>>>>>                        this.name, 0, "some longer string".length());
>>>>> 
>>>>> Which will result in an exception at best and memory
>>> corruption at worse
>>>>> depending on how this.name is referenced.
>>>> 
>>>> I think you are right, the compiler must not issue a load that did not
>>>> exist in the source code (the compiler *is* aware that some other
>>>> thread may be changing the variable).
>>>> 
>>>> The following transformation should be legal
>>>> 
>>>>       public final void setName(String name) {
>>>>            char[]  tmp = new char[name.length()]
>>>>            this.name = tmp;
>>>>            System.arraycopy(name.value, 0, tmp, 0, name.length());
>>>> 
>>>> Zhong Yu
>>>> 
>>>>> Gregg Wonderly
>>>>> 
>>>>> _______________________________________________
>>>>> Concurrency-interest mailing list
>>>>> Concurrency-interest@cs.oswego.edu
>>>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>> _______________________________________________
>>>> Concurrency-interest mailing list
>>>> Concurrency-interest@cs.oswego.edu
>>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>> 
>>> 
>>> _______________________________________________
>>> Concurrency-interest mailing list
>>> Concurrency-interest@cs.oswego.edu
>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>> 
>>> 
>>> -----
>>> No virus found in this message.
>>> Checked by AVG - www.avg.com
>>> Version: 2013.0.2904 / Virus Database: 2641/6158 - Release Date: 03/08/13
>>> Internal Virus Database is out of date.
>>> 
>> 
>> _______________________________________________
>> Concurrency-interest mailing list
>> Concurrency-interest@cs.oswego.edu
>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>> 
>> 
> 
> 
> 
> ------------------------------
> 
> _______________________________________________
> Concurrency-interest mailing list
> Concurrency-interest@cs.oswego.edu
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
> 
> 
> End of Concurrency-interest Digest, Vol 98, Issue 24
> ****************************************************
> 
> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message