Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-Id: <EA485DB7-CD98-4F08-B9CE-D79A5686E2F2@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: core-dev@hadoop.apache.org
In-Reply-To: <1466c1d60811161309w196d7f88of2c8c9cdfb98d6c4@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: Re: Removing JMM issues
Date: Sun, 16 Nov 2008 15:34:30 -0800
References: <1466c1d60811161309w196d7f88of2c8c9cdfb98d6c4@mail.gmail.com>


On Nov 16, 2008, at 1:09 PM, Peter Veentjer wrote:

> I have had a quick look at the source code of Hadoop and it appears
> there there are some issues with the JMM. In some places it is done
> correctly,  in some places partially and in some places it incorrect.

That is believable. Clearly some of the problems have been fixed, but  
Hadoop is moving fast and new code is being added every week. There  
have been bug reports on concurrency stuff that turned out to be false  
positives. *smile* It is amazing how much testing code gets when you  
run it on 20,000 nodes 24x7 and even rare cases have been hit in  
production.

> There also are some design issues with concurrency as well and I think
> the Hadoop project could benefit from overall solution instead of just
> putting out small fires.

Yeah, we've talked about it for a long time. (See HADOOP-869.) The  
particularly problematic parts of the call graph in the JobTracker are  
when the lower levels call back into the higher layers. We've had to  
be careful to preserve lock order in those cases to avoid deadlock.

> So who are the guys to get in touch with?

This list is exactly the right guys.

> Together with the Hadoop developers I want to further improve the
> quality of this very interesting project.

Jump right in. *Smile*

-- Owen