From core-dev-return-51314-apmail-hadoop-core-dev-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 01 06:10:09 2008 Return-Path: Delivered-To: apmail-hadoop-core-dev-archive@www.apache.org Received: (qmail 35872 invoked from network); 1 Oct 2008 06:10:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 1 Oct 2008 06:10:09 -0000 Received: (qmail 14258 invoked by uid 500); 1 Oct 2008 06:10:04 -0000 Delivered-To: apmail-hadoop-core-dev-archive@hadoop.apache.org Received: (qmail 14221 invoked by uid 500); 1 Oct 2008 06:10:04 -0000 Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-dev@hadoop.apache.org Received: (qmail 13850 invoked by uid 99); 1 Oct 2008 06:10:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Sep 2008 23:10:03 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 06:09:09 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 4540B234C212 for ; Tue, 30 Sep 2008 23:09:44 -0700 (PDT) Message-ID: <448197168.1222841384282.JavaMail.jira@brutus> Date: Tue, 30 Sep 2008 23:09:44 -0700 (PDT) From: "Hemanth Yamijala (JIRA)" To: core-dev@hadoop.apache.org Subject: [jira] Commented: (HADOOP-4149) JobQueueJobInProgressListener.jobUpdated() might not work as expected In-Reply-To: <789513532.1221057284309.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HADOOP-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635954#action_12635954 ] Hemanth Yamijala commented on HADOOP-4149: ------------------------------------------ To see an impact of this bug, I submitted three jobs (using the default scheduler), and changed the priority of one of them. In the jobqueue_details.jsp page, I saw that the job which changed priority is listed twice ! It is the same object being referred to, but as Amar pointed out, since it is not deleted, it shows up twice. I think schedulers could be affected by this, for e.g. if they do not check for a job's run state, they may think the job in the run list or queued list is still to be considered and work on them. It seems like a blocker based on this. Setting it as such. If someone feels not, please comment here. > JobQueueJobInProgressListener.jobUpdated() might not work as expected > --------------------------------------------------------------------- > > Key: HADOOP-4149 > URL: https://issues.apache.org/jira/browse/HADOOP-4149 > Project: Hadoop Core > Issue Type: Bug > Components: mapred > Reporter: Amar Kamat > > {{JobQueueJobInProgressListener}} uses a {{TreeSet}} to store the sorted collection of {{JobInProgress}} objects. The comparator used to sort the JIPs follow the following order > - priority (>=) > - start time (<=) > - job id [jt-identifier, job-index] (<=) > If any JIP object is changed w.r.t priority or start-time, then the TreeSet will be inconsistent. Hence doing a delete might not work. Consider the following > 1) jobs are submitted in the following order > ||number||jobid||priority|| > |1|j1|NORMAL| > |2|j2|LOW| > |3|j3|NORMAL| > 2) The sorted collection will be in the order : {{j1,j3,j2}} > 3) If job3's priority is changed to LOW then the collection wont change but delete will bail out on j1 itself as the comparator will return a -ve number. TreeSet uses the comparator both for sorting and deleting. If i indicates the index in the collection and obj represents the object under consideration, then looks like TreeSet.remove(obj) follows something like : > - continue to search if the compare(i, obj) is -ve > - bail out if the compare(i, obj) is +ve > - delete the obj of compare(i,obj) == 0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.