Return-Path: Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: (qmail 83096 invoked from network); 27 Aug 2010 21:06:15 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 27 Aug 2010 21:06:15 -0000 Received: (qmail 51522 invoked by uid 500); 27 Aug 2010 21:06:15 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 51421 invoked by uid 500); 27 Aug 2010 21:06:14 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 51409 invoked by uid 99); 27 Aug 2010 21:06:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 21:06:14 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 21:06:14 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RL5sKa007192 for ; Fri, 27 Aug 2010 21:05:54 GMT Message-ID: <15276713.40781282943154169.JavaMail.jira@thor> Date: Fri, 27 Aug 2010 17:05:54 -0400 (EDT) From: "Dick King (JIRA)" To: mapreduce-dev@hadoop.apache.org Subject: [jira] Created: (MAPREDUCE-2037) Capturing interim progress times, CPU usage, and memory usage, when tasks reach certain progress thresholds MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Capturing interim progress times, CPU usage, and memory usage, when tasks reach certain progress thresholds ----------------------------------------------------------------------------------------------------------- Key: MAPREDUCE-2037 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2037 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Dick King Assignee: Dick King Fix For: 0.22.0 We would like to capture the following information at certain progress thresholds as a task runs: * Time taken so far * CPU load [either at the time the data are taken, or exponentially smoothed] * Memory load [also either at the time the data are taken, or exponentially smoothed] This would be taken at intervals that depend on the task progress plateaus. For example, reducers have three progress ranges -- [0-1/3], (1/3-2/3], and (2/3-3/3] -- where fundamentally different activities happen. Mappers have different boundaries, I understand, that are not symmetrically placed. Data capture boundaries should coincide with activity boundaries. For the state information capture [CPU and memory] we should average over the covered interval. This data would flow in with the heartbeats. It would be placed in the job history as part of the task attempt completion event, so it could be processed by rumen or some similar tool and could drive a benchmark engine. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.