From mapreduce-dev-return-2540-apmail-hadoop-mapreduce-dev-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 06:09:49 2010 Return-Path: Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: (qmail 14675 invoked from network); 26 Oct 2010 06:09:48 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 26 Oct 2010 06:09:48 -0000 Received: (qmail 27128 invoked by uid 500); 26 Oct 2010 06:09:48 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 27002 invoked by uid 500); 26 Oct 2010 06:09:46 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 26986 invoked by uid 99); 26 Oct 2010 06:09:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 06:09:45 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 06:09:42 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q69LZN007361 for ; Tue, 26 Oct 2010 06:09:21 GMT Message-ID: <23354995.73441288073361026.JavaMail.jira@thor> Date: Tue, 26 Oct 2010 02:09:21 -0400 (EDT) From: "Ravi Gummadi (JIRA)" To: mapreduce-dev@hadoop.apache.org Subject: [jira] Created: (MAPREDUCE-2153) Bring in more job configuration properties in to the trace file MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org Bring in more job configuration properties in to the trace file --------------------------------------------------------------- Key: MAPREDUCE-2153 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2153 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Reporter: Ravi Gummadi To emulate distributed cache usage in gridmix jobs, there are 9 configuration properties needed to be available in trace file: (1) mapreduce.job.cache.files (2) mapreduce.job.cache.files.visibilities (3) mapreduce.job.cache.files.filesizes (4) mapreduce.job.cache.files.timestamps (5) mapreduce.job.cache.archives (6) mapreduce.job.cache.archives.visibilities (7) mapreduce.job.cache.archives.filesizes (8) mapreduce.job.cache.archives.timestamps (9) mapreduce.job.cache.symlink.create To emulate data compression in gridmix jobs, trace file should contain the following configuration properties: (1) mapreduce.map.output.compress (2) mapreduce.map.output.compress.codec (3) mapreduce.output.fileoutputformat.compress (4) mapreduce.output.fileoutputformat.compress.codec (5) mapreduce.output.fileoutputformat.compress.type Ideally, gridmix should set many job specific configuration properties like io.sort.mb, io.sort.factor, etc when running simulated jobs to get the same effect of original/real job in terms of spilled records, number of merges, etc. TraceBuilder should bring in all these properties into the generated trace file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.