Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2A2D9173DE for ; Tue, 24 Mar 2015 18:02:57 +0000 (UTC) Received: (qmail 56486 invoked by uid 500); 24 Mar 2015 18:02:50 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 56380 invoked by uid 500); 24 Mar 2015 18:02:50 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 56370 invoked by uid 99); 24 Mar 2015 18:02:50 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Mar 2015 18:02:50 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of xeonmailinglist@gmail.com designates 209.85.212.181 as permitted sender) Received: from [209.85.212.181] (HELO mail-wi0-f181.google.com) (209.85.212.181) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Mar 2015 18:02:22 +0000 Received: by wibg7 with SMTP id g7so81592410wib.1 for ; Tue, 24 Mar 2015 11:00:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type; bh=NPXjoedbkI3CxDJpH3BlmLLFoR98hbPAWmWjHoaBayU=; b=KPhZUPZ5x1u37epzFQZVa3xsvzGXJje6qLBsag5hu8jtCLuzcPx4yXa+infITYeyqf 49T3MOMjj8nJx/qcXX9aZJaPb+OXnOFyTqdqoEgYkUoSdlBSoe33EWbU5gWIQVOWvgBQ WXi29ZGs3aaZa1pjK/8xWoMe30sYrycEcNJPJjTDsXfV8fDfdY63hjNiE+/81Cn7jOk3 rUQ+v8dpcfV+Z6P+skXJvzdR/0Cxnme50KznHasGZRov1LGJP8zkRWNyaPtCger6kmGl cfV56rwG+xp4/bpz8Dsy0kXBoT9IlFinhAuEHvoQ7YFWZZ78+qQV0YiLdN0fKDTUnQJC a33w== X-Received: by 10.180.74.47 with SMTP id q15mr30172389wiv.90.1427220006575; Tue, 24 Mar 2015 11:00:06 -0700 (PDT) Received: from [10.101.227.190] ([194.117.18.101]) by mx.google.com with ESMTPSA id ub1sm7077478wjc.43.2015.03.24.11.00.05 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Mar 2015 11:00:05 -0700 (PDT) Message-ID: <5511A624.7050907@gmail.com> Date: Tue, 24 Mar 2015 18:00:04 +0000 From: xeonmailinglist-gmail User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: "user@hadoop.apache.org" Subject: Best tests for CPU or network bounded executions in hadoop-gridmix-2.6.0.jar Content-Type: multipart/alternative; boundary="------------090507010108090403000708" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------090507010108090403000708 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Hi, I am trying to see what are the current tests that the Gridmix2 package do in MapReduce2.x. I see that the Gridmix jar |hadoop-gridmix-2.6.0.jar|does not have anymore the WebDataSort, WebDataScan, MonsterQuery, Combiner, and Streaming tests. The current tests are in [1]. From all of these tests, what is the best test that is CPU bounded (like the PI calculation), and the best test that is network bounded (like the wordcount)? When I say CPU bounded tests, I say the tests that need lots of CPU calculation, and, for the network bounded, I mean the jobs that needs to transfer lots of data between mappers and reducers. [1] |~/Programs/hadoop/share/hadoop/tools/sources$ tree org/apache/hadoop/mapred/ org/apache/hadoop/mapred/ └── gridmix ├── CommonJobTest.java ├── DebugJobFactory.java ├── DebugJobProducer.java ├── DummyResourceCalculatorPlugin.java ├── GridmixTestUtils.java ├── TestCompressionEmulationUtils.java ├── TestDistCacheEmulation.java ├── TestFilePool.java ├── TestFileQueue.java ├── TestGridMixClasses.java ├── TestGridmixMemoryEmulation.java ├── TestGridmixRecord.java ├── TestGridmixStatistics.java ├── TestGridmixSubmission.java ├── TestGridmixSummary.java ├── TestHighRamJob.java ├── TestLoadJob.java ├── TestPseudoLocalFs.java ├── TestRandomAlgorithm.java ├── TestRandomTextDataGenerator.java ├── TestRecordFactory.java ├── TestResourceUsageEmulators.java ├── TestSleepJob.java └── TestUserResolve.java | ​ -- -- Thanks, --------------090507010108090403000708 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

Hi,

I am trying to see what are the current tests that the Gridmix2 package do in MapReduce2.x. I see that the Gridmix jar hadoop-gridmix-2.6.0.jardoes not have anymore the WebDataSort, WebDataScan, MonsterQuery, Combiner, and Streaming tests. The current tests are in [1].

From all of these tests, what is the best test that is CPU bounded (like the PI calculation), and the best test that is network bounded (like the wordcount)?

When I say CPU bounded tests, I say the tests that need lots of CPU calculation, and, for the network bounded, I mean the jobs that needs to transfer lots of data between mappers and reducers.

[1]

~/Programs/hadoop/share/hadoop/tools/sources$ tree org/apache/hadoop/mapred/
org/apache/hadoop/mapred/
└── gridmix
    ├── CommonJobTest.java
    ├── DebugJobFactory.java
    ├── DebugJobProducer.java
    ├── DummyResourceCalculatorPlugin.java
    ├── GridmixTestUtils.java
    ├── TestCompressionEmulationUtils.java
    ├── TestDistCacheEmulation.java
    ├── TestFilePool.java
    ├── TestFileQueue.java
    ├── TestGridMixClasses.java
    ├── TestGridmixMemoryEmulation.java
    ├── TestGridmixRecord.java
    ├── TestGridmixStatistics.java
    ├── TestGridmixSubmission.java
    ├── TestGridmixSummary.java
    ├── TestHighRamJob.java
    ├── TestLoadJob.java
    ├── TestPseudoLocalFs.java
    ├── TestRandomAlgorithm.java
    ├── TestRandomTextDataGenerator.java
    ├── TestRecordFactory.java
    ├── TestResourceUsageEmulators.java
    ├── TestSleepJob.java
    └── TestUserResolve.java
​
-- 
--

Thanks,
--------------090507010108090403000708--