Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <12792078.1191414410940.JavaMail.jira@brutus>
Date: Wed, 3 Oct 2007 05:26:50 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-1926) Design/implement a set of
 compression benchmarks for the map-reduce framework
In-Reply-To: <9035627.1190289152138.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532072 ] 

Hudson commented on HADOOP-1926:
--------------------------------

Integrated in Hadoop-Nightly #259 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/259/])

> Design/implement a set of compression benchmarks for the map-reduce framework
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-1926
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1926
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.15.0
>
>         Attachments: HADOOP-1926_1_20071002.patch, HADOOP-1926_2_20071002.patch
>
>
> It would be nice to benchmark various compression codecs for use in the hadoop (existing codecs like zlib, lzo and in-future bzip2 etc.) and run these along with our nightlies or weeklies.
> Here are some steps:
> a) Fix HADOOP-1851 ( Map output compression codec cannot be set independently of job output compression codec)
> b) Implement a random-text-writer along the lines of examples/randomwriter to generate large amounts of synthetic textual data for use in sort. One way to do this is to pick a word randomly from {{/usr/share/dict/words}} till we get enough bytes per map. To be safe, we could store an array of Strings of a snap-shot of the words in examples/RandomTextWriter.java.
> c) Take a dump of wikipedia (http://download.wikimedia.org/enwiki/) and/or the ebooks from Project Gutenberg (http://www.gutenberg.org/MIRRORS.ALL) and use them as non-synthetic data to run sort/wordcount against.
> For both b) and c) we should setup nightly/weekly benchmark runs with different codecs for reduce-outputs and map-outputs (shuffle) and track each.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.