hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "ProjectSuggestions" by NigelDaley
Date Sat, 22 Mar 2008 06:28:43 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by NigelDaley:
http://wiki.apache.org/hadoop/ProjectSuggestions

The comment on the change is:
Added research project ideas

------------------------------------------------------------------------------
- Here are some suggestions for some interesting Hadoop projects.  Rough estimates are given
in hours.  These estimates assume an existing understanding of Hadoop.  For more information,
please inquire on the [http://lucene.apache.org/hadoop/mailing_lists.html#Developers Hadoop
mailing lists].  Also, please update and add to these lists.
+ Here are some suggestions for some interesting Hadoop projects.  For more information, please
inquire on the [http://hadoop.apache.org/core/mailing_lists.html#Developers Hadoop mailing
lists].  Also, please update and add to these lists.
  
+  1. [#test_projects Test Projects]
+  1. [#research_projects Research Projects]
+  1. [#tool_investigations Tool Investigations]
+  1. [#random_ideas Random Ideas]
+ 
+ [[Anchor(test_projects)]]
  == Test Projects ==
+ Rough estimates are given in hours.  These estimates assume an existing understanding of
Hadoop.
  ||<rowstyle="background-color: #FFFFE0;"> '''Description''' || '''Estimate''' || '''Links'''
||
  || write Junit test cases that run the Hadoop examples || 16 || ||
  || !MapRed reliability tests || 40 || [https://issues.apache.org/jira/browse/HADOOP-2483
HADOOP-2483] ||
@@ -18, +25 @@

  || implement a Map-Reduce application which can be used to reliably launch speculative tasks
|| 40 || [http://issues.apache.org/jira/browse/HADOOP-2214 HADOOP 2214] ||
  
  
+ [[Anchor(research_projects)]]
+ == Research Projects ==
+ Here are some research project ideas, engineering ideas for new participants, and areas
where domain experts from other fields might add a lot of value by bringing their perspective
into the Hadoop discussion.
+ 
+  * '''Modeling of block placement and replication policies in HDFS'''
+    * Modeling of the expected time to data loss for a give HDFS cluster, given Hadoops replication
policy and protocols.
+    * Modeling of erasure codes and other approaches to replication that might have other
space-performance-reliability tradeoffs.
+ 
+  * '''HDFS Namespace Expansion'''
+ 
+    * Prototyping approaches to scaling the HDFS name space.  Goals - Keep it simple; Preserve
or increase meta-data operations / second; Very large numbers of files (billions to trillions)
& blocks
+ 
+  * '''Hadoop Security Design'''
+ 
+     * An end-to-end proposal for how to support authentication and client side data encryption/decryption,
so that large data sets can be stored in a public HDFS and only jobs launched by authenticated
users can map-reduce or browse the data.  See HADOOP-xxx
+ 
+  * '''Hod ports to various campus work queueing systems'''
+ 
+     * Hod currently supports Torque and has previously supported Condor.  We would like
to have ports to whichever system(s) are used on major campuses (SGE, ...).
+ 
+  * '''Integration of Virtualization (such as Xen) with Hadoop tools'''
+ 
+     * How does one integrate sandboxing of arbitrary user code in C++ and other languages
in a VM such as Xen with the Hadoop framework?  How does this interact with SGE, Torque, Condor?
+ 
+     * As each individual machine has more and more cores/cpus, it makes sense to partition
each machine into multiple virtual machines. That gives us a number of benefits:
+ 
+       * By assigning a virtual machine to a datanode, we effectively isolate the datanode
from the load on the machine caused by other processes, making the datanode more responsive/reliable.
+       * With multiple virtual machines on each machine, we can lower the granularity of
hod scheduling units, making it possible to schedule multiple tasktrackers on the same machine,
improving the overall utilization of the whole clusters.
+       * With virtualization, we can easily snapshot a virtual cluster before releasing it,
making it possible to re-activate the same cluster in the future and start to work from the
snapshot.
+ 
+  * '''Provisioning of long running Services via HOD'''
+ 
+     * Work on a computation model for services on the grid.  The model would include:
+ 
+       * Various tools for defining clients and servers of the service, and at the least
a C++ and Java instantiation of the abstractions
+       * Logical definitions of how to partition work onto a set of servers, i.e. a generalized
shard implementation
+       * A few useful abstractions like locks (exclusive and RW, fairness), leader election,
transactions,
+       * Various communication models for groups of servers belonging to a service, such
as broadcast, unicast, etc.
+       * Tools for assuring QoS, reliability, managing pools of servers for a service with
spares, etc.
+       * Integration with HDFS for persistence, as well as access to local filesystems
+       * Integration with ZooKeeper so that applications can use the namespace
+ 
+  * '''A Hadoop compatible framework for discovering network topology and identifying and
diagnosing hardware that is not functioning correctly'''
+ 
+  * '''An improved framework for debugging and performance optimizing hadoop and streaming
Hadoop jobs'''
+ 
+    * Some suggestions:
+ 
+     * A distributed profiler for measuring distributed map-reduce applications. This would
be real helpful for grid users. It should be able to provide standard profiler features ,
e.g. number of times a method is executed, time of execution, number of times a method caused
some kind of failures, etc; maybe accumulated over all instances of tasks that comprised that
application.
+ 
+  * '''Map reduce performance enhancements'''
+ 
+     * How can we improve the performance of the standard Hadoop performance sort benchmarks?
+ 
+  * '''Sort and shuffle optimization in MR framework'''
+ 
+     * Some example directions:
+       * Memory-based shuffling in MR framework
+       * Combining the results of several maps on rack or node before the shuffle.  This
can reduce seek work and intermediate storage.
+ 
+  * '''Work load characterization from various Hadoop sites'''
+ 
+     * A framework for capturing workload statistics and replaying workload simulations to
allow the assessment of framework improvements.
+ 
+  * '''Other ideas on how to improve the frameworks performance or stability'''
+ 
+  * '''Benchmark suite for Data Intensive Supercomputing'''
+ 
+     * Scientific computation research and software has benefited tremendously due to availability
of benchmark suites such as NAS Parallel Benchmarks. This was a kernel of 7 applications,
starting with EP (embarrassingly parallel) to SP, BT, LU (reflecting varying degree of parallelism
and communication patterns).  A suite for data-intensive supercomputing application benchmarks
would present a target that Hadoop (and other map-reduce implementations) should be optimized
for.
+ 
+  * '''Performance evaluation of existing Locality Sensitive Hashing schemes'''
+ 
+     * Research on new hashing schemes for filesystem namespace partitioning: [http://en.wikipedia.org/wiki/Locality_sensitive_hashing]
+ 
+  * '''An alternate view of files a collection of blocks'''
+ 
+     * Propose an API and sample use cases for a file as a repository of blocks where a user
can add and delete blocks to arbitrary parts of a file.  This would allow holes in files and
moving blocks from one file to another. How does this reconcile with the sequence-of-bytes
view of file? Such an approach may encourage new styles of applications.
+     * To push a bit more in a research direction: UNIX file systems are managed as a sequence-of-bytes
but usually (and in Hadoop's case exclusively) used as a sequence of records. If the filesystem
participates in the record management (like mainframes do for example) you can get same nice
semantic and performance improvements.
+ 
+ 
+ [[Anchor(tool_investigations)]]
  == Tool Investigations ==
  We are always looking for open source testing tools that add value to our development and
build process.  Here are some that need to be investigated.
- ||<rowstyle="background-color: #FFFFE0;"> '''Description''' || '''Estimate''' || '''Links'''
||
+ ||<rowstyle="background-color: #FFFFE0;"> '''Description''' || '''Links''' ||
- || evaluate new unit test frameworks (rewriting some existing test in the new framework
to show benefits) || 80 || [http://junit.org Junit4], [http://www.testng.org TestNG] ||
+ || evaluate new unit test frameworks (rewriting some existing test in the new framework
to show benefits) || [http://junit.org Junit4], [http://www.testng.org TestNG] ||
- || evaluate mock object frameworks for unit testing || 80 || [https://jmockit.dev.java.net
JMockit], [http://easymock.org EasyMock], [http://jmock.org JMock] ||
+ || evaluate mock object frameworks for unit testing || [https://jmockit.dev.java.net JMockit],
[http://easymock.org EasyMock], [http://jmock.org JMock] ||
- || evaluate PMD || 4 || [http://pmd.sourceforge.net PMD] ||
+ || evaluate PMD || [http://pmd.sourceforge.net PMD] ||
- || evaluate concurrency test tools || 12 || [http://javapathfinder.sourceforge.net PathFinder],
[[BR]][http://www.alphaworks.ibm.com/tech/contest ConTest],[[BR]][http://code.google.com/p/checked-uncontended-lock
CheckUncontendedLock],[[BR]][http://code.google.com/p/multithreadedtc MultithreadedTC] ||
+ || evaluate concurrency test tools || [http://javapathfinder.sourceforge.net PathFinder],
[[BR]][http://www.alphaworks.ibm.com/tech/contest ConTest],[[BR]][http://code.google.com/p/checked-uncontended-lock
CheckUncontendedLock],[[BR]][http://code.google.com/p/multithreadedtc MultithreadedTC] ||
- || evaluate Fortify || 8 || [http://opensource.fortifysoftware.com Fortify Tools] ||
+ || evaluate Fortify || [http://opensource.fortifysoftware.com Fortify Tools] ||
- || evaluate dashboards like QALab and Panopticode || 25 || [http://qalab.sourceforge.net
QALab], [http://www.panopticode.org Panopticode] ||
+ || evaluate dashboards like QALab and Panopticode || [http://qalab.sourceforge.net QALab],
[http://www.panopticode.org Panopticode] ||
- || evaluate Faban || 12 || [http://faban.sunsource.net/ Faban], [http://weblogs.java.net/blog/sdo/archive/2007/04/simple_benchmar.html
Blog] ||
+ || evaluate Faban || [http://faban.sunsource.net/ Faban], [http://weblogs.java.net/blog/sdo/archive/2007/04/simple_benchmar.html
Blog] ||
- || evaluate NCSS, a source code metrics suite || 12 || [http://www.kclee.de/clemens/java/javancss
JavaNCSS] ||
+ || evaluate NCSS, a source code metrics suite || [http://www.kclee.de/clemens/java/javancss
JavaNCSS] ||
- || evaluate Java !PathFinder, a software model checker || 12 || [http://javapathfinder.sourceforge.net
JavaPathFinder] ||
+ || evaluate Java !PathFinder, a software model checker || [http://javapathfinder.sourceforge.net
JavaPathFinder] ||
- || evaluate SA4J, a structural dependency analysis || 12 || [http://www.alphaworks.ibm.com/tech/sa4j
SA4J] ||
+ || evaluate SA4J, a structural dependency analysis || [http://www.alphaworks.ibm.com/tech/sa4j
SA4J] ||
- || evaluate JDepend, generates design quality metrics || 12 || [http://www.clarkware.com/software/JDepend.html
JDepend] ||
+ || evaluate JDepend, generates design quality metrics || [http://www.clarkware.com/software/JDepend.html
JDepend] ||
- || evaluate Dependency Finder, generates design quality metrics and dependency graphs ||
12 || [http://depfind.sourceforge.net DepFind] ||
+ || evaluate Dependency Finder, generates design quality metrics and dependency graphs ||
[http://depfind.sourceforge.net DepFind] ||
- || evaluate Classycle, finds class and package cyclic dependencies || 12 || [http://classycle.sourceforge.net
Classycle] ||
+ || evaluate Classycle, finds class and package cyclic dependencies || [http://classycle.sourceforge.net
Classycle] ||
- || evaluate XRadar, an extensible code report tool || 12 || [http://xradar.sourceforge.net
XRadar] ||
+ || evaluate XRadar, an extensible code report tool || [http://xradar.sourceforge.net XRadar]
||
- || evaluate Crap4j, combines cyclomatic complexity and code coverage || 20 || [http://www.crap4j.org/
Crap4j] ||
+ || evaluate Crap4j, combines cyclomatic complexity and code coverage || [http://www.crap4j.org/
Crap4j] ||
- || evaluate Eclipse TPTP, a test and performance tools platform || 12 || [http://www.eclipse.org/tptp
TPTP] ||
+ || evaluate Eclipse TPTP, a test and performance tools platform || [http://www.eclipse.org/tptp
TPTP] ||
- || evaluate J!UnitFactory || 15 || [http://junitfactory.com JUnit Factory] ||
+ || evaluate J!UnitFactory || [http://junitfactory.com JUnit Factory] ||
- || evaluate code review applications like Codestriker and Review Board || 25 || [http://codestriker.sourceforge.net
Codestriker], [http://www.review-board.org Review Board] ||
+ || evaluate code review applications like Codestriker and Review Board || [http://codestriker.sourceforge.net
Codestriker], [http://www.review-board.org Review Board] ||
- || evaluate test automation frameworks || 40 || [http://staf.sourceforge.net/index.php STAF]
||
+ || evaluate test automation frameworks || [http://staf.sourceforge.net/index.php STAF] ||
  
+ [[Anchor(random_ideas)]]
  == Random Ideas ==
  ||<rowstyle="background-color: #FFFFE0;"> '''Description''' || '''Estimate''' || '''Links'''
||
  || Implement a advanced ''job control'' framework to help chain multiple Map-Reduce jobs
i.e. investigate/improve upon existing [http://hadoop.apache.org/core/docs/r0.15.3/mapred_tutorial.html#JobControl
org.apache.hadoop.mapred.jobcontrol] package. || tbd || ||

Mime
View raw message