incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Trivial Update of "TajoProposal" by HyunsikChoi
Date Mon, 25 Feb 2013 07:06:31 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "TajoProposal" page has been changed by HyunsikChoi:
http://wiki.apache.org/incubator/TajoProposal?action=diff&rev1=5&rev2=6

Comment:
Improved some sections.

  Tajo is a distributed data warehouse system for Hadoop.
  
  = Proposal =
- Tajo is a relational and distributed data warehouse system for Hadoop. Tajo is designed
for low-latency and scalable ad-hoc queries, online aggregation and ETL on large-data sets
by leveraging advanced database techniques. It supports SQL standards. Tajo is inspired by
Dryad, MapReduce, Dremel, Scope, and parallel databases. It has its own query engine which
allows direct control of distributed execution and data flow. As a result, Tajo has a variety
of query evaluation strategies and more optimization opportunities. In addition, Tajo will
have a native columnar execution and and its optimizer. Tajo will be an alternative choice
to Hive/Pig on the top of MapReduce.
+ Tajo is a relational and distributed data warehouse system for Hadoop. Tajo is designed
for low-latency and scalable ad-hoc queries, online aggregation and ETL on large-data sets
by leveraging advanced database techniques. It supports SQL standards. Tajo is inspired by
Dryad, MapReduce, Dremel, Scope, and parallel databases. Tajo uses HDFS as a primary storage
layer, and it has its own query engine which allows direct control of distributed execution
and data flow. As a result, Tajo has a variety of query evaluation strategies and more optimization
opportunities. In addition, Tajo will have a native columnar execution and and its optimizer.
Tajo will be an alternative choice to Hive/Pig on the top of MapReduce.
  
  = Background =
  Big data analysis has gained much attention in the industrial. Open source communities have
proposed scalable and distributed solutions for ad-hoc queries on big data. However, there
is still room for improvement. Markets need more faster and efficient solutions. Recently,
some alternatives (e.g., Cloudera's Impala and Amazon Redshift) have come out.
@@ -19, +19 @@

  
  = Current Status =
  
- Tajo is under alpha stage. Users can submit usual SQL queries except for nested queries.
The queries are executed across a number of clusters. We have carried out benchmark test using
TPC-H 1TB on 32 cluster nodes. Tajo already supports various row/column file formats, such
as CSV, RowFile (we have developed), RCFile, and Trevni.
+ Tajo is in the alpha stage. Users can execute usual SQL queries (e.g., selection, projection,
group-by, join, union and sort) except for nested queries. Tajo provides various row/column
storage formats, such as CSV, RowFile (a row-store file we have implemented), RCFile, and
Trevni, and it also has a rudimentary ETL feature to transform one data format to another
data format. In addition, Tajo provides hash and range repartitions. By using both repartition
methods, Tajo processes aggregation, join, and sort queries over a number of cluster nodes.
To evaluate the performance, we have carried out benchmark test using TPC-H 1TB on 32 cluster
nodes.
  
  == Meritocracy ==
  
  We will discuss the milestone and the future plan in an open forum. We plan to encourage
an environment that supports a meritocracy. The contributors will have different privileges
according to their contributions.
  
  == Community ==
- Big data analysis has gained attention from open source communities, industrial and academic
areas. Some projects related to Hadoop already have very large and active communities. We
expect that Tajo also will establish an active community. Since Tajo is relatively mature
than other projects that aims at low-latency projects and is already under alpha stage, it
will attract a large community soon.
+ Big data analysis has gained attention from open source communities, industrial and academic
areas. Some projects related to Hadoop already have very large and active communities. We
expect that Tajo also will establish an active community. Since Tajo is relatively mature
than other projects that aims at low-latency system projects that support SQL and is already
in the alpha stage, it will attract a large community soon.
  
  == Core Developers ==
  Core developers are very experienced in the Apache Hadoop ecosystem. To achieve more diversity
of developers, we will be eager to recruit developers from diverse companies.
-  * Eli Reisman <ereisman at apache dot org>
+  * Eli Reisman <ereisman AT apache DOT org>
-  * Hyunsik Choi <hyunsik at apache dot org>
+  * Hyunsik Choi <hyunsik AT apache DOT org>
-  * Jihoon Son <ghoonson at gmail dot com>
+  * Jihoon Son <ghoonson AT gmail DOT com>
-  * Jin Ho Kim <jhkim at gruter dot com>
+  * Jin Ho Kim <jhkim AT gruter DOT com>
-  * Sangwook Kim <swkim at inervit dot com>
+  * Sangwook Kim <swkim AT inervit DOT com>
  
  == Alignment ==
  Tajo employs Apache Hadoop Yarn as a resource management platform for large clusters. It
uses HDFS as a primary storage layer. It already supports Hadoop-related data formats (RCFile,
Trevni) and will support ORC file. In addition, we have a plan to integrate Tajo with other
products of Hadoop ecosystem. Tajo's modules are well organized, and these modules can also
be used for other projects.
@@ -45, +45 @@

  Most of codes have been developed by two core developers, who are Hyunsik Choi and Jihoon
Son. However, they are guaranteed to have enough time to develop Tajo for years. As you can
see the commit history, they have participated in this project for about two years. Recently,
Tajo has been supported by two IT companies in Korea. In addition, we will be eager to recruit
additional committers in order to mitigate this risk.
  
  == Inexperience with Open Source ==
- Most of the initial committers have experience working on open source projects. Hyunsik
Choi has experience as committers and PMC on other Apache projects.
+ Most of the initial committers have experience working on open source projects. Eli Reisman
and Hyunsik Choi have experience as committers and PMC members on other Apache projects.
  
  == Homogeneous Developers ==
- Although they have three affiliations, what the core developers are all in South Korea is
a risk. This is because their offline activities are limited due to their location. Since
we surely recognize this risk, we will write more complete documents and presentation materials
as early as possible. Then, we will be eager to recruit additional committers around the world.
+ Although they have four affiliations, what most of core developers are in South Korea is
a risk. This is because their offline activities are limited due to their location. Since
we surely recognize this risk, we will write more complete documents and presentation materials
as early as possible. Then, we will be eager to recruit additional committers around the world.
  
  == Reliance on Salaried Developers ==
- Hyunsik Choi and Jihoon Son belong to Database lab., Korea Univ. They will be paid by the
lab to contribute Tajo for two years. Other core developers are paid by their employer to
contribute to this project. In addition, we will be eager to recruit additional committers
including salaried and non-salaried developers.
+ It is expected that Tajo development will occur on both salaried time and on volunteer time.
Hyunsik Choi and Jihoon Son belong to Database lab., Korea Univ. They will be paid by the
lab to contribute Tajo for years. Jin Ho Kim and Sangwook Kim are paid by their employer to
contribute to this project. Eli Reisman will contribute to this project on volunteer time.
In addition, we will be eager to recruit additional committers including salaried and non-salaried
developers.
  
  == Relationships with Other Apache Products ==
  Tajo has some overlapping function with Apache Incubator Drill. However, it is more mature
than Drill. In addition, there are some significant differences. Drill is a distributed system
specialized for low-latency query processing by using column operation and streaming intermediate
data. Drill has very simple query optimizer. However, some queries including big-big table
join and sort are not available in that manner. Drill will support some of query types.
  
  In contrast, Tajo has advanced query optimization system. Tajo mainly aims at scalable and
efficient processing on all query types. By using the query optimizer, Tajo will only chase
low latency query processing for some query types that can be executed in online aggregation
manner.
  
- Besides, Tez being voted in incubator-general@a.o has some overlapping functions with Tajo.
However, Tez is under pre-alpha stage and may be a prototype. When Tez becomes feasible, Tajo
could use Tez as an underlying framework according to the applicability. However, Tajo will
still use its row/native columnar execution engine and its optimizer. Tajo may be potentially
the first application of Tez.
+ Besides, Tez has some overlapping functions with Tajo. However, Tez is under pre-alpha stage
and may be a prototype. When Tez becomes feasible, Tajo could use Tez as an underlying framework
according to the applicability. However, Tajo will still use its row/native columnar execution
engine and its optimizer. Tajo may be potentially the first application of Tez.
  
  == A Excessive Fascination with the Apache Brand ==
  We believe that the Apache brand will help us to find contributors and to grow the community.
The community and development process will make this project more stable and ubiquitous APIs.
In addition, Tajo depends other project in Apache Hadoop ecosystem. We expect that cooperative
work occurs with other projects in the same place.
  
  = Documentation =
- Tajo's demonstration paper was accepted to IEEE ICDE 2013. Since this conference will be
held in April 2013, we cannot publicly show the paper. So, we attached some presentation material.
Checkout this [[http://dbserver.korea.ac.kr/~hyunsik/files/tajo_intro.pdf|Slide]].
+ Tajo's demonstration paper was accepted to IEEE ICDE 2013. Since this conference will be
held in April 2013, we cannot publicly show the paper. So, we attached some presentation material.
Checkout this [[http://www.slideshare.net/hyunsikchoi/tajo-intro|Slide]].
  
- In addition, some documents (e.g., getting started) are available on [[http://tajo-project.github.com/tajo/]]
+ In addition, some documents (e.g., getting started) are available at [[http://tajo-project.github.com/tajo/]]
  
  = Initial Source =
  The initial source code has been developed in the Database Lab. Korea Univ. This is implemented
in Java and has almost 100,000 lines except for parser and protobuf generated codes. Currently,
initial source code is already avilable on GitHub at [[https://github.com/tajo-project/tajo]].
@@ -108, +108 @@

    * http://wiki.apache.org/tajo
  
  = Initial Committers =
-  * Eli Reisman <ereisman at apache dot org>
+  * Eli Reisman <ereisman AT apache DOT org>
-  * Hyunsik Choi <hyunsik at apache dot org>
+  * Hyunsik Choi <hyunsik AT apache DOT org>
-  * Jihoon Son <ghoonson at gmail dot com>
+  * Jihoon Son <ghoonson AT gmail DOT com>
-  * Jin Ho Kim <jhkim at gruter dot com>
+  * Jin Ho Kim <jhkim AT gruter DOT com>
-  * Sangwook Kim <swkim at inervit dot com>
+  * Sangwook Kim <swkim AT inervit DOT com>
  
  = Affiliations =
   * Eli Reisman (Hortonworks)
@@ -131, +131 @@

  
  == Champion ==
  
-  * Jakob Homan <ghoman at apache dot org>
+  * Jakob Homan <ghoman AT apache DOT org>
  
  == Nominated Mentors ==
  
-  * Chris Mattman <mattmann at apache dot org>
+  * Chris Mattman <chris DOT a DOT mattmann AT jpl DOT nasa DOT gov>
-  * Jakob Homan <jghoman at apache dot org>
+  * Jakob Homan <jghoman AT apache DOT org>
-  * Owen O'Malley <omalley at apache dot org>
+  * Owen O'Malley <omalley AT apache DOT org>
  
  == Sponsoring Entity ==
  Apache Incubator

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message