tajo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hyun...@apache.org
Subject git commit: Update documentation by TAJO-317. (Keuntae Park, jihoon, and hyunsik)
Date Sat, 07 Dec 2013 19:43:49 GMT
Updated Branches:
  refs/heads/master 3c2a6343f -> 7e47f6b01


Update documentation by TAJO-317. (Keuntae Park, jihoon, and hyunsik)


Project: http://git-wip-us.apache.org/repos/asf/incubator-tajo/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tajo/commit/7e47f6b0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tajo/tree/7e47f6b0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tajo/diff/7e47f6b0

Branch: refs/heads/master
Commit: 7e47f6b017717fcc8917b6881fc90c4ade37a8b2
Parents: 3c2a634
Author: Hyunsik Choi <hyunsik@apache.org>
Authored: Sun Dec 8 04:42:37 2013 +0900
Committer: Hyunsik Choi <hyunsik@apache.org>
Committed: Sun Dec 8 04:42:37 2013 +0900

----------------------------------------------------------------------
 .../src/site/markdown/tajo-0.8.0-doc.md         | 60 ++++++++++++++++++--
 1 file changed, 56 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-tajo/blob/7e47f6b0/tajo-project/src/site/markdown/tajo-0.8.0-doc.md
----------------------------------------------------------------------
diff --git a/tajo-project/src/site/markdown/tajo-0.8.0-doc.md b/tajo-project/src/site/markdown/tajo-0.8.0-doc.md
index 2de57f8..17aaa67 100644
--- a/tajo-project/src/site/markdown/tajo-0.8.0-doc.md
+++ b/tajo-project/src/site/markdown/tajo-0.8.0-doc.md
@@ -32,6 +32,7 @@
         * [Launch a Tajo Cluster](#LaunchATajoCluster)
     * [First Query Execution](#FirstQueryExecution)
     * [Distributed mode on HDFS cluster](#DistributedMode)
+    * [Note] (#TutorialNote)
 * [Configuration](#Configuration)
     * [Preliminary](#Preliminary)
         * [catalog-site.xml and tajo-site.xml](#catalog-site_and_tajo-site)
@@ -42,6 +43,7 @@
         * [TajoMaster Heap Memory Size](#TajoMasterHeap) 
         * [Temporary Data Directory](#TemporaryDataDir) 
         * [Maximum number of parallel running tasks for each worker](#MaximumParallelRunningTasks)

+        * [Worker Resource] (#ResourceConfiguration)
     * [Catalog Configuration](#CatalogConfiguration)
     * [RPC/Http Service Configuration and Default Addresses](#DefaultPorts)
         * [Tajo Master](#TajoMasterDefaultPorts)            
@@ -249,6 +251,11 @@ Then, execute start-tajo.sh
 $ $TAJO_HOME/bin/start-tajo.sh
 ```
 
+## <a name="TutorialNote"></a>Note
+ * In default, each worker is set to very little resource capacity. In order to increase
parallel degree, see [this section] (#MaximumParallelRunningTasks).
+ * In default, TajoMaster listens on 127.0.0.1 for clients. To allow remote clients to access
TajoMaster, please set **tajo.master.client-rpc.address** config to tajo-site.xml. You can
get more information from [RPC/Http Service Configuration and Default Addresses](#DefaultPorts)
section.
+ 
+
 Enjoy Apache Tajo!
 
 # <a name="Configuration"></a>Configuration
@@ -342,19 +349,64 @@ TajoWorker stores temporary data on local file system due to out-of-core
algorit
 
 ### <a name="MaximumParallelRunningTasks"></a>Maximum number of parallel running
tasks for each worker
 
-Each worker can execute multiple tasks at a time. Tajo allows users to specify the maximum
number of parallel running tasks for each worker.
+In Tajo, the capacity of running tasks in parallel are determined by available resources
and workload of running queries. In order to specify it, please see [Worker Resources] (#ResourceConfiguration)
section.
+
+### <a name="ResourceConfiguration"></a>Worker Resources
+
+Each worker can execute multiple tasks simultaneously.
+In Tajo, users can specify the total size of memory and the number of disks for each worker.
Available resources affect how many tasks are executed simultaneously.
+
+In order to specify the resource capacity of each worker, you should add the following configs
to *tajo-site.xml* :
+
+| property name                   | description              | value type          | default
value           |
+| ------------------------------- | ------------------------ | ------------------- | -----------------------
|
+| tajo.worker.resource.cpu-cores  | the number of cpu cores  | integer             | 1  
                    |
+| tajo.worker.resource.memory-mb  | memory size (MB)         | integer             | 1024
                   |
+| tajo.worker.resource.disks      | the number of disks      | integer             | 1  
                    |
+
+_NOTE: Currently, QueryMaster requests 512MB memory and 1.0 disk per task for the backward
compatibility._
+
+#### Example
+
+Assume that you want to give 5120 MB memory, 6.0 disks, and 24 cores on each worker. The
example configuration is as follows:
 
 *tajo-site.xml*
 
 ```
   <property>
-    <name>tajo.worker.parallel-execution.max-num</name>
-    <value>12</value>
+    <name>tajo.worker.resource.tajo.worker.resource.cpu-cores</name>
+    <value>24</value>
+  </property>
+  
+   <property>
+    <name>tajo.worker.resource.memory-mb</name>
+    <value>5120</value>
   </property>
+  
+  <property>
+    <name>tajo.worker.resource.tajo.worker.resource.disks</name>
+    <value>6.0</value>
+  </property>  
 ```
 
+#### Dedicated Mode
+Tajo provides a dedicated mode that allows each worker in a Tajo cluster to use whole available
system resources including cpu-cores, memory, and disks. For this mode, a user should add
the following config to *tajo-site.xml* :
+
+```xml
+  <property>
+    <name>tajo.worker.resource.dedicated</name>
+    <value>true</value>
+  </property>
+```
+
+In addition, it can limit the memory capacity used for Tajo worker as follows:
+
+| property name                                | description                            
         | value type          | default value           |
+| ---------------------------------------------| ------------------------------------------------
| ------------------- | ----------------------- |
+| tajo.worker.resource.dedicated-memory-ratio  | how much memory to be used in whole memory
         | float               | 0.8                     |
+
 ## <a name="CatalogConfiguration"></a>Catalog Configuration
-If you want to customize the catalog service, copy $TAJO_HOME/conf/catalog-site.xml.templete
to catalog-site.xml. Then, add the following configs to catalog-site.xml. Note that the default
configs are enough to launch Tajo cluster in most cases.
+If you want to customize the catalog service, copy $TAJO_HOME/conf/catalog-site.xml.template
to catalog-site.xml. Then, add the following configs to catalog-site.xml. Note that the default
configs are enough to launch Tajo cluster in most cases.
 
 * tajo.catalog.master.addr - If you want to launch a Tajo cluster in distributed mode, you
must specify this address. For more detail information, see [Default Ports](#DefaultPorts).
 * tajo.catalog.store.class - If you want to change the persistent storage of the catalog
server, specify the class name. Its default value is tajo.catalog.store.DerbyStore. In the
current version, Tajo provides three persistent storage classes as follows:


Mime
View raw message