tajo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jihoon...@apache.org
Subject tajo git commit: TAJO-1314: Documentation for the support of the swift. (jihoon)
Date Mon, 02 Mar 2015 01:16:37 GMT
Repository: tajo
Updated Branches:
  refs/heads/master d769d7872 -> 862514d8a


TAJO-1314: Documentation for the support of the swift. (jihoon)


Project: http://git-wip-us.apache.org/repos/asf/tajo/repo
Commit: http://git-wip-us.apache.org/repos/asf/tajo/commit/862514d8
Tree: http://git-wip-us.apache.org/repos/asf/tajo/tree/862514d8
Diff: http://git-wip-us.apache.org/repos/asf/tajo/diff/862514d8

Branch: refs/heads/master
Commit: 862514d8ad044a08bed6999660e15ac8fbf01e75
Parents: d769d78
Author: Jihoon Son <jihoonson@apache.org>
Authored: Mon Mar 2 10:16:23 2015 +0900
Committer: Jihoon Son <jihoonson@apache.org>
Committed: Mon Mar 2 10:16:23 2015 +0900

----------------------------------------------------------------------
 CHANGES                                         |   2 +
 tajo-docs/src/main/sphinx/index.rst             |   1 +
 tajo-docs/src/main/sphinx/swift_integration.rst | 110 +++++++++++++++++++
 3 files changed, 113 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/tajo/blob/862514d8/CHANGES
----------------------------------------------------------------------
diff --git a/CHANGES b/CHANGES
index 160c54a..729ea2a 100644
--- a/CHANGES
+++ b/CHANGES
@@ -16,6 +16,8 @@ Release 0.11.0 - unreleased
   
   TASKS
 
+    TAJO-1314: Documentation for the support of the swift. (jihoon)
+
 
   SUB TASKS
 

http://git-wip-us.apache.org/repos/asf/tajo/blob/862514d8/tajo-docs/src/main/sphinx/index.rst
----------------------------------------------------------------------
diff --git a/tajo-docs/src/main/sphinx/index.rst b/tajo-docs/src/main/sphinx/index.rst
index 1222f54..0ab50b6 100644
--- a/tajo-docs/src/main/sphinx/index.rst
+++ b/tajo-docs/src/main/sphinx/index.rst
@@ -41,6 +41,7 @@ Table of Contents:
    backup_and_restore
    hcatalog_integration
    hbase_integration
+   swift_integration
    jdbc_driver
    tajo_client_api
    faq

http://git-wip-us.apache.org/repos/asf/tajo/blob/862514d8/tajo-docs/src/main/sphinx/swift_integration.rst
----------------------------------------------------------------------
diff --git a/tajo-docs/src/main/sphinx/swift_integration.rst b/tajo-docs/src/main/sphinx/swift_integration.rst
new file mode 100644
index 0000000..753142d
--- /dev/null
+++ b/tajo-docs/src/main/sphinx/swift_integration.rst
@@ -0,0 +1,110 @@
+*************************************
+OpenStack Swift Integration
+*************************************
+
+Tajo supports OpenStack Swift as one of the underlying storage types.
+In Tajo, Swift objects are represented and recognized by the same URI format as in Hadoop.
+
+You don't need to run Hadoop to run Tajo on Swift, but need to configure it.
+You will also need to configure Swift and Tajo.
+
+For details, please see the following sections.
+
+======================
+Swift configuration
+======================
+
+This step is not mandatory, but is strongly recommended to configure the Swift's proxy-server
with ``list_endpoints`` for better performance.
+More information is available `here <http://docs.openstack.org/developer/swift/middleware.html#module-swift.common.middleware.list_endpoints>`_.
+
+======================
+Hadoop configurations
+======================
+
+You need to configure Hadoop to specify how to access Swift objects.
+Here is an example of ``${HADOOP_HOME}/etc/hadoop/core-site.xml``.
+
+-----------------------
+Common configurations
+-----------------------
+
+.. code-block:: xml
+
+  <property>
+    <name>fs.swift.impl</name>
+    <value>org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem</value>
+    <description>File system implementation for Swift</description>
+  </property>
+  <property>
+    <name>fs.swift.blocksize</name>
+    <value>131072</value>
+    <description>Split size in KB</description>
+  </property>
+
+----------------------------
+Configurations per provider
+----------------------------
+
+.. code-block:: xml
+
+  <property>
+    <name>fs.swift.service.${PROVIDER}.auth.url</name>
+    <value>http://127.0.0.1/v2.0/tokens</value>
+    <description>Keystone authenticaiton URL</description>
+  </property>
+  <property>
+    <name>fs.swift.service.${PROVIDER}.auth.endpoint.prefix</name>
+    <value>/endpoints/AUTH_</value>
+    <description>Keystone endpoints prefix</description>
+  </property>
+  <property>
+    <name>fs.swift.service.${PROVIDER}.http.port</name>
+    <value>8080</value>
+    <description>HTTP port</description>
+  </property>
+  <property>
+    <name>fs.swift.service.${PROVIDER}.region</name>
+    <value>regionOne</value>
+    <description>Region name</description>
+  </property>
+  <property>
+    <name>fs.swift.service.${PROVIDER}.tenant</name>
+    <value>demo</value>
+    <description>Tenant name</description>
+  </property>
+  <property>
+    <name>fs.swift.service.${PROVIDER}.username</name>
+    <value>tajo</value>
+  </property>
+  <property>
+    <name>fs.swift.service.${PROVIDER}.password</name>
+    <value>tajo_password</value>
+  </property>
+  <property>
+    <name>fs.swift.service.${PROVIDER}.location-aware</name>
+    <value>true</value>
+    <description>Flag to enable the location-aware computing</description>
+  </property>
+
+======================
+Tajo configuration
+======================
+
+Finally, you need to configure the classpath of Tajo by adding the following line to ``${TAJO_HOME}/conf/tajo-evn.sh``.
+
+.. code-block:: sh
+
+  export TAJO_CLASSPATH=$HADOOP_HOME/share/hadoop/tools/lib/hadoop-openstack-x.x.x.jar
+
+======================
+Querying on Swift
+======================
+
+Given a provider name *tajo* and a Swift container name *demo*, you can create a Tajo table
with data on Swift as follows.
+
+.. code-block:: sql
+
+  default> create external table swift_table (id int32, name text, score float, type text)
using text with ('text.delimiter'='|') location 'swift://demo.tajo/test.tbl';
+
+Once a table is created, you can execute any SQL queries on that table as other tables stored
on HDFS.
+For query execution details, please refer to :doc:`sql_language`.
\ No newline at end of file


Mime
View raw message