falcon-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From venkat...@apache.org
Subject [2/5] git commit: FALCON-379 Document limitations of Hive support in Falcon. Contributed by Venkatesh Seetharam
Date Tue, 06 May 2014 18:52:54 GMT
FALCON-379 Document limitations of Hive support in Falcon. Contributed by Venkatesh Seetharam


Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/8c48a2ce
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/8c48a2ce
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/8c48a2ce

Branch: refs/heads/master
Commit: 8c48a2ce01641c136c00ff4c543602bff8fdb054
Parents: b08eacb
Author: Venkatesh Seetharam <venkatesh@apache.org>
Authored: Tue May 6 11:22:21 2014 -0700
Committer: Venkatesh Seetharam <venkatesh@apache.org>
Committed: Tue May 6 11:22:21 2014 -0700

----------------------------------------------------------------------
 CHANGES.txt                               |  2 ++
 docs/src/site/twiki/HiveIntegration.twiki | 32 +++++++++++++++++++++++++-
 2 files changed, 33 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/8c48a2ce/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 6b20f60..ca659e3 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -40,6 +40,8 @@ Release Version: 0.5-incubating
     Venkatesh Seetharam)
    
   IMPROVEMENTS
+    FALCON-379 Document limitations of Hive support in Falcon (Venkatesh Seetharam)
+
     FALCON-421 Upgrade Hadoop to 2.4.0 (Sowmya Ramesh via Venkatesh Seetharam)
 
     FALCON-386 Upgrade to oozie 4.0.1. (Sowmya Ramesh via Venkatesh Seetharam)

http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/8c48a2ce/docs/src/site/twiki/HiveIntegration.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/HiveIntegration.twiki b/docs/src/site/twiki/HiveIntegration.twiki
index 774a831..2af5a6b 100644
--- a/docs/src/site/twiki/HiveIntegration.twiki
+++ b/docs/src/site/twiki/HiveIntegration.twiki
@@ -113,9 +113,39 @@ accessing the data in tables
 <verbatim>
 bin/hadoop dfs -copyFromLocal $LFS/share/lib/hcatalog/hcatalog-pig-adapter-0.5.0-incubating.jar
share/lib/hcatalog
 </verbatim>
+   * Oozie 4.x with Hadoop-2.x
+Replication jobs are submitted to oozie on the destination cluster. Oozie runs a table export
job
+on RM on source cluster. Oozie server on the target cluster must be configured with source
hadoop
+configs else jobs fail with errors on secure and non-secure clusters as below:
+<verbatim>
+org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not found for ApplicationAttempt
appattempt_1395965672651_0010_000002
+</verbatim>
+
+Make sure all oozie servers that falcon talks to has the hadoop configs configured in oozie-site.xml
+<verbatim>
+<property>
+      <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
+      <value>*=/etc/hadoop/conf,arpit-new-falcon-1.cs1cloud.internal:8020=/etc/hadoop-1,arpit-new-falcon-1.cs1cloud.internal:8032=/etc/hadoop-1,arpit-new-falcon-2.cs1cloud.internal:8020=/etc/hadoop-2,arpit-new-falcon-2.cs1cloud.internal:8032=/etc/hadoop-2,arpit-new-falcon-5.cs1cloud.internal:8020=/etc/hadoop-3,arpit-new-falcon-5.cs1cloud.internal:8032=/etc/hadoop-3</value>
+      <description>
+          Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
+          the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
+          used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
+          the relevant Hadoop *-site.xml files. If the path is relative is looked within
+          the Oozie configuration directory; though the path can be absolute (i.e. to point
+          to Hadoop client conf/ directories in the local filesystem.
+      </description>
+    </property>
+</verbatim>
 
 ---+++ Hive
 
+   * Dated Partitions
+Falcon does not work well when table partition contains multiple dated columns. Falcon only
works
+with a single dated partition. This is being tracked in FALCON-357 which is a limitation
in Oozie.
+<verbatim>
+catalog:default:table4#year=${YEAR};month=${MONTH};day=${DAY};hour=${HOUR};minute=${MINUTE}
+</verbatim>
+
    * [[https://issues.apache.org/jira/browse/HIVE-5550][Hive table import fails for tables
created with default text and sequence file formats using HCatalog API]]
 For some arcane reason, hive substitutes the output format for text and sequence to be prefixed
with Hive.
 Hive table import fails since it compares against the input and output formats of the source
table and they are
@@ -140,7 +170,7 @@ org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer#checkTable
                 .getMsg(" Table inputformat/outputformats do not match"));
       }
 </verbatim>
-
+The above is not an issue with Hive 0.13.
 
 ---++ Hive Examples
 Following is an example entity configuration for lifecycle management functions for tables
in Hive.


Mime
View raw message