falcon-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From venkat...@apache.org
Subject [2/5] git commit: FALCON-379 Document limitations of Hive support in Falcon. Contributed by Venkatesh Seetharam
Date Tue, 06 May 2014 19:01:36 GMT
FALCON-379 Document limitations of Hive support in Falcon. Contributed by Venkatesh Seetharam


Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/7547a974
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/7547a974
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/7547a974

Branch: refs/heads/branch-0.5
Commit: 7547a9740a772762faf82475148abc01b8df51af
Parents: 07843b0
Author: Venkatesh Seetharam <venkatesh@apache.org>
Authored: Tue May 6 11:22:27 2014 -0700
Committer: Venkatesh Seetharam <venkatesh@apache.org>
Committed: Tue May 6 11:22:27 2014 -0700

----------------------------------------------------------------------
 CHANGES.txt                               |  2 ++
 docs/src/site/twiki/HiveIntegration.twiki | 32 +++++++++++++++++++++++++-
 2 files changed, 33 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/7547a974/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index b26a93e..8ae3ba4 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -28,6 +28,8 @@ Release Version: 0.5-incubating
     Venkatesh Seetharam)
    
   IMPROVEMENTS
+    FALCON-379 Document limitations of Hive support in Falcon (Venkatesh Seetharam)
+
     FALCON-421 Upgrade Hadoop to 2.4.0 (Sowmya Ramesh via Venkatesh Seetharam)
 
     FALCON-386 Upgrade to oozie 4.0.1. (Sowmya Ramesh via Venkatesh Seetharam)

http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/7547a974/docs/src/site/twiki/HiveIntegration.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/HiveIntegration.twiki b/docs/src/site/twiki/HiveIntegration.twiki
index 774a831..2af5a6b 100644
--- a/docs/src/site/twiki/HiveIntegration.twiki
+++ b/docs/src/site/twiki/HiveIntegration.twiki
@@ -113,9 +113,39 @@ accessing the data in tables
 <verbatim>
 bin/hadoop dfs -copyFromLocal $LFS/share/lib/hcatalog/hcatalog-pig-adapter-0.5.0-incubating.jar
share/lib/hcatalog
 </verbatim>
+   * Oozie 4.x with Hadoop-2.x
+Replication jobs are submitted to oozie on the destination cluster. Oozie runs a table export
job
+on RM on source cluster. Oozie server on the target cluster must be configured with source
hadoop
+configs else jobs fail with errors on secure and non-secure clusters as below:
+<verbatim>
+org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not found for ApplicationAttempt
appattempt_1395965672651_0010_000002
+</verbatim>
+
+Make sure all oozie servers that falcon talks to has the hadoop configs configured in oozie-site.xml
+<verbatim>
+<property>
+      <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
+      <value>*=/etc/hadoop/conf,arpit-new-falcon-1.cs1cloud.internal:8020=/etc/hadoop-1,arpit-new-falcon-1.cs1cloud.internal:8032=/etc/hadoop-1,arpit-new-falcon-2.cs1cloud.internal:8020=/etc/hadoop-2,arpit-new-falcon-2.cs1cloud.internal:8032=/etc/hadoop-2,arpit-new-falcon-5.cs1cloud.internal:8020=/etc/hadoop-3,arpit-new-falcon-5.cs1cloud.internal:8032=/etc/hadoop-3</value>
+      <description>
+          Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
+          the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
+          used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
+          the relevant Hadoop *-site.xml files. If the path is relative is looked within
+          the Oozie configuration directory; though the path can be absolute (i.e. to point
+          to Hadoop client conf/ directories in the local filesystem.
+      </description>
+    </property>
+</verbatim>
 
 ---+++ Hive
 
+   * Dated Partitions
+Falcon does not work well when table partition contains multiple dated columns. Falcon only
works
+with a single dated partition. This is being tracked in FALCON-357 which is a limitation
in Oozie.
+<verbatim>
+catalog:default:table4#year=${YEAR};month=${MONTH};day=${DAY};hour=${HOUR};minute=${MINUTE}
+</verbatim>
+
    * [[https://issues.apache.org/jira/browse/HIVE-5550][Hive table import fails for tables
created with default text and sequence file formats using HCatalog API]]
 For some arcane reason, hive substitutes the output format for text and sequence to be prefixed
with Hive.
 Hive table import fails since it compares against the input and output formats of the source
table and they are
@@ -140,7 +170,7 @@ org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer#checkTable
                 .getMsg(" Table inputformat/outputformats do not match"));
       }
 </verbatim>
-
+The above is not an issue with Hive 0.13.
 
 ---++ Hive Examples
 Following is an example entity configuration for lifecycle management functions for tables
in Hive.


Mime
View raw message