hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-2725) Fix flaky testing infrastructure
Date Wed, 18 Jan 2012 20:06:39 GMT

    [ https://issues.apache.org/jira/browse/HIVE-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188669#comment-13188669
] 

Ashutosh Chauhan commented on HIVE-2725:
----------------------------------------

On the same build machine after applying hive-2589_2.patch following tests pass which apache
jenkins reports as failures:
{code}
$ svn info
Path: .
URL: https://svn.apache.org/repos/asf/hive/trunk
Repository Root: https://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1233014
Node Kind: directory
Schedule: normal
Last Changed Author: cws
Last Changed Rev: 1232766
Last Changed Date: 2012-01-18 07:10:02 +0000 (Wed, 18 Jan 2012)

$ svn diff
Index: metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
===================================================================
--- metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java	(revision 1233014)
+++ metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java	(working copy)
@@ -26,14 +26,17 @@
 import java.io.IOException;
 import java.util.AbstractMap;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.Collections;
 import java.util.Formatter;
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
 import java.util.Properties;
+import java.util.Set;
 import java.util.Timer;
 import java.util.regex.Pattern;
 
@@ -1562,7 +1565,23 @@
             part.getParameters().get(Constants.DDL_TIME) == null) {
           part.putToParameters(Constants.DDL_TIME, Long.toString(time));
         }
+ 
+        Map<String,String> tblParams = tbl.getParameters();
+        String inheritProps =  hiveConf.getVar(ConfVars.METASTORE_PART_INHERIT_TBL_PROPS).trim();
+        // Default value is empty string in which case no properties will be inherited.
+        // * implies all properties needs to be inherited
+        Set<String> inheritKeys = new HashSet<String>(Arrays.asList(inheritProps.split(",")));
+        if(inheritKeys.contains("*")){
+          inheritKeys =  tblParams.keySet();
+        }
 
+        for (String key : inheritKeys) {
+          String paramVal = tblParams.get(key);
+          if(null != paramVal){ // add the property only if it exists in table properties
+            part.putToParameters(key, paramVal);
+          }
+        }
+
         success = ms.addPartition(part);
 
       } finally {
Index: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
===================================================================
--- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java	(revision 1233014)
+++ common/src/java/org/apache/hadoop/hive/conf/HiveConf.java	(working copy)
@@ -119,6 +119,7 @@
       HiveConf.ConfVars.METASTORE_EVENT_EXPIRY_DURATION,
       HiveConf.ConfVars.METASTORE_RAW_STORE_IMPL,
       HiveConf.ConfVars.METASTORE_END_FUNCTION_LISTENERS,
+      HiveConf.ConfVars.METASTORE_PART_INHERIT_TBL_PROPS,
       };
 
   /**
@@ -296,6 +297,7 @@
     METASTORE_NON_TRANSACTIONAL_READ("javax.jdo.option.NonTransactionalRead", true),
     METASTORE_CONNECTION_USER_NAME("javax.jdo.option.ConnectionUserName", "APP"),
     METASTORE_END_FUNCTION_LISTENERS("hive.metastore.end.function.listeners", ""),
+    METASTORE_PART_INHERIT_TBL_PROPS("hive.metastore.partition.inherit.table.properties",""),
 
     // CLI
     CLIIGNOREERRORS("hive.cli.errors.ignore", false),

$ ant test -Dtestcase=TestCliDriver -Dqfile=repair.q
    [junit] Running org.apache.hadoop.hive.cli.TestCliDriver
    [junit] Begin query: repair.q
    [junit] diff -a /home/hortonas/hive-trunk/build/ql/test/logs/clientpositive/repair.q.out
/home/hortonas/hive-trunk/ql/src/test/results/clientpositive/repair.q.out
    [junit] Done query: repair.q
    [junit] Cleaning up TestCliDriver
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 2.733 sec

$ ant test -Dtestcase=TestCliDriver -Dqfile=router_join_ppr.q
    [junit] Running org.apache.hadoop.hive.cli.TestCliDriver
    [junit] Begin query: router_join_ppr.q
    [junit] diff -a /home/hortonas/hive-trunk/build/ql/test/logs/clientpositive/router_join_ppr.q.out
/home/hortonas/hive-trunk/ql/src/test/results/clientpositive/router_join_ppr.q.out
    [junit] Done query: router_join_ppr.q
    [junit] Cleaning up TestCliDriver
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 23.946 sec


 [junit] Begin query: sample1.q
    [junit] Deleted file:/home/hortonas/hive-trunk/build/ql/test/data/warehouse/dest1
    [junit] diff -a /home/hortonas/hive-trunk/build/ql/test/logs/clientpositive/sample1.q.out
/home/hortonas/hive-trunk/ql/src/test/results/clientpositive/sample1.q.out
    [junit] Done query: sample1.q
    [junit] Cleaning up TestCliDriver
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 13.215 sec

$ ant test -Dtestcase=TestCliDriver -Dqfile=source.q
 [junit] Running org.apache.hadoop.hive.cli.TestCliDriver
    [junit] Begin query: source.q
    [junit] diff -a /home/hortonas/hive-trunk/build/ql/test/logs/clientpositive/source.q.out
/home/hortonas/hive-trunk/ql/src/test/results/clientpositive/source.q.out
    [junit] Done query: source.q
    [junit] Cleaning up TestCliDriver
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 2.46 sec


{code}

Cant figure out whats wrong with apache jenkins server or in hive test infra.

                
> Fix flaky testing infrastructure 
> ---------------------------------
>
>                 Key: HIVE-2725
>                 URL: https://issues.apache.org/jira/browse/HIVE-2725
>             Project: Hive
>          Issue Type: Bug
>          Components: Testing Infrastructure
>    Affects Versions: 0.9.0
>            Reporter: Ashutosh Chauhan
>
> To begin with  org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert2_overwrite_partitions
and org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl5 are failing on trunk
for a while now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message