hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashish Thusoo (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-151) HiveQL Query execution bug: java.lang.NullPointerException
Date Sat, 27 Dec 2008 00:25:44 GMT

    [ https://issues.apache.org/jira/browse/HIVE-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659291#action_12659291
] 

Ashish Thusoo commented on HIVE-151:
------------------------------------

explain <query>;

shows that the fetch task (Stage-0) which should be dependent on Stage-3 is actually a root
task. So I think what is happening is that task is trying to read from the output of Stage-3
which has not even started yet!! This looks like a bad bug...

The output of the explain is as follows:

hive> explain SELECT t11.subject, t22.object , t33.subject , t66.object FROM ( SELECT t1.subject
FROM triples t1 WHERE t1.pre
dicate='http://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL' AND t1.object='http://ontos/OntosMiner/Common.English/
ontology#Citation' ) t11 JOIN ( SELECT t2.subject , t2.object FROM triples t2 WHERE t2.predicate='http://sofa.semanticweb.org
/sofa/v1.0/system#__LABEL_REL' ) t22 ON (t11.subject=t22.subject) JOIN ( SELECT t3.subject
, t3.object FROM triples t3 WHERE 
t3.predicate='http://www.ontosearch.com/2007/12/ontosofa-ns#_from' ) t33 ON (t11.subject=t33.object)
JOIN ( SELECT t4.subject
 FROM triples t4 WHERE t4.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL'
AND t4.object='http://ont
os/OntosMiner/Common.English/ontology#Author' ) t44 ON (t44.subject=t33.subject) JOIN ( SELECT
t5.subject, t5.object as obh F
ROM triples t5 WHERE t5.predicate='http://www.ontosearch.com/2007/12/ontosofa-ns#_to' ) t55
ON (t55.subject=t44.subject) JOIN
 ( SELECT t6.subject, t6.object FROM triples t6 WHERE t6.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__LABEL_REL'
 ) t66 ON (t66.subject=t55.obh);
OK
ABSTRACT SYNTAX TREE:
  (TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_JOIN (TOK_JOIN (TOK_JOIN (TOK_JOIN (TOK_SUBQUERY (TOK_QUERY
(TOK_FROM (TOK_TABREF tripl
es t1)) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_COLREF
t1 subject))) (TOK_WHERE (A
ND (= (TOK_COLREF t1 predicate) 'http://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL')
(= (TOK_COLREF t1 object) 'h
ttp://ontos/OntosMiner/Common.English/ontology#Citation'))))) t11) (TOK_SUBQUERY (TOK_QUERY
(TOK_FROM (TOK_TABREF triples t2)
) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_COLREF
t2 subject)) (TOK_SELEXPR (TOK_CO
LREF t2 object))) (TOK_WHERE (= (TOK_COLREF t2 predicate) 'http://sofa.semanticweb.org/sofa/v1.0/system#__LABEL_REL'))))
t22)
 (= (TOK_COLREF t11 subject) (TOK_COLREF t22 subject))) (TOK_SUBQUERY (TOK_QUERY (TOK_FROM
(TOK_TABREF triples t3)) (TOK_INSE
RT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_COLREF t3 subject))
(TOK_SELEXPR (TOK_COLREF t3 obj
ect))) (TOK_WHERE (= (TOK_COLREF t3 predicate) 'http://www.ontosearch.com/2007/12/ontosofa-ns#_from'))))
t33) (= (TOK_COLREF 
t11 subject) (TOK_COLREF t33 object))) (TOK_SUBQUERY (TOK_QUERY (TOK_FROM (TOK_TABREF triples
t4)) (TOK_INSERT (TOK_DESTINATI
ON (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_COLREF t4 subject))) (TOK_WHERE (AND
(= (TOK_COLREF t4 predicate) 'h
ttp://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL') (= (TOK_COLREF t4 object) 'http://ontos/OntosMiner/Common.Engl
ish/ontology#Author'))))) t44) (= (TOK_COLREF t44 subject) (TOK_COLREF t33 subject))) (TOK_SUBQUERY
(TOK_QUERY (TOK_FROM (TOK
_TABREF triples t5)) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR
(TOK_COLREF t5 subject)) (
TOK_SELEXPR (TOK_COLREF t5 object) obh)) (TOK_WHERE (= (TOK_COLREF t5 predicate) 'http://www.ontosearch.com/2007/12/ontosofa-
ns#_to')))) t55) (= (TOK_COLREF t55 subject) (TOK_COLREF t44 subject))) (TOK_SUBQUERY (TOK_QUERY
(TOK_FROM (TOK_TABREF triple
s t6)) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_COLREF
t6 subject)) (TOK_SELEXPR (T
OK_COLREF t6 object))) (TOK_WHERE (= (TOK_COLREF t6 predicate) 'http://sofa.semanticweb.org/sofa/v1.0/system#__LABEL_REL'))))
 t66) (= (TOK_COLREF t66 subject) (TOK_COLREF t55 obh)))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR
TOK_TMP_FILE)) (TOK_SELECT (T
OK_SELEXPR (TOK_COLREF t11 subject)) (TOK_SELEXPR (TOK_COLREF t22 object)) (TOK_SELEXPR (TOK_COLREF
t33 subject)) (TOK_SELEXP
R (TOK_COLREF t66 object)))))

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-2 is a root stage
  Stage-3 depends on stages: Stage-2
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Alias -> Map Operator Tree:
        t66:t6 
            Select Operator
              expressions:
                    expr: subject
                    type: string
                    expr: object
                    type: string
                    expr: predicate
                    type: string
              Filter Operator
                predicate:
                    expr: (2 = 'http://sofa.semanticweb.org/sofa/v1.0/system#__LABEL_REL')
                    type: boolean
                Select Operator
                  expressions:
                        expr: 0
                        type: string
                        expr: 1
                        type: string
                  Reduce Output Operator
                    key expressions:
                          expr: 0
                          type: string
                    sort order: +
                    Map-reduce partition columns:
                          expr: 0
                          type: string
                    tag: 1
                    value expressions:
                          expr: 0
                          type: string
                          expr: 1
                          type: string
      Reduce Operator Tree:
        Join Operator
          condition map:
               Inner Join 0 to 1
          condition expressions:
            0 {VALUE.0} {VALUE.1} {VALUE.2} {VALUE.3} {VALUE.4} {VALUE.5} {VALUE.6} {VALUE.7}
            1 {VALUE.0} {VALUE.1}
          Select Operator
            expressions:
                  expr: 3
                  type: string
                  expr: 5
                  type: string
                  expr: 0
                  type: string
                  expr: 9
                  type: string
            File Output Operator
              compressed: true
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat

  Stage: Stage-2
    Map Reduce
      Alias -> Map Operator Tree:
        t22:t2 
            Select Operator
              expressions:
                    expr: subject
                    type: string
                    expr: object
                    type: string
                    expr: predicate
                    type: string
              Filter Operator
                predicate:
                    expr: (2 = 'http://sofa.semanticweb.org/sofa/v1.0/system#__LABEL_REL')
                    type: boolean
                Select Operator
                  expressions:
                        expr: 0
                        type: string
                        expr: 1
                        type: string
                  Reduce Output Operator
                    key expressions:
                          expr: 0
                          type: string
                    sort order: +
                    Map-reduce partition columns:
                          expr: 0
                          type: string
                    tag: 1
                    value expressions:
                          expr: 0
                          type: string
                          expr: 1
                          type: string
        t33:t3 
            Select Operator
              expressions:
                    expr: subject
                    type: string
                    expr: object
                    type: string
                    expr: predicate
                    type: string
              Filter Operator
                predicate:
                    expr: (2 = 'http://www.ontosearch.com/2007/12/ontosofa-ns#_from')
                    type: boolean
                Select Operator
                  expressions:
                        expr: 0
                        type: string
                        expr: 1
                        type: string
                  Reduce Output Operator
                    key expressions:
                          expr: 1
                          type: string
                    sort order: +
                    Map-reduce partition columns:
                          expr: 1
                          type: string
                    tag: 2
                    value expressions:
                          expr: 0
                          type: string
                          expr: 1
                          type: string
        t11:t1 
            Select Operator
              expressions:
                    expr: subject
                    type: string
                    expr: predicate
                    type: string
                    expr: object
                    type: string
              Filter Operator
                predicate:
                    expr: ((1 = 'http://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL')
and (2 = 'http://ontos/Ontos
Miner/Common.English/ontology#Citation'))
                    type: boolean
                Select Operator
                  expressions:
                        expr: 0
                        type: string
                  Reduce Output Operator
                    key expressions:
                          expr: 0
                          type: string
                    sort order: +
                    Map-reduce partition columns:
                          expr: 0
                          type: string
                    tag: 0
                    value expressions:
                          expr: 0
                          type: string
      Reduce Operator Tree:
        Join Operator
          condition map:
               Inner Join 0 to 1
               Inner Join 0 to 1
          condition expressions:
            0 {VALUE.0}
            1 {VALUE.0} {VALUE.1}
            2 {VALUE.0} {VALUE.1}
          File Output Operator
            compressed: true
            table:
                input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                output format: org.apache.hadoop.mapred.SequenceFileOutputFormat
                name: binary_table

  Stage: Stage-3
    Map Reduce
      Alias -> Map Operator Tree:
        $INTNAME 
          Reduce Output Operator
            key expressions:
                  expr: 3
                  type: string
            sort order: +
            Map-reduce partition columns:
                  expr: 3
                  type: string
            tag: 0
            value expressions:
                  expr: 3
                  type: string
                  expr: 4
                  type: string
                  expr: 0
                  type: string
                  expr: 1
                  type: string
                  expr: 2
                  type: string
        t55:t5 
            Select Operator
              expressions:
                    expr: subject
                    type: string
                    expr: object
                    type: string
                    expr: predicate
                    type: string
              Filter Operator
                predicate:
                    expr: (2 = 'http://www.ontosearch.com/2007/12/ontosofa-ns#_to')
                    type: boolean
                Select Operator
                  expressions:
                        expr: 0
                        type: string
                        expr: 1
                        type: string
                  Reduce Output Operator
                    key expressions:
                          expr: 0
                          type: string
                    sort order: +
                    Map-reduce partition columns:
                          expr: 0
                          type: string
                    tag: 2
                    value expressions:
                          expr: 0
                          type: string
                          expr: 1
                          type: string
        t44:t4 
            Select Operator
              expressions:
                    expr: subject
                    type: string
                    expr: predicate
                    type: string
                    expr: object
                    type: string
              Filter Operator
                predicate:
                    expr: ((1 = 'http://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL')
and (2 = 'http://ontos/Ontos
Miner/Common.English/ontology#Author'))
                    type: boolean
                Select Operator
                  expressions:
                        expr: 0
                        type: string
                  Reduce Output Operator
                    key expressions:
                          expr: 0
                          type: string
                    sort order: +
                    Map-reduce partition columns:
                          expr: 0
                          type: string
                    tag: 1
                    value expressions:
                          expr: 0
                          type: string
      Reduce Operator Tree:
        Join Operator
          condition map:
               Inner Join 0 to 1
               Inner Join 1 to 1
          condition expressions:
            0 {VALUE.0} {VALUE.1} {VALUE.2} {VALUE.3} {VALUE.4}
            1 {VALUE.0}
            2 {VALUE.0} {VALUE.1}
          Reduce Output Operator
            key expressions:
                  expr: 7
                  type: string
            sort order: +
            Map-reduce partition columns:
                  expr: 7
                  type: string
            tag: 0
            value expressions:
                  expr: 0
                  type: string
                  expr: 1
                  type: string
                  expr: 5
                  type: string
                  expr: 2
                  type: string
                  expr: 3
                  type: string
                  expr: 4
                  type: string
                  expr: 6
                  type: string
                  expr: 7
                  type: string

  Stage: Stage-0
    Fetch Operator
      limit: -1


Time taken: 0.69 seconds


> HiveQL Query execution bug: java.lang.NullPointerException
> ----------------------------------------------------------
>
>                 Key: HIVE-151
>                 URL: https://issues.apache.org/jira/browse/HIVE-151
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Query Processor
>         Environment: Ubuntu Linux 386, Hadoop 0.19.0 Hive trunk
>            Reporter: Viacheslav Seledkin
>             Fix For: 0.2.0
>
>         Attachments: _user_hive_warehouse_triples_part-00000.tar.gz
>
>
> Executing a query 
> ------------------------------------- query start ----------------------------------------------------
> SELECT t11.subject, t22.object , t33.subject , t55.object, t66.object 
> FROM 
> ( 
> 	SELECT t1.subject 
> 	FROM triples t1  
> 	WHERE 
> 		t1.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL'  
> 	AND  
> 		t1.object='http://ontos/OntosMiner/Common.English/ontology#Citation' 
> ) t11  
> JOIN  
> ( 
> 	SELECT t2.subject , t2.object 
> 	FROM triples t2  
> 	WHERE  
> 			t2.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__LABEL_REL' 
> ) t22 
> ON (t11.subject=t22.subject)  		
> JOIN  
> ( 
> 	SELECT t3.subject , t3.object  
> 	FROM triples t3  
> 	WHERE  
> 			t3.predicate='http://www.ontosearch.com/2007/12/ontosofa-ns#_from'  
> 				
> ) t33
> ON (t11.subject=t33.object) 
> JOIN  
> ( 
> 	SELECT t4.subject  
> 	FROM triples t4  
> 	WHERE  
> 		t4.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL'  
> 	AND  
> 		t4.object='http://ontos/OntosMiner/Common.English/ontology#Author' 
> 				
> ) t44
> ON (t44.subject=t33.subject) 
> JOIN  
> ( 
> 	SELECT t5.subject, t5.object
> 	FROM triples t5  
> 	WHERE  
> 		t5.predicate='http://www.ontosearch.com/2007/12/ontosofa-ns#_to'  
> ) t55
> ON (t55.subject=t44.subject) 
> JOIN  
> ( 
> 	SELECT t6.subject, t6.object
> 	FROM triples t6  
> 	WHERE  
> 		t6.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__LABEL_REL'  
> ) t66
> ON (t66.subject=t55.object)
> ------------------------------------- query end ----------------------------------------------------
> on table 
> ------------------------------------- table start ----------------------------------------------------
> CREATE TABLE triples (foo string,subject string, predicate string, object string, foo2
string)
> ------------------------------------- table end -----------------------------------------------------
> gives the foolowing output 
> ------------------------------------ console output ----------------------------------------------
>  INFO [main] (Driver.java:156) - Starting command: SELECT t11.subject, t22.object , t33.subject
, t66.object  FROM  (  	SELECT t1.subject  	FROM triples t1   	WHERE  		t1.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL'
  	AND   		t1.object='http://ontos/OntosMiner/Common.English/ontology#Citation'  ) t11   JOIN
  (  	SELECT t2.subject , t2.object  	FROM triples t2   	WHERE   			t2.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__LABEL_REL'
 ) t22  ON (t11.subject=t22.subject)  		 JOIN   (  	SELECT t3.subject , t3.object   	FROM
triples t3   	WHERE   			t3.predicate='http://www.ontosearch.com/2007/12/ontosofa-ns#_from'
  				 ) t33 ON (t11.subject=t33.object)  JOIN   (  	SELECT t4.subject   	FROM triples t4
  	WHERE   		t4.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL'
  	AND   		t4.object='http://ontos/OntosMiner/Common.English/ontology#Author'  				 ) t44
ON (t44.subject=t33.subject)  JOIN   (  	SELECT t5.subject, t5.object as obh 	FROM triples
t5   	WHERE   		t5.predicate='http://www.ontosearch.com/2007/12/ontosofa-ns#_to'   ) t55 ON
(t55.subject=t44.subject)  JOIN   (  	SELECT t6.subject, t6.object 	FROM triples t6   	WHERE
  		t6.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__LABEL_REL'   ) t66 ON (t66.subject=t55.obh)
>  INFO [main] (ParseDriver.java:249) - Parsing command: SELECT t11.subject, t22.object
, t33.subject , t66.object  FROM  (  	SELECT t1.subject  	FROM triples t1   	WHERE  		t1.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL'
  	AND   		t1.object='http://ontos/OntosMiner/Common.English/ontology#Citation'  ) t11   JOIN
  (  	SELECT t2.subject , t2.object  	FROM triples t2   	WHERE   			t2.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__LABEL_REL'
 ) t22  ON (t11.subject=t22.subject)  		 JOIN   (  	SELECT t3.subject , t3.object   	FROM
triples t3   	WHERE   			t3.predicate='http://www.ontosearch.com/2007/12/ontosofa-ns#_from'
  				 ) t33 ON (t11.subject=t33.object)  JOIN   (  	SELECT t4.subject   	FROM triples t4
  	WHERE   		t4.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__INSTANCEOF_REL'
  	AND   		t4.object='http://ontos/OntosMiner/Common.English/ontology#Author'  				 ) t44
ON (t44.subject=t33.subject)  JOIN   (  	SELECT t5.subject, t5.object as obh 	FROM triples
t5   	WHERE   		t5.predicate='http://www.ontosearch.com/2007/12/ontosofa-ns#_to'   ) t55 ON
(t55.subject=t44.subject)  JOIN   (  	SELECT t6.subject, t6.object 	FROM triples t6   	WHERE
  		t6.predicate='http://sofa.semanticweb.org/sofa/v1.0/system#__LABEL_REL'   ) t66 ON (t66.subject=t55.obh)
>  INFO [main] (ParseDriver.java:263) - Parse Completed
>  INFO [main] (HiveMetaStore.java:126) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>  INFO [main] (ObjectStore.java:124) - ObjectStore, initialize called
>  INFO [main] (ObjectStore.java:146) - found resource jpox.properties at file:/home/vseledkin/workspace/HiveDrv/bin/jpox.properties
>  WARN [main] (Log4JLogger.java:98) - Bundle "org.jpox" has an optional dependency to
"org.eclipse.equinox.registry" but it cannot be resolved
>  WARN [main] (Log4JLogger.java:98) - Bundle "org.jpox" has an optional dependency to
"org.eclipse.core.runtime" but it cannot be resolved
>  INFO [main] (Log4JLogger.java:79) - ================= Persistence Configuration ===============
>  INFO [main] (Log4JLogger.java:79) - JPOX Persistence Factory - Vendor: "JPOX"  Version:
"1.2.2"
>  INFO [main] (Log4JLogger.java:79) - JPOX Persistence Factory initialised for datastore
URL="jdbc:derby:;databaseName=metastore_db;create=true" driver="org.apache.derby.jdbc.EmbeddedDriver"
userName="APP"
>  INFO [main] (Log4JLogger.java:79) - ===========================================================
>  INFO [main] (Log4JLogger.java:79) - Initialising Catalog "", Schema "APP" using "SchemaTable"
auto-start option
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of org.apache.hadoop.hive.metastore.model.MDatabase
since it was managed previously
>  INFO [main] (Log4JLogger.java:79) - No manager for annotations was found in the CLASSPATH
so all annotations are ignored.
>  WARN [main] (Log4JLogger.java:98) - MetaData Parser encountered an error in file "jar:file:/home/vseledkin/workspace/hive/build/hive_metastore.jar!/package.jdo"
at line 282, column 13 : The content of element type "class" must match "(extension*,implements*,datastore-identity?,primary-key?,inheritance?,version?,join*,foreign-key*,index*,unique*,column*,field*,property*,query*,fetch-group*,extension*)".
- Please check your specification of DTD and the validity of the MetaData XML that you have
specified.
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of org.apache.hadoop.hive.metastore.model.MStorageDescriptor
since it was managed previously
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of org.apache.hadoop.hive.metastore.model.MSerDeInfo
since it was managed previously
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of org.apache.hadoop.hive.metastore.model.MTable
since it was managed previously
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of org.apache.hadoop.hive.metastore.model.MPartition
since it was managed previously
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Class : org.apache.hadoop.hive.metastore.model.MDatabase
[Table : DBS, InheritanceStrategy : new-table]
>  INFO [main] (Log4JLogger.java:79) - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
is tagged as "embedded-only" so does not have its own datastore table.
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Class : org.apache.hadoop.hive.metastore.model.MSerDeInfo
[Table : SERDES, InheritanceStrategy : new-table]
>  INFO [main] (Log4JLogger.java:79) - The class "org.apache.hadoop.hive.metastore.model.MOrder"
is tagged as "embedded-only" so does not have its own datastore table.
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Class : org.apache.hadoop.hive.metastore.model.MStorageDescriptor
[Table : SDS, InheritanceStrategy : new-table]
>  INFO [main] (Log4JLogger.java:79) - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
is tagged as "embedded-only" so does not have its own datastore table.
>  INFO [main] (Log4JLogger.java:79) - The class "org.apache.hadoop.hive.metastore.model.MOrder"
is tagged as "embedded-only" so does not have its own datastore table.
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Class : org.apache.hadoop.hive.metastore.model.MTable
[Table : TBLS, InheritanceStrategy : new-table]
>  INFO [main] (Log4JLogger.java:79) - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
is tagged as "embedded-only" so does not have its own datastore table.
>  INFO [main] (Log4JLogger.java:79) - The class "org.apache.hadoop.hive.metastore.model.MOrder"
is tagged as "embedded-only" so does not have its own datastore table.
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Class : org.apache.hadoop.hive.metastore.model.MPartition
[Table : PARTITIONS, InheritanceStrategy : new-table]
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MSerDeInfo.parameters
[Table : SERDE_PARAMS]
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MPartition.parameters
[Table : PARTITION_PARAMS]
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MPartition.values
[Table : PARTITION_KEY_VALS]
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MTable.parameters
[Table : TABLE_PARAMS]
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MTable.partitionKeys
[Table : PARTITION_KEYS]
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MStorageDescriptor.bucketCols
[Table : BUCKETING_COLS]
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MStorageDescriptor.cols
[Table : COLUMNS]
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MStorageDescriptor.parameters
[Table : SD_PARAMS]
>  INFO [main] (Log4JLogger.java:79) - Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MStorageDescriptor.sortCols
[Table : SORT_COLS]
>  INFO [main] (Log4JLogger.java:79) - Validating 1 unique key(s) for table SERDES
>  INFO [main] (Log4JLogger.java:79) - Validating 0 foreign key(s) for table SERDES
>  INFO [main] (Log4JLogger.java:79) - Validating 1 index(es) for table SERDES
>  INFO [main] (Log4JLogger.java:79) - Validating 2 unique key(s) for table PARTITIONS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 foreign key(s) for table PARTITIONS
>  INFO [main] (Log4JLogger.java:79) - Validating 4 index(es) for table PARTITIONS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 unique key(s) for table TBLS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 foreign key(s) for table TBLS
>  INFO [main] (Log4JLogger.java:79) - Validating 4 index(es) for table TBLS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 unique key(s) for table SDS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 foreign key(s) for table SDS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 index(es) for table SDS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 unique key(s) for table DBS
>  INFO [main] (Log4JLogger.java:79) - Validating 0 foreign key(s) for table DBS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 index(es) for table DBS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 unique key(s) for table SORT_COLS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 foreign key(s) for table SORT_COLS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 index(es) for table SORT_COLS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 unique key(s) for table TABLE_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 foreign key(s) for table TABLE_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 index(es) for table TABLE_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 unique key(s) for table COLUMNS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 foreign key(s) for table COLUMNS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 index(es) for table COLUMNS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 unique key(s) for table PARTITION_KEYS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 foreign key(s) for table PARTITION_KEYS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 index(es) for table PARTITION_KEYS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 unique key(s) for table SD_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 foreign key(s) for table SD_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 index(es) for table SD_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 unique key(s) for table PARTITION_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 foreign key(s) for table PARTITION_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 index(es) for table PARTITION_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 unique key(s) for table PARTITION_KEY_VALS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 foreign key(s) for table PARTITION_KEY_VALS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 index(es) for table PARTITION_KEY_VALS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 unique key(s) for table SERDE_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 foreign key(s) for table SERDE_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 index(es) for table SERDE_PARAMS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 unique key(s) for table BUCKETING_COLS
>  INFO [main] (Log4JLogger.java:79) - Validating 1 foreign key(s) for table BUCKETING_COLS
>  INFO [main] (Log4JLogger.java:79) - Validating 2 index(es) for table BUCKETING_COLS
>  INFO [main] (Log4JLogger.java:79) - Catalog "", Schema "APP" initialised - managing
14 classes
>  INFO [main] (Log4JLogger.java:79) - >> Found StoreManager org.jpox.store.rdbms.RDBMSManager
>  INFO [main] (ObjectStore.java:110) - Initialized ObjectStore
>  INFO [main] (SemanticAnalyzer.java:3086) - Starting Semantic Analysis
>  INFO [main] (SemanticAnalyzer.java:3088) - Completed phase 1 of Semantic Analysis
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:3091) - Completed getting MetaData in Semantic Analysis
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1, string reducesinkvalue2, string reducesinkvalue3, string reducesinkvalue4}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1, string reducesinkvalue2, string reducesinkvalue3, string reducesinkvalue4,
string reducesinkvalue5, string reducesinkvalue6, string reducesinkvalue7}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:579) - Get metadata for source tables
>  INFO [main] (HiveMetaStore.java:164) - 0: get_table : db=default tbl=triples
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct triples { string tid, string subject,
string predicate, string object, string type}
>  INFO [main] (SemanticAnalyzer.java:595) - Get metadata for subqueries
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (SemanticAnalyzer.java:602) - Get metadata for destination tables
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1, string reducesinkvalue2, string reducesinkvalue3, string reducesinkvalue4}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1, string reducesinkvalue2, string reducesinkvalue3, string reducesinkvalue4,
string reducesinkvalue5, string reducesinkvalue6, string reducesinkvalue7}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string reducesinkkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string reducesinkvalue0,
string reducesinkvalue1}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_sortable_table { string joinkey0}
>  INFO [main] (SemanticAnalyzer.java:3107) - Completed partition pruning
>  INFO [main] (SemanticAnalyzer.java:3111) - Completed sample pruning
>  INFO [main] (MetaStoreUtils.java:461) - DDL: struct binary_table { string temporarycol0,
string temporarycol1, string temporarycol2, string temporarycol3, string temporarycol4}
>  INFO [main] (SemanticAnalyzer.java:3120) - Completed plan generation
>  INFO [main] (Driver.java:173) - Semantic Analysis Completed
> Total MapReduce jobs = 3
>  INFO [main] (SessionState.java:254) - Total MapReduce jobs = 3
> Number of reducers = 1
>  INFO [main] (SessionState.java:254) - Number of reducers = 1
> In order to change numer of reducers use:
>  INFO [main] (SessionState.java:254) - In order to change numer of reducers use:
>   set mapred.reduce.tasks = <number>
>  INFO [main] (SessionState.java:254) -   set mapred.reduce.tasks = <number>
>  WARN [main] (ExecDriver.java:109) - Number of reduce tasks not specified. Defaulting
to jobconf value of: 1
>  INFO [main] (ExecDriver.java:238) - Adding input file /user/hive/warehouse/triples
>  WARN [main] (JobClient.java:547) - Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
>  INFO [main] (FileInputFormat.java:181) - Total input paths to process : 1
> Starting Job = job_200812091129_0144, Tracking URL = http://ubunder.avicomp.com:50030/jobdetails.jsp?jobid=job_200812091129_0144
>  INFO [main] (SessionState.java:254) - Starting Job = job_200812091129_0144, Tracking
URL = http://ubunder.avicomp.com:50030/jobdetails.jsp?jobid=job_200812091129_0144
> Kill Command = /home/vseledkin/workspace/HiveDrv/programs/hadoop-0.19.0 job  -Dmapred.job.tracker=ubunder.avicomp.com:9001
-kill job_200812091129_0144
>  INFO [main] (SessionState.java:254) - Kill Command = /home/vseledkin/workspace/HiveDrv/programs/hadoop-0.19.0
job  -Dmapred.job.tracker=ubunder.avicomp.com:9001 -kill job_200812091129_0144
>  map = 0%,  reduce =0%
>  INFO [main] (SessionState.java:254) -  map = 0%,  reduce =0%
>  map = 50%,  reduce =0%
>  INFO [main] (SessionState.java:254) -  map = 50%,  reduce =0%
>  map = 100%,  reduce =0%
>  INFO [main] (SessionState.java:254) -  map = 100%,  reduce =0%
>  map = 100%,  reduce =100%
>  INFO [main] (SessionState.java:254) -  map = 100%,  reduce =100%
> ERROR [main] (SessionState.java:263) - Ended Job = job_200812091129_0144 with errors
> Ended Job = job_200812091129_0144 with errors
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver
> ERROR [main] (SessionState.java:263) - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver
> ------------------------------------ console output end ----------------------------------------
> and the stack trace in hadoop logs 
> ------------------------------------ stack trace ---------------------------------------------------
> java.lang.NullPointerException
> 	at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:81)
> 	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> 	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:337)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:155)
> ------------------------------------ stack trace end ---------------------------------------------
> attached file contains table data to test problematic query

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message