hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pradeep Kamath <prade...@yahoo-inc.com>
Subject Re: alter table add partition error
Date Tue, 22 Jun 2010 17:21:53 GMT
Hi Paul,
   I have attached a draft of a patch (incomplete). The problem I am 
facing is that to make this change I seem to need to change the thrift 
idl (hive_metastore.thrift). Looking at the code structure it seems like 
the generate java files are checked in rather than generated through the 
build (is this so that the build does not depend on the thrift 
compiler?). I tried downloading and building thrift but hit issues in 
the make install - wanted to check with you if the patch looks correct 
so far or if I am on the wrong path. Also if the direction is right, I 
need some guidance on generating the java files from the modified idl.

Thanks,
Pradeep

Paul Yang wrote:
>
> Well, the section of code you are referring to is automatically 
> generated by the thrift compiler so we can't modify it. As the 
> inability to return null is a limitation of Thrift, we can work around 
> that issue by throwing an exception instead.
>
>  
>
> Check out this thread:
>
> http://publists.facebook.com/pipermail/thrift/2007-September/000085.html
>
>  
>
>  
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Monday, June 21, 2010 5:46 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>  
>
> Hi Paul,
>
>   Yes I don't mind working on a patch though I have no knowledge of 
> how thrift code works. I was also confused by your suggestion for a 
> fix. I thought the fix would be in 
> ThriftHiveMetaStore.Client.recv_get_partition() -- to first check 
> if(result.o1 != null) throw result.o1, then to just return 
> result.success and not throw the "unknown result" exception. I tried 
> something along these lines and the alter table worked. Am I missing 
> something?
>
>  
>
> Thanks,
>
> Pradeep
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* Paul Yang [mailto:pyang@facebook.com]
> *Sent:* Monday, June 21, 2010 2:35 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>  
>
> Ah okay. So what's going on is that getPartition() is supposed to 
> return null if the specified partition doesn't exist. But because 
> Thrift can't handle null return values, we are seeing this exception. 
> This isn't a problem with a local metastore because Thrift isn't used.
>
>  
>
> One solution to this problem would be modify 
> ObjectStore.getPartition() to throw a NoSuchObjectException instead of 
> returning null for non-existent partitions. Then Hive.getPartition() 
> can catch the exception and return null to retain the original behavior.
>
>  
>
> Pradeep, are you interested in working on a patch for this problem?
>
>  
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Monday, June 21, 2010 2:11 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>  
>
> I tried debugging in code a little more. Here is what I found:
>
> The code in ThriftHiveMetaStore eventually makes a call -- 
> get_partition() passing the partition key values for the partition I 
> am trying to add using alter table. I assume this is to check that the 
> partition doesn't already exist.
>
> I added a debug line in the following code:
>
>   public Partition recv_get_partition() throws MetaException, TException
>
>     {
>
>       TMessage msg = iprot_.readMessageBegin();
>
>       if (msg.type == TMessageType.EXCEPTION) {
>
>         TApplicationException x = TApplicationException.read(iprot_);
>
>         iprot_.readMessageEnd();
>
>         throw x;
>
>       }
>
>       get_partition_result result = new get_partition_result();
>
>      
>
>       result.read(iprot_);
>
>       *System.err.println("XXX: result:" + result);*
>
>       iprot_.readMessageEnd();
>
>       if (result.isSetSuccess()) {
>
>         return result.success;
>
>       }
>
>       if (result.o1 != null) {
>
>         throw result.o1;
>
>       }
>
> *      throw new 
> TApplicationException(TApplicationException.MISSING_RESULT, 
> "get_partition failed: unknown result");*
>
>     }
>
>  
>
> I also put a debug statements in the read() method:
>
>     public void read(TProtocol iprot) throws TException {
>
>       TField field;
>
>       iprot.readStructBegin();
>
> *      System.err.println("XXX: Reading TProtocol object:");*
>
>       while (true)
>
>       {
>
>         field = iprot.readFieldBegin();
>
> *        System.err.println("XXX: field just read:" + field);*
>
>         if (field.type == TType.STOP) {
>
>           break;
>
>  
>
> *I got *
>
> *XXX: Reading TProtocol object:*
>
> *XXX: field just read:<TField name:'' type:0 field-id:0>*
>
> *XXX: result:get_partition_result(success:null, o1:null)*
>
>  
>
> The field read in the thrift response message is of type "STOP" and 
> with id of type SUCCESS. This seems right since there are no existing 
> partitions. But the way the rest of the code handles this, results in 
> the exception.
>
>  
>
> Any pointers?
>
>  
>
> TIA,
>
> Pradeep
>
> ------------------------------------------------------------------------
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Friday, June 18, 2010 2:51 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>  
>
> *Looks like the standalone script works fine against the existing 
> partition:*
>
> *./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name 
> default dummy datestamp=20100602/srcid=100/action=view/testid=10*
>
> Partition(parameters={'transient_lastDdlTime': '1276881277'}, 
> tableName='dummy', createTime=1276881277, lastAccessTime=0, 
> values=['20100602', '100', 'view', '10'], dbName='default', 
> sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat',

> sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat', 
> cols=[FieldSchema(comment=None, type='string', name='partition_name'), 
> FieldSchema(comment=None, type='int', name='partition_id')], 
> compressed=False, bucketCols=[], numBuckets=-1, parameters={}, 
> serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe',

> name=None, parameters={'serialization.format': '1'}), 
> location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))
>
> [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
>
>  
>
> *However when I tried to add another partition with the hive cli using 
> thrift:*
>
> *hive  -e "ALTER TABLE dummy add partition(datestamp = '20100602', 
> srcid = '100',action='click',testid='10') location 
> '/user/pradeepk/dummy/20100602/100/click/10';"*
>
> 10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml 
> found in the classpath. Usage of hadoop-site.xml is deprecated. 
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to 
> override properties of core-default.xml, mapred-default.xml and 
> hdfs-default.xml respectively
>
> Hive history 
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException: 
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
>  
>
> *tail -30 /tmp/pradeepk/hive.log*
>
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>         at java.lang.reflect.Method.invoke(Method.java:597)
>
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
>  
>
> 2010-06-18 14:49:14,124 ERROR exec.DDLTask 
> (SessionState.java:printError(277)) - FAILED: Error in metadata: 
> org.apache.thrift.TApplicationException: get_partition failed: unknown 
> result
>
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.thrift.TApplicationException: get_partition failed: unknown 
> result
>
>         at 
> org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
>
>         at 
> org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
>
>         at 
> org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
>
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>
>         at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>
>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
>
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
>
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
>
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>         at java.lang.reflect.Method.invoke(Method.java:597)
>
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Caused by: org.apache.thrift.TApplicationException: get_partition 
> failed: unknown result
>
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
>
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
>
>         at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
>
>         at 
> org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
>
>         ... 15 more
>
>  
>
> 2010-06-18 14:49:14,124 ERROR ql.Driver 
> (SessionState.java:printError(277)) - FAILED: Execution Error, return 
> code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* Paul Yang [mailto:pyang@facebook.com]
> *Sent:* Friday, June 18, 2010 2:19 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>  
>
> Looks like the thrift python libraries aren't in your path. run:
>
>  
>
> export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
>
>  
>
> before trying the remote command
>
>  
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Friday, June 18, 2010 1:38 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>  
>
> Sorry, don't know python -- so couldn't make sense out of the 
> following error when I run the command you suggested:
>
> [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote 
> -h localhost:9080 get_partition_by_name default dummy 
> datestamp=20100602/srcid=100/action=view/testid=10
>
> Traceback (most recent call last):
>
>   File "./ThriftHiveMetastore-remote", line 11, in ?
>
>     from thrift.transport import TTransport
>
> ImportError: No module named thrift.transport
>
>  
>
> Thanks,
>
> Pradeep
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* Paul Yang [mailto:pyang@facebook.com]
> *Sent:* Friday, June 18, 2010 12:10 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>  
>
> Hi Pradeep,
>
>  
>
> In trunk/metastore/src/gen-py/hive_metastore/, there is a script 
> called ThriftHiveMetastore-remote that can be used to test out the 
> thrift server independent of the CLI. As a quick test to narrow down 
> the problem, after the partition is created, can you try running
>
>  
>
> ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name 
> default dummy datestamp=20100602/srcid=100/action=view/testid=10
>
>  
>
> Cheers,
>
> Paul
>
>  
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Friday, June 18, 2010 11:26 AM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>  
>
> Hi Ning,
>
>  I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH 
> from an "env" command printed just before the following line in 
> execHiveCmd.sh
>
> exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
>
>  
>
> There is only one occurrence of libthrift.jar whose cksum matches the 
> one from lib/libthrift.jar from a hive checkout build.
>
>  
>
> The hive_config_without_thrift is a conf directory containing a 
> hive-site.xml with jdo settings to connect to the database directly. 
> By default I have hive set up to use a hive-site.xml [2] which 
> connects to a thrift server running on localhost.
>
>  
>
> Any pointers appreciated!
>
>  
>
> Thanks,
>
> Pradeep
>
>  
>
> [1]
>
> *HADOOP_CLASSPATH=/*home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar*:/home/y/libexec/hive_client/lib/libthrift.jar*:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
>
>  
>
> *CLASSPATH=/*home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar*:/home/y/libexec/hive_client/lib/libthrift.jar*:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
>
>  
>
> [2]
>
> *cat /home/y/libexec/hive_client/conf/hive-site.xml *
>
> <?xml version="1.0"?>
>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
>  
>
> <configuration>
>
>  
>
> <!-- Hive Configuration can either be stored in this file or in the 
> hadoop configuration files  -->
>
> <!-- that are implied by Hadoop setup 
> variables.                                                -->
>
> <!-- Aside from Hadoop setup variables - this file is provided as a 
> convenience so that Hive    -->
>
> <!-- users do not have to edit hadoop configuration files (that may be 
> managed as a centralized -->
>
> <!-- 
> resource).                                                                          
      
> -->
>
>  
>
> <!-- Hive Execution Parameters -->
>
> <property>
>
> *  <name>hive.metastore.uris</name>*
>
> *  <value>thrift://localhost:9080</value>*
>
>   <description>host and port for the thrift metastore server</description>
>
> </property>
>
>  
>
> <property>
>
> *  <name>hive.metastore.local</name>*
>
> *  <value>false</value>*
>
>   <description>controls whether to connect to remove metastore server 
> or open a new metastore server in Hive Client JVM</description>
>
> </property>
>
>  
>
> <property>
>
>   <name>hive.metastore.warehouse.dir</name>
>
>   <value>/user/pradeepk/hive/warehouse</value>
>
>   <description>location of default database for the 
> warehouse</description>
>
> </property>
>
>  
>
>  
>
> </configuration>
>
> ------------------------------------------------------------------------
>
> *From:* Ning Zhang [mailto:nzhang@facebook.com]
> *Sent:* Friday, June 18, 2010 10:50 AM
> *To:* hive-user@hadoop.apache.org
> *Subject:* Re: alter table add partition error
>
>  
>
> Pradeep,
>
>  
>
> I ran the commands you provided and it succeeded with the expected 
> behavior. 
>
>  
>
> One possibility is that there are multiple versions of libthrift.jar 
> in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive 
> CLASSPATH so that no other libthrift.jar is there? What is in 
> hive_config_without_thrift? 
>
>  
>
> Thanks,
>
> Ning
>
>  
>
> On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
>
>  
>
> I think there are two separate issues here -- I want to open a jira 
> for the first one since I am now able to reproduce it even with text 
> format with builtin Serdes. Essentially this is a bug in the thrift 
> code (not sure if it is in the client or server) since the same alter 
> table statement works fine when the hive client does not use thrift. 
> Here are the details:
>
>  
>
> *cat create_dummy.sql*
>
> CREATE external TABLE if not exists dummy (
>
>  
>
>   partition_name string
>
>   ,partition_id int
>
> )
>
> PARTITIONED BY ( datestamp string, srcid string, action string, testid 
> string )
>
> row format delimited
>
> stored as textfile
>
> location '/user/pradeepk/dummy';
>
>  
>
> *hive -f create_dummy.sql*
>
> 10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml 
> found in the classpath. Usage of hadoop-site.xml is deprecated. 
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to 
> override properties of core-default.xml, mapred-default.xml and 
> hdfs-default.xml respectively
>
> Hive history 
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
>
> OK
>
> Time taken: 0.627 seconds
>
> * *
>
> *hive  -e "ALTER TABLE dummy add partition(datestamp = '20100602', 
> srcid = '100',action='view',testid='10') location 
> '/user/pradeepk/dummy/20100602/100/view/10';"*
>
> 10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml 
> found in the classpath. Usage of hadoop-site.xml is deprecated. 
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to 
> override properties of core-default.xml, mapred-default.xml and 
> hdfs-default.xml respectively
>
> Hive history 
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException: 
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
>  
>
> *hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add 
> partition(datestamp = '20100602', srcid = 
> '100',action='view',testid='10') location 
> '/user/pradeepk/dummy/20100602/100/view/10';"*
>
> 10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml 
> found in the classpath. Usage of hadoop-site.xml is deprecated. 
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to 
> override properties of core-default.xml, mapred-default.xml and 
> hdfs-default.xml respectively
>
> Hive history 
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
>
> *OK*
>
> Time taken: 5.849 seconds
>
>  
>
> Is there some thrift setting I am missing or is this a bug? -- If it 
> is the latter, I can open a jira with the above details.
>
>  
>
> Thanks,
>
> Pradeep
>
>  
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Thursday, June 17, 2010 1:25 PM
> *To:* hive-user@hadoop.apache.org <mailto:hive-user@hadoop.apache.org>
> *Subject:* RE: alter table add partition error
>
>  
>
> Here are the create table and alter table statements:
>
> CREATE external TABLE if not exists mytable (
>
>  
>
>   bc string
>
>   ,src_spaceid string
>
>   ,srcpvid string
>
>   ,dstpvid string
>
>   ,dst_spaceid string
>
>   ,page_params map<string, string>
>
>   ,clickinfo map<string, string>
>
>   ,viewinfo array<map<string, string>>
>
>  
>
> )
>
> PARTITIONED BY ( datestamp string, srcid string, action string, testid 
> string )
>
> row format serde 'com.yahoo.mySerde'
>
> stored as inputformat 
> 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 
> 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
>
> location '/user/pradeepk/mytable';
>
>  
>
> hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add 
> partition(datestamp = '20091101', srcid = 
> '19174',action='click',testid='NOTESTID') location 
> '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
>
>  
>
> I get the following error:
>
> Hive history 
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException: 
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
> If I don't use thrift and use a hive-site.xml to directly talk to the 
> db, the alter table seems to succeed:
>
> hive --auxpath ult-serde.jar *--config hive_conf_without_thrift* -e 
> "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = 
> '19174',action='click',testid='NOTESTID') location 
> '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
>
>  
>
> However I get errors when I try to run a query:
>
> [pradeepk@chargesize:~/dev]*hive --auxpath ult-serde.jar --config 
> hive_conf_without_thrift -e "select src_spaceid from  
> ult_search_austria_ult where datestamp='20091101';"*
>
> 10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml 
> found in the classpath. Usage of hadoop-site.xml is deprecated. 
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to 
> override properties of core-default.xml, mapred-default.xml and 
> hdfs-default.xml respectively
>
> Hive history 
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
>
> Total MapReduce jobs = 1
>
> Launching Job 1 out of 1
>
> Number of reduce tasks is set to 0 since there's no reduce operator
>
> java.lang.IllegalArgumentException: Can not create a Path from an 
> empty string
>
>         at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
>
>         at org.apache.hadoop.fs.Path.<init>(Path.java:90)
>
>         at org.apache.hadoop.fs.Path.<init>(Path.java:50)
>
>         at 
> org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
>
>         at 
> org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
>
>         at 
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
>
>         at 
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>
>         at 
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
>
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>
>         at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>
>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
>
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>
>  
>
> Any help is much appreciated.
>
>  
>
> Pradeep
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* Ashish Thusoo [mailto:athusoo@facebook.com]
> *Sent:* Thursday, June 17, 2010 11:15 AM
> *To:* hive-user@hadoop.apache.org <mailto:hive-user@hadoop.apache.org>
> *Subject:* RE: alter table add partition error
>
>  
>
> hmm... Can you send the exact command and also the create table 
> command for this table.
>
>  
>
> Ashish
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Thursday, June 17, 2010 9:09 AM
> *To:* hive-user@hadoop.apache.org <mailto:hive-user@hadoop.apache.org>
> *Subject:* RE: alter table add partition error
>
> Sorry -- that was a cut-paste error -- I don't have the action part -- 
> so I am specifying key-value pairs. Since what I am trying to do seems 
> like a basic operation, I am wondering if it's something to do with my 
> Serde -- unfortunately the error I see gives me no clue of what could 
> be wrong -- any help would be greatly appreciated!
>
>  
>
> Thanks,
>
> Pradeep
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* yq he [mailto:hhh.heyq@gmail.com]
> *Sent:* Wednesday, June 16, 2010 5:54 PM
> *To:* hive-user@hadoop.apache.org <mailto:hive-user@hadoop.apache.org>
> *Subject:* Re: alter table add partition error
>
>  
>
> Hi Pradeep,
>
>  
>
> partition definition need to be key-value pairs. partition key 
> `action` seems missed the value.
>
>  
>
> Thanks
>
> Yongqiang
>
> On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath 
> <pradeepk@yahoo-inc.com <mailto:pradeepk@yahoo-inc.com>> wrote:
>
> Hi,
>
>     I am trying to create an external table against already existing 
> data in sequencefile format. However I have written a custom Serde to 
> interpret the data. I am able to create the table fine but get the 
> exception shown in the session output below when I try to add 
> partition -- any help would be greatly appreciated.
>
>  
>
> Thanks,
>
> Pradeep
>
>  
>
> == session output ===
>
>  
>
> [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER 
> TABLE mytable add partition(datestamp = '20091101', srcid = 
> '10',action) location '/user/pradeepk/mytable/20091101/10';"
>
> 10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml 
> found in the classpath. Usage of hadoop-site.xml is deprecated. 
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to 
> override properties of core-default.xml, mapred-default.xml and 
> hdfs-default.xml respectively
>
> Hive history 
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException: 
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
> [pradeepk@chargesize:~/dev/howl]
>
>  
>
> == session output ===
>
>  
>
> /tmp/pradeepk/hive.log has:
>
> 2010-06-16 17:09:00,841 ERROR exec.DDLTask 
> (SessionState.java:printError(269)) - FAILED: Error in metadata: 
> org.apache.thrift.TApplicationException: get_partition failed: unknown 
> result
>
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.thrift.TApplicationException: get_partition failed: unknown 
> result
>
>     at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
>
>     at 
> org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
>
>     at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
>
>     at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>
>     at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>
>     at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
>
>     at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>
>     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
>
>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>
>     at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>
>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
>
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>     at java.lang.reflect.Method.invoke(Method.java:597)
>
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> *Caused by: org.apache.thrift.TApplicationException: get_partition 
> failed: unknown result*
>
> *    at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)*
>
> *    at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)*
>
> *    at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)*
>
> *    at 
> org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)*
>
> *    ... 15 more*
>
>  
>
> The thrift server messages are:
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : 
> db=default tbl=mytable
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store 
> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>
> 10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize 
> called
>
> 10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : 
> db=default tbl=mytable
>
>  
>
>  
>
>  
>
>  
>


Mime
View raw message