hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Yang <py...@facebook.com>
Subject RE: alter table add partition error
Date Mon, 21 Jun 2010 21:34:45 GMT
Ah okay. So what's going on is that getPartition() is supposed to return null if the specified
partition doesn't exist. But because Thrift can't handle null return values, we are seeing
this exception. This isn't a problem with a local metastore because Thrift isn't used.

One solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException
instead of returning null for non-existent partitions. Then Hive.getPartition() can catch
the exception and return null to retain the original behavior.

Pradeep, are you interested in working on a patch for this problem?

From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 2:11 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error

I tried debugging in code a little more. Here is what I found:
The code in ThriftHiveMetaStore eventually makes a call - get_partition() passing the partition
key values for the partition I am trying to add using alter table. I assume this is to check
that the partition doesn't already exist.
I added a debug line in the following code:
  public Partition recv_get_partition() throws MetaException, TException
    {
      TMessage msg = iprot_.readMessageBegin();
      if (msg.type == TMessageType.EXCEPTION) {
        TApplicationException x = TApplicationException.read(iprot_);
        iprot_.readMessageEnd();
        throw x;
      }
      get_partition_result result = new get_partition_result();

      result.read(iprot_);
      System.err.println("XXX: result:" + result);
      iprot_.readMessageEnd();
      if (result.isSetSuccess()) {
        return result.success;
      }
      if (result.o1 != null) {
        throw result.o1;
      }
      throw new TApplicationException(TApplicationException.MISSING_RESULT, "get_partition
failed: unknown result");
    }

I also put a debug statements in the read() method:
    public void read(TProtocol iprot) throws TException {
      TField field;
      iprot.readStructBegin();
      System.err.println("XXX: Reading TProtocol object:");
      while (true)
      {
        field = iprot.readFieldBegin();
        System.err.println("XXX: field just read:" + field);
        if (field.type == TType.STOP) {
          break;

I got
XXX: Reading TProtocol object:
XXX: field just read:<TField name:'' type:0 field-id:0>
XXX: result:get_partition_result(success:null, o1:null)

The field read in the thrift response message is of type "STOP" and with id of type SUCCESS.
This seems right since there are no existing partitions. But the way the rest of the code
handles this, results in the exception.

Any pointers?

TIA,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 2:51 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error

Looks like the standalone script works fine against the existing partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'}, tableName='dummy', createTime=1276881277,
lastAccessTime=0, values=['20100602', '100', 'view', '10'], dbName='default', sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat',
sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat', cols=[FieldSchema(comment=None,
type='string', name='partition_name'), FieldSchema(comment=None, type='int', name='partition_id')],
compressed=False, bucketCols=[], numBuckets=-1, parameters={}, serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe',
name=None, parameters={'serialization.format': '1'}), location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]

However when I tried to add another partition with the hive cli using thrift:
hive  -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='click',testid='10')
location '/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath.
Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml
to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed:
unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

tail -30 /tmp/pradeepk/hive.log
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

2010-06-18 14:49:14,124 ERROR exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error
in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
        at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
        at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
        at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
        ... 15 more

2010-06-18 14:49:14,124 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution
Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error

Looks like the thrift python libraries aren't in your path. run:

export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/

before trying the remote command

From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error

Sorry, don't know python - so couldn't make sense out of the following error when I run the
command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
  File "./ThriftHiveMetastore-remote", line 11, in ?
    from thrift.transport import TTransport
ImportError: No module named thrift.transport

Thanks,
Pradeep

________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error

Hi Pradeep,

In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote
that can be used to test out the thrift server independent of the CLI. As a quick test to
narrow down the problem, after the partition is created, can you try running

ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10

Cheers,
Paul

From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error

Hi Ning,
 I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command
printed just before the following line in execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"

There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar
from a hive checkout build.

The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings
to connect to the database directly. By default I have hive set up to use a hive-site.xml
[2] which connects to a thrift server running on localhost.

Any pointers appreciated!

Thanks,
Pradeep

[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<!-- Hive Configuration can either be stored in this file or in the hadoop configuration
files  -->
<!-- that are implied by Hadoop setup variables.                                      
         -->
<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that
Hive    -->
<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized
-->
<!-- resource).                                                                       
         -->

<!-- Hive Execution Parameters -->
<property>
  <name>hive.metastore.uris</name>
  <value>thrift://localhost:9080</value>
  <description>host and port for the thrift metastore server</description>
</property>

<property>
  <name>hive.metastore.local</name>
  <value>false</value>
  <description>controls whether to connect to remove metastore server or open a new
metastore server in Hive Client JVM</description>
</property>

<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/user/pradeepk/hive/warehouse</value>
  <description>location of default database for the warehouse</description>
</property>


</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error

Pradeep,

I ran the commands you provided and it succeeded with the expected behavior.

One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop
& hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar
is there? What is in hive_config_without_thrift?

Thanks,
Ning

On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:

I think there are two separate issues here - I want to open a jira for the first one since
I am now able to reproduce it even with text format with builtin Serdes. Essentially this
is a bug in the thrift code (not sure if it is in the client or server) since the same alter
table statement works fine when the hive client does not use thrift. Here are the details:

cat create_dummy.sql
CREATE external TABLE if not exists dummy (

  partition_name string
  ,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';

hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath.
Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml
to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds

hive  -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10')
location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath.
Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml
to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed:
unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602',
srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath.
Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml
to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds

Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open
a jira with the above details.

Thanks,
Pradeep


________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org>
Subject: RE: alter table add partition error

Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (

  bc string
  ,src_spaceid string
  ,srcpvid string
  ,dstpvid string
  ,dst_spaceid string
  ,page_params map<string, string>
  ,clickinfo map<string, string>
  ,viewinfo array<map<string, string>>

)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';

hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101',
srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

I get the following error:
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed:
unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table
seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift
-e "select src_spaceid from  ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath.
Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml
to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty string
        at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
        at org.apache.hadoop.fs.Path.<init>(Path.java:90)
        at org.apache.hadoop.fs.Path.<init>(Path.java:50)
        at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
        at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
        at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

Any help is much appreciated.

Pradeep

________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org>
Subject: RE: alter table add partition error

hmm... Can you send the exact command and also the create table command for this table.

Ashish

________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value
pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something
to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong
- any help would be greatly appreciated!

Thanks,
Pradeep

________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org>
Subject: Re: alter table add partition error

Hi Pradeep,

partition definition need to be key-value pairs. partition key `action` seems missed the value.

Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pradeepk@yahoo-inc.com<mailto:pradeepk@yahoo-inc.com>>
wrote:
Hi,
    I am trying to create an external table against already existing data in sequencefile
format. However I have written a custom Serde to interpret the data. I am able to create the
table fine but get the exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.

Thanks,
Pradeep

== session output ===

[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp
= '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath.
Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml
to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed:
unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]

== session output ===

/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error
in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable





Mime
View raw message