hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arvind Prabhakar (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1176) 'create if not exists' fails for a table name with 'select' in it
Date Wed, 16 Jun 2010 18:36:24 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879441#action_12879441
] 

Arvind Prabhakar commented on HIVE-1176:
----------------------------------------

bq. Can you elaborate on what you mean by 'some collections were being fetched as semi-populated
proxies with missing session context leading to NPEs'? Is there something I can do to reproduce
this?

@Paul: Here are the steps to reproduce this problem:

# Startout with a clean workspace checkout and apply the updated patch HIVE-1176-2.patch.

# Manually revert the file {{metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java}}
to its previous state
# run {{ant package}} from the root of the workspace
# run {{ant test}} from within metastore

You should see failures like the following:
{code}
    [junit] testPartition() failed.
    [junit] java.lang.NullPointerException
    [junit] 	at org.datanucleus.store.mapped.scostore.AbstractMapStore.validateKeyForWriting(AbstractMapStore.java:333)
    [junit] 	at org.datanucleus.store.mapped.scostore.JoinMapStore.put(JoinMapStore.java:252)
    [junit] 	at org.datanucleus.sco.backed.Map.put(Map.java:640)
    [junit] 	at org.apache.hadoop.hive.metastore.api.Table.putToParameters(Table.java:359)
    [junit] 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table(HiveMetaStore.java:1281)
    [junit] 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:140)
    [junit] 	at org.apache.hadoop.hive.metastore.TestHiveMetaStore.testAlterTable(TestHiveMetaStore.java:728)
    [junit] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    ...
{code}

If you look at {{src/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java}} you would
notice that the line causing this exception should ideally be a {{HashMap}} and not an {{org.datanucleus.store.mapped.scostore.AbstractMapStore}}
as indicated by the stack trace. This happens because the datanucleus JDO framework replaces
collections with its own implementation in order to allow lazy-dereferencing and optimize
for database connections/queries/memory consumption etc.

Lazy loading of collections (and second class objects in general) can be disabled at a global
level or at entity level. Disabling this globally is generally not recommended unless there
is evidence backed by extensive testing that supports that change. Disabling at an entity
level is still OK provided the entity object graph is fully dereferenced at all times. This
could lead to extensive memory consumption in the system in case the entity graph is huge.


My approach towards fixing the problem was to *not* change the default behavior in the general
case. Instead I felt that it was better to circumvent this problem in the case of a remote
metastore by creating a copy explicitly. If you have other suggestions on how to address this,
please let me know.

Also - more information on the lazy dereferencing mechanism used by datanucleus framework
can be found [here|http://www.datanucleus.org/plugins/core/sco.html].


> 'create if not exists' fails for a table name with 'select' in it
> -----------------------------------------------------------------
>
>                 Key: HIVE-1176
>                 URL: https://issues.apache.org/jira/browse/HIVE-1176
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Metastore, Query Processor
>            Reporter: Prasad Chakka
>            Assignee: Arvind Prabhakar
>             Fix For: 0.6.0
>
>         Attachments: HIVE-1176-1.patch, HIVE-1176-2.patch, HIVE-1176.lib-files.tar.gz,
HIVE-1176.patch
>
>
> hive> create table if not exists tmp_select(s string, c string, n int);
> org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception:
javax.jdo.JDOUserException JDOQL Single-String query should always start with SELECT)
>         at org.apache.hadoop.hive.ql.metadata.Hive.getTablesForDb(Hive.java:441)
>         at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:423)
>         at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:5538)
>         at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:5192)
>         at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275)
>         at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
>         at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: MetaException(message:Got exception: javax.jdo.JDOUserException JDOQL Single-String
query should always start with SELECT)
>         at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:612)
>         at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:450)
>         at org.apache.hadoop.hive.ql.metadata.Hive.getTablesForDb(Hive.java:439)
>         ... 15 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message