hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: enough alter tables in the same .q file eventually fail
Date Thu, 22 Apr 2010 17:54:41 GMT
On Mon, Apr 19, 2010 at 10:59 AM, Edward Capriolo <edlinuxguru@gmail.com>wrote:

>
>
> On Mon, Apr 12, 2010 at 10:39 AM, Edward Capriolo <edlinuxguru@gmail.com>wrote:
>
>>
>>
>> On Sat, Apr 10, 2010 at 10:30 AM, Edward Capriolo <edlinuxguru@gmail.com>wrote:
>>
>>>
>>>
>>> On Thu, Apr 8, 2010 at 6:58 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>>
>>>> Typo in Ed's last email (table name):
>>>> echo "create external table if not exists ed*_*test ( dat string )
>>>> partitioned by (dummy string) location '/tmp/a';" > test.q
>>>>
>>>>
>>>> On Thu, Apr 8, 2010 at 3:14 PM, Edward Capriolo <edlinuxguru@gmail.com>wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 8, 2010 at 5:22 PM, Edward Capriolo <edlinuxguru@gmail.com
>>>>> > wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Apr 8, 2010 at 5:14 PM, Paul Yang <pyang@facebook.com>
wrote:
>>>>>>
>>>>>>> Seems to be fixed in 0.6. Here's what I got:
>>>>>>>
>>>>>>> test.q:
>>>>>>> alter table tmp_pyang_t ADD PARTITION (ds='2') LOCATION '/tmp/blah2';
>>>>>>> alter table tmp_pyang_t ADD PARTITION (ds='3') LOCATION '/tmp/blah2';
>>>>>>> alter table tmp_pyang_t ADD PARTITION (ds='4') LOCATION '/tmp/blah2';
>>>>>>> alter table tmp_pyang_t ADD PARTITION (ds='5') LOCATION '/tmp/blah2';
>>>>>>> alter table tmp_pyang_t ADD PARTITION (ds='6') LOCATION '/tmp/blah2';
>>>>>>> alter table tmp_pyang_t ADD PARTITION (ds='7') LOCATION '/tmp/blah2';
>>>>>>> alter table tmp_pyang_t ADD PARTITION (ds='8') LOCATION '/tmp/blah2';
>>>>>>>
>>>>>>>
>>>>>>> Hive history
>>>>>>> file=/tmp/pyang/hive_job_log_pyang_201004081410_378771152.txt
>>>>>>> OK
>>>>>>> Time taken: 4.101 seconds
>>>>>>> OK
>>>>>>> Time taken: 0.558 seconds
>>>>>>> OK
>>>>>>> Time taken: 0.453 seconds
>>>>>>> OK
>>>>>>> Time taken: 0.416 seconds
>>>>>>> OK
>>>>>>> Time taken: 0.378 seconds
>>>>>>> OK
>>>>>>> Time taken: 0.457 seconds
>>>>>>> OK
>>>>>>> Time taken: 0.454 seconds
>>>>>>>
>>>>>>>
>>>>>>> Can you the stack trace in /tmp/<username>/hive.log?
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Prasad Chakka [mailto:pchakka@facebook.com]
>>>>>>> Sent: Thursday, April 08, 2010 1:03 PM
>>>>>>> To: hive-user@hadoop.apache.org
>>>>>>> Subject: Re: enough alter tables in the same .q file eventually
fail
>>>>>>>
>>>>>>> There was a bug that got fixed where each request was creating
a
>>>>>>> separate metastore client. That could be it or something similar
that hasn't
>>>>>>> gotten fixed.
>>>>>>>
>>>>>>> On Apr 8, 2010, at 11:47 AM, Edward Capriolo wrote:
>>>>>>>
>>>>>>> > Hive 5.0 mysql as metastore backend. Using external tables
with
>>>>>>> location for partitions...
>>>>>>> >
>>>>>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329'
, mid
>>>>>>> = '000843') LOCATION 'hit_date=20100329/mid=000843';
>>>>>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329'
, mid
>>>>>>> = '000844') LOCATION 'hit_date=20100329/mid=000844';
>>>>>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329'
, mid
>>>>>>> = '000849') LOCATION 'hit_date=20100329/mid=000849';
>>>>>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329'
, mid
>>>>>>> = '000850') LOCATION 'hit_date=20100329/mid=000850';
>>>>>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329'
, mid
>>>>>>> = '000851') LOCATION 'hit_date=20100329/mid=000851';
>>>>>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329'
, mid
>>>>>>> = '000852') LOCATION 'hit_date=20100329/mid=000852';
>>>>>>> >
>>>>>>> > Eventually this fails after a number of entries.
>>>>>>> >
>>>>>>> > Time taken: 0.159 seconds
>>>>>>> > OK
>>>>>>> > Time taken: 0.17 seconds
>>>>>>> > OK
>>>>>>> > Time taken: 0.241 seconds
>>>>>>> > FAILED: Error in metadata: Unable to fetch table XXXXX_action
>>>>>>> > FAILED: Execution Error, return code 1 from
>>>>>>> org.apache.hadoop.hive.ql.exec.DDLTask
>>>>>>> >
>>>>>>> > Restarting the process after removing the already added
tables
>>>>>>> works until it breaks again. Anyone ever dealt with this?
>>>>>>> >
>>>>>>> > Doing one hive -e per table always works but takes a lot
longer
>>>>>>> ...3 seconds a partition rather then ~.5 seconds.
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>> It does not happen after 4 or 5 more like 100 or 1000+. I will try
to
>>>>>> track this down a bit.
>>>>>>
>>>>>> Edward
>>>>>>
>>>>>
>>>>>
>>>>> Try this:
>>>>>
>>>>> echo "create external table if not exists edtest ( dat string )
>>>>> partitioned by (dummy string) location '/tmp/a';" > test.q
>>>>>  for i in {1..3000} ; do echo "alter table ed_test add partition
>>>>> (dummy='${i}') location '/tmp/duh';" ; done >> test.q
>>>>> hive -f test.q
>>>>>
>>>>> On Hive 5.0 I get a failure mid way.
>>>>> Edward
>>>>>
>>>>
>>>>
>>> Also trying to do selects from the table without enough pruning in the
>>> where clause causes the same error, sometimes it comes as a JDBC/jpox access
>>> denied error.
>>>
>>
>>
>> Also, there are problems working with this type of table as well. :(
>>
>> $ hive -e "explain select * from XXXXX_action "
>> Hive history
>> file=/tmp/XXXXXX/hive_job_log_media6_201004121029_170696698.txt
>> FAILED: Error in semantic analysis: javax.jdo.JDODataStoreException:
>> Access denied for user 'hivadm'@'XXXXXX' (using password: YES)
>> NestedThrowables:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXXX' (using
>> password: YES)
>>
>> Interestingly enough if we specify some partitions we can dodge this
>> error. I get the fealing that the select * is trying to select too many
>> partitions and causing this error.
>>
>> 2010-04-12 10:33:02,789 ERROR metadata.Hive (Hive.java:getPartition(629))
>> - javax.jdo.JDODataStoreException: Access denied for user 'hivadm'@'rs01
>> .sd.pl.pvt' (using password: YES)
>>     at
>> org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:289)
>>     at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:274)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:551)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(ObjectStore.java:716)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:704)
>>     at
>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(HiveMetaStore.java:593)
>>     at
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:418)
>>     at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:620)
>>     at
>> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:215)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:4883)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:5224)
>>     at
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
>>     at
>> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:44)
>>     at
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
>>     at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275)
>>     at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
>>     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
>>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
>>     at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
>>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:251)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> NestedThrowablesStackTrace:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:946)
>>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2985)
>>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:885)
>>     at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3436)
>>     at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1247)
>>     at com.mysql.jdbc.Connection.createNewIO(Connection.java:2775)
>>     at com.mysql.jdbc.Connection.<init>(Connection.java:1555)
>>     at
>> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
>>     at
>> org.datanucleus.store.rdbms.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:142)
>>     at
>> org.datanucleus.store.rdbms.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:118)
>>     at
>> org.datanucleus.store.rdbms.ConnectionProviderPriorityList.getConnection(ConnectionProviderPriorityList.java:59)
>>     at
>> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:458)
>>     at
>> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getXAResource(ConnectionFactoryImpl.java:418)
>>     at
>> org.datanucleus.ConnectionManagerImpl.enlistResource(ConnectionManagerImpl.java:329)
>>     at
>> org.datanucleus.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:210)
>>     at
>> org.datanucleus.store.rdbms.ConnectionFactoryImpl.getConnection(ConnectionFactoryImpl.java:345)
>>     at
>> org.datanucleus.store.AbstractStoreManager.getConnection(AbstractStoreManager.java:267)
>>     at
>> org.datanucleus.store.rdbms.query.SQLEvaluator.evaluate(SQLEvaluator.java:91)
>>     at
>> org.datanucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:287)
>>     at org.datanucleus.store.query.Query.executeQuery(Query.java:1475)
>>     at
>> org.datanucleus.store.rdbms.query.JDOQLQuery.executeQuery(JDOQLQuery.java:244)
>>     at org.datanucleus.store.query.Query.executeWithArray(Query.java:1357)
>>     at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:265)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:551)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(ObjectStore.java:716)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:704)
>>     at
>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(HiveMetaStore.java:593)
>>     at
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:418)
>>     at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:620)
>>     at
>> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:215)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:4883)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:5224)
>>     at
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
>>     at
>> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:44)
>>     at
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
>>     at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275)
>>     at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
>>     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
>>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
>>     at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
>>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:251)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>
>> 2010-04-12 10:33:02,790 ERROR parse.SemanticAnalyzer
>> (SemanticAnalyzer.java:genMapRedTasks(4886)) -
>> org.apache.hadoop.hive.ql.metadata.HiveExcepti
>> on: javax.jdo.JDODataStoreException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>> NestedThrowables:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:630)
>>     at
>> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:215)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:4883)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:5224)
>>     at
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
>>     at
>> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:44)
>>     at
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
>>     at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275)
>>     at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
>>     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
>>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
>>     at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
>>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:251)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: javax.jdo.JDODataStoreException: Access denied for user
>> 'hivadm'@'XXXXX.domain.whatetever' (using password: YES)
>> NestedThrowables:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at
>> org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:289)
>>     at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:274)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:551)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(ObjectStore.java:716)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:704)
>>     at
>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(HiveMetaStore.java:593)
>>     at
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:418)
>>     at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:620)
>>     ... 17 more
>> Caused by: java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:946)
>>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2985)
>>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:885)
>>     at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3436)
>>     at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1247)
>>     at com.mysql.jdbc.Connection.createNewIO(Connection.java:2775)
>>     at com.mysql.jdbc.Connection.<init>(Connection.java:1555)
>>     at
>> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
>>     at
>> org.datanucleus.store.rdbms.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:142)
>>     at
>> org.datanucleus.store.rdbms.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:118)
>>     at
>> org.datanucleus.store.rdbms.ConnectionProviderPriorityList.getConnection(ConnectionProviderPriorityList.java:59)
>>     at
>> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:458)
>>     at
>> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getXAResource(ConnectionFactoryImpl.java:418)
>>     at
>> org.datanucleus.ConnectionManagerImpl.enlistResource(ConnectionManagerImpl.java:329)
>>     at
>> org.datanucleus.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:210)
>>
>>     at
>> org.datanucleus.store.rdbms.ConnectionFactoryImpl.getConnection(ConnectionFactoryImpl.java:345)
>>     at
>> org.datanucleus.store.AbstractStoreManager.getConnection(AbstractStoreManager.java:267)
>>     at
>> org.datanucleus.store.rdbms.query.SQLEvaluator.evaluate(SQLEvaluator.java:91)
>>     at
>> org.datanucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:287)
>>     at org.datanucleus.store.query.Query.executeQuery(Query.java:1475)
>>     at
>> org.datanucleus.store.rdbms.query.JDOQLQuery.executeQuery(JDOQLQuery.java:244)
>>     at org.datanucleus.store.query.Query.executeWithArray(Query.java:1357)
>>     at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:265)
>>     ... 23 more
>>
>> 2010-04-12 10:33:02,793 ERROR ql.Driver
>> (SessionState.java:printError(248)) - FAILED: Error in semantic analysis:
>> javax.jdo.JDODataStoreException:
>>  Access denied for user 'hivadm'@'XXXXX.domain.whatetever' (using
>> password: YES)
>> NestedThrowables:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>> org.apache.hadoop.hive.ql.parse.SemanticException:
>> javax.jdo.JDODataStoreException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using passwo
>> rd: YES)
>> NestedThrowables:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:4887)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:5224)
>>     at
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
>>     at
>> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:44)
>>     at
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
>>     at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275)
>>     at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
>>     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
>>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
>>     at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
>>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:251)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
>> javax.jdo.JDODataStoreException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (us
>> ing password: YES)
>> NestedThrowables:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:630)
>>     at
>> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:215)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:4883)
>>     ... 15 more
>> Caused by: javax.jdo.JDODataStoreException: Access denied for user
>> 'hivadm'@'XXXXX.domain.whatetever' (using password: YES)
>> NestedThrowables:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at
>> org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:289)
>>     at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:274)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:551)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(ObjectStore.java:716)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:704)
>>     at
>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(HiveMetaStore.java:593)
>>     at
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:418)
>>     at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:620)
>>     ... 17 more
>> Caused by: java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:946)
>>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2985)
>>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:885)
>>     at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3436)
>>     at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1247)
>>     at com.mysql.jdbc.Connection.createNewIO(Connection.java:2775)
>>     at com.mysql.jdbc.Connection.<init>(Connection.java:1555)
>>     at
>> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
>>     at
>> org.datanucleus.store.rdbms.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:142)
>>     at
>> org.datanucleus.store.rdbms.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:118)
>>     at
>> org.datanucleus.store.rdbms.ConnectionProviderPriorityList.getConnection(ConnectionProviderPriorityList.java:59)
>>     at
>> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:458)
>>     at
>> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getXAResource(ConnectionFactoryImpl.java:418)
>>     at
>> org.datanucleus.ConnectionManagerImpl.enlistResource(ConnectionManagerImpl.java:329)
>>
>> Access denied for user 'hivadm'@'XXXXX.domain.whatetever' (using
>> password: YES)
>> NestedThrowables:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>> org.apache.hadoop.hive.ql.parse.SemanticException:
>> javax.jdo.JDODataStoreException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using passwo
>> rd: YES)
>> NestedThrowables:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:4887)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:5224)
>>     at
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
>>     at
>> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:44)
>>     at
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
>>     at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275)
>>     at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
>>     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
>>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
>>     at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
>>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:251)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
>> javax.jdo.JDODataStoreException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (us
>> ing password: YES)
>> NestedThrowables:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:630)
>>     at
>> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:215)
>>     at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:4883)
>>     ... 15 more
>> Caused by: javax.jdo.JDODataStoreException: Access denied for user
>> 'hivadm'@'XXXXX.domain.whatetever' (using password: YES)
>> NestedThrowables:
>> java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at
>> org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:289)
>>     at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:274)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:551)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(ObjectStore.java:716)
>>     at
>> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:704)
>>     at
>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(HiveMetaStore.java:593)
>>     at
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:418)
>>     at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:620)
>>     ... 17 more
>> Caused by: java.sql.SQLException: Access denied for user 'hivadm'@'XXXXX.domain.whatetever'
>> (using password: YES)
>>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:946)
>>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2985)
>>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:885)
>>     at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3436)
>>     at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1247)
>>     at com.mysql.jdbc.Connection.createNewIO(Connection.java:2775)
>>     at com.mysql.jdbc.Connection.<init>(Connection.java:1555)
>>     at
>> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
>>     at
>> org.datanucleus.store.rdbms.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:142)
>>     at
>> org.datanucleus.store.rdbms.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:118)
>>     at
>> org.datanucleus.store.rdbms.ConnectionProviderPriorityList.getConnection(ConnectionProviderPriorityList.java:59)
>>     at
>> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:458)
>>     at
>> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getXAResource(ConnectionFactoryImpl.java:418)
>>     at
>> org.datanucleus.ConnectionManagerImpl.enlistResource(ConnectionManagerImpl.java:329)
>>     at
>> org.datanucleus.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:210)
>>     at
>> org.datanucleus.store.rdbms.ConnectionFactoryImpl.getConnection(ConnectionFactoryImpl.java:345)
>>     at
>> org.datanucleus.store.AbstractStoreManager.getConnection(AbstractStoreManager.java:267)
>>     at
>> org.datanucleus.store.rdbms.query.SQLEvaluator.evaluate(SQLEvaluator.java:91)
>>     at
>> org.datanucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:287)
>>     at org.datanucleus.store.query.Query.executeQuery(Query.java:1475)
>>     at
>> org.datanucleus.store.rdbms.query.JDOQLQuery.executeQuery(JDOQLQuery.java:244)
>>     at org.datanucleus.store.query.Query.executeWithArray(Query.java:1357)
>>     at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:265)
>>     ... 23 more
>>
>>
>>
>>
>>
> The same problem occurs dropping partitions. At this stage any program I
> write that does heavy duty work I need to prefilter with a unix tool like
> 'split' to make sure I do not blow up half way through 1000 inserts or 1000
> drops. It is really anti-productive.
>
>
>

Also when using this type of layout with external tables, queries on
partitions that do not exist blow up...


hive>  select OFFER_ID from XXXXX_act where hit_date=20990410 and
mid=000979;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201004221345_0002, Tracking URL =
http://rs01.hadoop.pvt:50030/jobdetails.jsp?jobid=job_201004221345_0002
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job
-Dmapred.job.tracker=rs01.hadoop.pvt:34311 -kill job_201004221345_0002
2010-04-22 13:52:10,004 Stage-1 map = 0%,  reduce = 0%
2010-04-22 13:52:44,173 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201004221345_0002 with errors

Failed tasks with most(4) failures :
Task URL:
http://rs01.hadoop.pvt:50030/taskdetails.jsp?jobid=job_201004221345_0002&tipid=task_201004221345_0002_m_000000

FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.ExecDriver
hive>


...arge...

Mime
View raw message