hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajesh Balamohan <rajesh.balamo...@gmail.com>
Subject Re: msck repair table and hive v2.1.0
Date Fri, 15 Jul 2016 01:55:16 GMT
Hi Stephen,

Can you try by turning off multi-threaded approach by setting
"hive.mv.files.thread=0"?  You mentioned that your tables tables are in s3,
but the external table created was pointing to HDFS. Was that intentional?

~Rajesh.B

On Fri, Jul 15, 2016 at 6:58 AM, Stephen Sprague <spragues@gmail.com> wrote:

> in the meantime given my tables are in s3 i've written a utility to do a
> 'aws s3 ls' on the bucket and folder in question, change the folder syntax
> to partition syntax and then issued my own 'alter table ... add partition'
> for each partition.
>
>
> so essentially it does what msck repair tables does but in a non-portable
> way.  oh well.  gotta do what ya gotta do.
>
> On Wed, Jul 13, 2016 at 9:29 PM, Stephen Sprague <spragues@gmail.com>
> wrote:
>
>> hey guys,
>> i'm using hive version 2.1.0 and i can't seem to get msck repair table to
>> work.  no matter what i try i get the 'ol NPE.  I've set the log level to
>> 'DEBUG' but yet i still am not seeing any smoking gun.
>>
>> would anyone here have any pointers or suggestions to figure out what's
>> going wrong?
>>
>> thanks,
>> Stephen.
>>
>>
>>
>> hive> create external table foo (a int) partitioned by (date_key bigint)
>> location 'hdfs:/tmp/foo';
>> OK
>> Time taken: 3.359 seconds
>>
>> hive> msck repair table foo;
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.DDLTask
>>
>>
>> from the log...
>>
>> 2016-07-14T04:08:02,431 DEBUG [MSCK-GetPaths-1]:
>> httpclient.RestStorageService (:()) - Found 13 objects in one batch
>> 2016-07-14T04:08:02,431 DEBUG [MSCK-GetPaths-1]:
>> httpclient.RestStorageService (:()) - Found 0 common prefixes in one batch
>> 2016-07-14T04:08:02,433 ERROR [main]: metadata.HiveMetaStoreChecker (:())
>> - java.lang.NullPointerException
>> 2016-07-14T04:08:02,434 WARN  [main]: exec.DDLTask (:()) - Failed to run
>> metacheck:
>> org.apache.hadoop.hive.ql.metadata.HiveException:
>> java.lang.NullPointerException
>>         at
>> org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.getAllLeafDirs(HiveMetaStoreChecker.java:444)
>>         at
>> org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.getAllLeafDirs(HiveMetaStoreChecker.java:448)
>>         at
>> org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.getAllLeafDirs(HiveMetaStoreChecker.java:388)
>>         at
>> org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.findUnknownPartitions(HiveMetaStoreChecker.java:309)
>>         at
>> org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:285)
>>         at
>> org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:230)
>>         at
>> org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkMetastore(HiveMetaStoreChecker.java:109)
>>         at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1814)
>>         at
>> org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:403)
>>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
>>         at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
>>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
>>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562)
>>         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313)
>>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
>>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072)
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
>>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
>>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>
>


-- 
~Rajesh.B

Mime
View raw message