hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Haviv <daniel.ha...@veracity-group.com>
Subject Re: Hive Alter Partition Location Issue
Date Thu, 30 Apr 2015 14:03:29 GMT
I think you have an extra '/' in the HDFS URI

Daniel

> On 30 באפר׳ 2015, at 16:46, Harsha N <harsha.hadoop12@gmail.com> wrote:
> 
> Thanks for your reply,
> 
> analyze table table1 partition (dt=201501) compute statistics;--returns the same error
> Total jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> java.io.IOException: cannot find dir = hdfs:///data/dt =201501/1430201400/
> in pathToPartitionInfo: [hdfs:/data/dt=201501/1430201400/]
> 	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:347)
> 
> To Add On 
> I am working on External tables in Hive 0.13.1-cdh5.3.2
> 
> -Harsha
> 
>> On Thu, Apr 30, 2015 at 12:37 AM, Mich Talebzadeh <mich@peridale.co.uk> wrote:
>> Hi Harsha,
>> 
>>  
>> 
>> Have you updated stats on table1 after partition adding? In other words it is possible
that the optimiser is not aware of that partition yet?
>> 
>>  
>> 
>> analyze table table1 partition (dt=201501) compute statistics;
>> 
>>  
>> 
>> HTH
>> 
>>  
>> 
>> Mich Talebzadeh
>> 
>>  
>> 
>> http://talebzadehmich.wordpress.com
>> 
>>  
>> 
>> Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN
978-0-9563693-0-7.
>> 
>> co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4
>> 
>> Publications due shortly:
>> 
>> Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence
Cache
>> 
>> Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out
shortly
>> 
>>  
>> 
>> NOTE: The information in this email is proprietary and confidential. This message
is for the designated recipient only, if you are not the intended recipient, you should destroy
it immediately. Any information in this message shall not be understood as given or endorsed
by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the
responsibility of the recipient to ensure that this email is virus free, therefore neither
Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
>> 
>>  
>> 
>> From: Harsha N [mailto:harsha.hadoop12@gmail.com] 
>> Sent: 30 April 2015 07:24
>> To: user@hive.apache.org
>> Subject: Hive Alter Partition Location Issue
>> 
>>  
>> 
>> Hi All,
>> 
>> Can experts share your view on Hive behaviour in below scenario. 
>> 
>>  
>> 
>> I am facing below issue on using alter partition locations in hive.
>> 
>>  
>> 
>> select count(*) from table1 where dt = 201501;
>> 
>>  
>> 
>> Total jobs = 1
>> 
>> Launching Job 1 out of 1
>> 
>> Number of reduce tasks determined at compile time: 1
>> 
>> In order to change the average load for a reducer (in bytes):
>> 
>>   set hive.exec.reducers.bytes.per.reducer=<number>
>> 
>> In order to limit the maximum number of reducers:
>> 
>>   set hive.exec.reducers.max=<number>
>> 
>> In order to set a constant number of reducers:
>> 
>>   set mapreduce.job.reduces=<number>
>> 
>> java.io.IOException: cannot find dir = hdfs:///data/dt =201501/1430201400/
>> 
>> in pathToPartitionInfo: [hdfs:/data/dt=201501/1430201400/]
>> 
>>  
>> 
>> Below are the steps I have followed.
>> 
>> I have altered a partition location in hive using below command.
>> 
>> ALTER TABLE table1 PARTITION (dt=201501) SET LOCATION 'hdfs:///data/dt=201501/1430201400/';
>> 
>>  
>> 
>> I have inserted new data into this new location.
>> 
>>  
>> 
>> INSERT INTO TABLE table1
>> 
>> SELECT * FROM table2 where dt=201501
>> 
>>  
>> 
>> select count(*) from table1 where dt = 201501; doesn't work but 
>> 
>> select * from table1 where dt = 201501 works good.
>> 
>>  
>> 
>> Please let me know if you need more information.
>> 
>>  
>> 
>> Thanks
>> 
>> Harsha
>> 
> 

Mime
View raw message