hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "核弹头す" <510688...@qq.com>
Subject 回复: Hive Query on Hbase snapshot error
Date Thu, 24 Sep 2015 11:44:51 GMT
Thank you for you reply.


The snapshot should be no problem. Here is the following output for the giving command.


 
2015-09-24 19:37:41,981 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated.
Instead, use io.native.lib.available
 
Snapshot Info
 
----------------------------------------
 
   Name: test_snapshot
 
   Type: FLUSH
 
  Table: test_table
 
 Format: 2
 
Created: 2015-09-24T16:00:11
 


 
Table Descriptor
 
----------------------------------------
 
'test_table', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|',
coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
coprocessor$5 => '|org.apache.hadoop.hbase.regionserver.LocalIndexSplitter|805306366|'},
{NAME => 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE
=> '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL =>
'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false',
BLOCKCACHE => 'true'}
 


 


 
32 HFiles (0 in archive), total size 236.1 G (100.00% 236.1 G shared with the source table)
 
0 Logs, total size 0





------------------ 原始邮件 ------------------
发件人: "Sandeep Nemuri";<nhsandeep6@gmail.com>;
发送时间: 2015年9月24日(星期四) 晚上6:43
收件人: "Hive user group"<user@hive.apache.org>; 

主题: Re: Hive Query on Hbase snapshot error



hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot test_snapshot -stats -schema


On Thu, Sep 24, 2015 at 3:43 PM, Sandeep Nemuri <nhsandeep6@gmail.com> wrote:
You can check snapshot state if it is healthy or not using below command.



On Thu, Sep 24, 2015 at 2:55 PM, 核弹头す <510688540@qq.com> wrote:

Hi all,




I am using hive to query on base snapshot. But I got the following  error:

FAILED: IllegalArgumentException org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
Couldn't read snapshot info from:/tmp/hbase-huser/hbase/.hbase-snapshot/test_table_snap0/.snapshotinfo

The following is the steps that I do:

1, on the hbase we already have an table: test_table. And I use command: snapshot ‘test_table’
,’test_table_snap0’   to create a  snapshot。

2,on the hive create an external table like:

CREATE EXTERNAL TABLE `test_table_snapshot`(

  `id` int,

  `alias` string,

  `kdt_id` int,

  `created_time` string,

  `update_time` string,

  `title` string,

  `price` bigint,

  `goods_platform` int,

  `buy_url` string,

  `class1` int,

  `class2` string,

  `goods_type` tinyint,

  `sold_status` tinyint,

  `is_display` tinyint,

  `is_delete` tinyint,

  `num` bigint,

  `buy_way` boolean,

  `source` tinyint,

  `content` string,

  `picture` string,

  `is_virtual` tinyint)

ROW FORMAT SERDE

  'org.apache.hadoop.hive.hbase.HBaseSerDe'

STORED BY

  'org.apache.hadoop.hive.hbase.HBaseStorageHandler'

WITH SERDEPROPERTIES (

  'field.delim'='\u0001',

  'hbase.columns.mapping'=':key,cf1:alalis,cf1:kdt_id,cf1:created_time,cf1:update_time,cf1:title,cf1:price,cf1:goods_platform,cf1:buy_url,cf1:class1,cf1:class2,cf1:goods_type,cf1:sold_stattus,cf1:is_display,cf1:is_delete,cf1:num,cf1:buy_way,cf1:source,cf1:content,cf1:picture,cf1:is_virtual',

  'line.delim'='\n',

  'serialization.format'='\u0001')

TBLPROPERTIES (

  ‘hbase.table.name'='test_table')




4, on Hive, execute “select * from test_table_snapshot”,  then I get the above error.




I have set the “hive.hbase.snapshot.restoredir” value which is same as the root dir on
hbase, and I also set the zookeep server. I checked the base root directory on hdfs and I
can see the snapshot files. But when I use the above query on hive. it seems that the base.tmp.dir
is used to find the snapshot info. BTW, if I directly query on hbase, there is no problem.
I do not know what happened?




The following is the hive error log:



FAILED: SemanticException org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't
read snapshot info from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo

org.apache.hadoop.hive.ql.parse.SemanticException: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
Couldn't read snapshot info from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo

        at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:117)

        at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:182)

        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10207)

        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)

        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)

        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:421)

        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)

        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112)

        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1160)

        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)

        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)

        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)

        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)

        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)

        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)

        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)

        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: java.lang.IllegalArgumentException: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
Couldn't read snapshot info from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo

        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:401)

        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureInputJobProperties(HBaseStorageHandler.java:317)

        at org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobPropertiesForStorageHandler(PlanUtils.java:809)

        at org.apache.hadoop.hive.ql.plan.PlanUtils.configureInputJobPropertiesForStorageHandler(PlanUtils.java:779)

        at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.convertToWork(SimpleFetchOptimizer.java:379)

        at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.access$000(SimpleFetchOptimizer.java:319)

        at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:135)

        at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:106)

        ... 22 more

Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read snapshot
info from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo

        at org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.readSnapshotInfo(SnapshotDescriptionUtils.java:307)

        at org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.copySnapshotForScanner(RestoreSnapshotHelper.java:727)

        at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl.setInput(TableSnapshotInputFormatImpl.java:364)

        at org.apache.hadoop.hive.hbase.HBaseTableSnapshotInputFormatUtil.configureJob(HBaseTableSnapshotInputFormatUtil.java:77)

        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:387)

        ... 29 more








-- 
  Regards  Sandeep Nemuri


 
 




-- 
  Regards  Sandeep Nemuri
Mime
View raw message