kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Yang <liy...@apache.org>
Subject Re: how to load table from hive
Date Wed, 30 Dec 2015 09:54:22 GMT
Hive mode does not matters. What to check is if the right hive conf is on
classpath.

Run `bin/find-hive-dependency.sh` and check if the printed classpath
contains the expected hive config dir.

On Mon, Dec 28, 2015 at 10:48 PM, zz <zzzzes@vip.qq.com> wrote:

> when I execute : hive -e "describe kylin_sales" , it work well . Also I
> can execute sql for metastore table from mysql client successfully !
> I guess metastore is OK . I config hive as remote model, but Kylin can't
> use it . From Kylin's log , seems "DataNucleus" regard it as embedded
> model then use DERBY DB. Actually, I use mysql to store metadata .
>
> below is my full log, who know what's wrong with it ...
>
> ESTER=ADMIN;REQ_TIME=GMT-08:00 2015-12-28
> 06:25:40;URI=/kylin/api/tables;METHOD=GET;QUERY_STRING=ext=true&project=learn_kylin;PAYLOAD=;RESP_STATUS=200;
> 2015-12-28 14:26:03,159 INFO  [http-bio-7070-exec-1]
> Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use
> mapreduce.job.reduces
> 2015-12-28 14:26:03,159 INFO  [http-bio-7070-exec-1]
> Configuration.deprecation: mapred.min.split.size is deprecated. Instead,
> use mapreduce.input.fileinputformat.split.minsize
> 2015-12-28 14:26:03,159 INFO  [http-bio-7070-exec-1]
> Configuration.deprecation: mapred.reduce.tasks.speculative.execution is
> deprecated. Instead, use mapreduce.reduce.speculative
> 2015-12-28 14:26:03,159 INFO  [http-bio-7070-exec-1]
> Configuration.deprecation: mapred.min.split.size.per.node is deprecated.
> Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
> 2015-12-28 14:26:03,159 INFO  [http-bio-7070-exec-1]
> Configuration.deprecation: mapred.input.dir.recursive is deprecated.
> Instead, use mapreduce.input.fileinputformat.input.dir.recursive
> 2015-12-28 14:26:03,159 INFO  [http-bio-7070-exec-1]
> Configuration.deprecation: mapred.min.split.size.per.rack is deprecated.
> Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
> 2015-12-28 14:26:03,159 INFO  [http-bio-7070-exec-1]
> Configuration.deprecation: mapred.max.split.size is deprecated. Instead,
> use mapreduce.input.fileinputformat.split.maxsize
> 2015-12-28 14:26:03,160 INFO  [http-bio-7070-exec-1]
> Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is
> deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
> 2015-12-28 14:26:03,446 INFO  [http-bio-7070-exec-1]
> metastore.HiveMetaStore: 0: Opening raw store with implemenation
> class:org.apache.hadoop.hive.metastore.ObjectStore
> 2015-12-28 14:26:03,481 INFO  [http-bio-7070-exec-1]
> metastore.ObjectStore: ObjectStore, initialize called
> 2015-12-28 14:26:03,734 INFO  [http-bio-7070-exec-1]
> DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will
> be ignored
> 2015-12-28 14:26:03,735 INFO  [http-bio-7070-exec-1]
> DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown
> unknown - will be ignored
> 2015-12-28 14:26:06,771 INFO  [http-bio-7070-exec-1]
> metastore.ObjectStore: Setting MetaStore object pin classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 2015-12-28 14:26:09,744 INFO  [http-bio-7070-exec-1]
> DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 2015-12-28 14:26:09,744 INFO  [http-bio-7070-exec-1]
> DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
> "embedded-only" so does not have its own datastore table.
> 2015-12-28 14:26:10,175 INFO  [http-bio-7070-exec-1]
> DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 2015-12-28 14:26:10,175 INFO  [http-bio-7070-exec-1]
> DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
> "embedded-only" so does not have its own datastore table.
> 2015-12-28 14:26:10,325 INFO  [http-bio-7070-exec-1] DataNucleus.Query:
> Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0"
> since the connection used is closing
> 2015-12-28 14:26:10,327 INFO  [http-bio-7070-exec-1]
> metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
> 2015-12-28 14:26:10,329 INFO  [http-bio-7070-exec-1]
> metastore.ObjectStore: Initialized ObjectStore
> 2015-12-28 14:26:10,665 INFO  [http-bio-7070-exec-1]
> metastore.HiveMetaStore: Added admin role in metastore
> 2015-12-28 14:26:10,667 INFO  [http-bio-7070-exec-1]
> metastore.HiveMetaStore: Added public role in metastore
> 2015-12-28 14:26:10,772 INFO  [http-bio-7070-exec-1]
> metastore.HiveMetaStore: No user is added in admin role, since config is
> empty
> 2015-12-28 14:26:10,922 INFO  [http-bio-7070-exec-1]
> metastore.HiveMetaStore: 0: get_table : db=default tbl=kylin_sales
> 2015-12-28 14:26:10,923 INFO  [http-bio-7070-exec-1] HiveMetaStore.audit:
> ugi=ocp       ip=unknown-ip-addr      cmd=get_table : db=default
> tbl=kylin_sales
> NoSuchObjectException(message:default.kylin_sales table not found)
>         at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1808)
>
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "ShaoFeng Shi";<shaofengshi@apache.org>;
> *发送时间:* 2015年12月25日(星期五) 晚上8:53
> *收件人:* "user"<user@kylin.apache.org>;
> *主题:* Re: how to load table from hive
>
> what's the output of executing this command?:
>
> hive -e "describe <your_table_name>"
>
> 2015-12-25 11:19 GMT+08:00 zz <zzzzes@vip.qq.com>:
>
>> Hi Yang & all,
>> I checked hive config by "hive -e set" in the server which kylin
>> deployed, seems nothing wrong.
>> Then I tried to run sample.sh in kylin/bin , it created tables:
>> "kylin_cal_dt"  "kylin_category_groupings"  "kylin_sales" successfully and
>> I can find these in metastore DB (mysql) .
>> Need I do some config in kylin.properties to make it load hive table
>> metadata by hive remote model? I guess some property make it use hive
>> default Derby .
>> Below is my hive-site.xml:
>>
>> <configuration>
>>    <property>
>>       <name>javax.jdo.option.ConnectionURL</name>
>>
>> <value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true</value>
>>       <description>metadata is stored in a MySQL server</description>
>>    </property>
>>    <property>
>>       <name>javax.jdo.option.ConnectionDriverName</name>
>>       <value>com.mysql.jdbc.Driver</value>
>>       <description>MySQL JDBC driver class</description>
>>    </property>
>>    <property>
>>       <name>javax.jdo.option.ConnectionUserName</name>
>>       <value>hive</value>
>>       <description>user name for connecting to mysql server</description>
>>    </property>
>>    <property>
>>       <name>javax.jdo.option.ConnectionPassword</name>
>>       <value>password</value>
>>       <description>password for connecting to mysql server</description>
>>    </property>
>>    <property>
>>         <name>hive.metastore.uris</name>
>>         <value>thrift://10.224.243.127:9083</value>
>>    </property>
>>    <property>
>>         <name>hive.metastore.schema.verification</name>
>>         <value>false</value>
>>    </property>
>>    <property>
>>         <name>hive.metastore.warehouse.dir</name>
>>         <value>/user/hive/warehouse</value>
>>    </property>
>>   <!--<property>
>>         <name>hive.server.thrift.port</name>
>>         <value>10000</value>
>>    </property> -->
>> </configuration>
>>
>>
>>
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "Li Yang";<liyang@apache.org>;
>> *发送时间:* 2015年12月25日(星期五) 上午10:10
>> *收件人:* "user"<user@kylin.apache.org>;
>> *主题:* Re: how to load table from hive
>>
>> What matters is the hive-site.xml Kylin picks up. Kylin detects hive
>> config by command `hive -e set`. You can try hive command line on the node
>> where Kylin is deployed and make sure it connects to the desired metastore.
>>
>> Try diagnose `bin/find-hive-dependency.sh` if you still got problem.
>>
>> On Fri, Dec 25, 2015 at 9:49 AM, zz <zzzzes@vip.qq.com> wrote:
>>
>>> Hi Kylin developers,
>>>
>>> I'm a newbie for Kylin. Glad to see the excellent OLAP tool. As a
>>> Chinese acmen , I like the name.
>>>
>>> After I download and run it , I can access the website(
>>> http://IP:7070/kylin) and create project successfully . However,  when
>>> I try to load Hive table,  "NoSuchObjectException" alert (Oops...) !
>>>
>>> From kylin.log in tomcat , I found some sensitive tracks:
>>> 2015-12-24 15:01:16,613 INFO  [http-bio-7070-exec-6]
>>> metastore.ObjectStore: ObjectStore, initialize called
>>> 2015-12-24 15:01:16,926 INFO  [http-bio-7070-exec-6]
>>> DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will
>>> be ignored
>>> 2015-12-24 15:01:16,927 INFO  [http-bio-7070-exec-6]
>>> DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown
>>> unknown - will be ignored
>>> ...
>>> 2015-12-24 15:01:24,350 INFO  [http-bio-7070-exec-6]
>>> metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
>>>
>>> I'm not sure these ignored property is OK or not.
>>> I config Hive as remote mode(use mysql to store metadata, config thrift
>>> uris) but kylin try to access DERBY which I do not use !
>>>
>>> Could you tell me how to config it ?
>>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
>

Mime
View raw message