hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Pawar <nitinpawar...@gmail.com>
Subject Re: Need help with Installation hive 0.11
Date Thu, 10 Oct 2013 07:39:46 GMT
Baahu,
from hive-0.11 , hcat is part of hive.

So you basically do not need to configure your ports etc in hcat_server.sh

what you need to set is hive.metastore.uris in hive-site.xml. Default port
is 9083 I suppose



On Thu, Oct 10, 2013 at 12:36 PM, Baahu <bahubali@gmail.com> wrote:

> Thanks Nitin for your help.I was able to run hive and also start meta
> store finally !!
> The tar should contain required config files(instead of having templates),
> otherwise it turns out to be a difficult for newbies like me to go ahead
> and learn/use hive.
> Also I could not find the port number for metastore and had to hardcode it
> in hcat_server.sh.
>
>
> On Mon, Oct 7, 2013 at 3:29 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:
>
>> So here is what I did quickly. (I did not follow the step by step guide
>> from hive wiki)
>> 1) I already have a running hadoop cluster
>> 2) Downloaded the bin-tar.gz from hive.apache.org links
>> 3) Uncompressed the binaries
>> 4) Inside the conf folder there are few .template files (I don't know
>> there is no hive-site.xml there. In core hadoop projects the core-site.xml
>> is empty but its present)
>> 5) I just copied all the template files non template files (eg.
>> hive-defaults.xml.template to hive-defaults.xml and so on)
>> 6) I created a a new hive-site.xml file
>> 7) added following properties to it (which are kind of missing from
>> hive-defaults but we will talk on dev forums for that)
>>
>>    - javax.jdo.option.ConnectionURL - I use mysql as my meta store so i
>>    have pointed this to
>>    jdbc:mysql://<myhost>:<myport>/<mydbname>?createDatabaseIfNotExist=true
>>    - javax.jdo.option.ConnectionDriverName - this is set to
>>    com.mysql.jdbc.driver  I have downloaded mysql driver from oracle and
>>    stored in hive lib folder.
>>    - javax.jdo.option.ConnectionUserName - this is my db username
>>    - javax.jdo.option.ConnectionPassword - this is password for the
>>    username i used above
>>
>> 8) export HADOOP_HOME variable to where my hadoop installation is present
>> 9) export HIVE_HOME to the directory where I have extracted the tar ball
>> 10) Granted access to new machine on mysql db
>> 11) started hive metastore and verified show tables.
>>
>> This worked. Of course this does not take care of security on hive yet.
>>
>> if run into any issues after this do let me know.
>>
>>
>>  On Sat, Oct 5, 2013 at 7:50 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:
>>
>>> Baahu,
>>>
>>> hive has two conf files. One is hive-site.xml and other is hive-env.sh
>>>
>>> From hive-0.11 hcat is bundled with hive so you specially need not do
>>> anything special to setup hcat.
>>> you just need to setup few environment variables like HIVE_HOME,
>>> HIVE_CONF_DIR, HADOOP_HOME.
>>>
>>> and then if you have setup the hive-site.xml correctly then it should
>>> work out of the box.
>>>
>>> If you have downloaded the already built tarball. you need not compile
>>> it. Plus its really a long build if you run it with unit test cases etc.
>>>
>>> If possible tonight I will  setup a vm and test out the steps mentioned
>>> on hive wiki.
>>>
>>>
>>> On Sat, Oct 5, 2013 at 7:21 PM, Baahu <bahubali@gmail.com> wrote:
>>>
>>>> Mikhail,
>>>> The hive-site.xml was missing and while googling I found some
>>>> suggestion of copying
>>>> hive-0.11.0-bin/hcatalog/etc/hcatalog/proto-hive-site.xml as hive-site.xml
>>>> to the directory hive-0.11.0-bin/conf .
>>>> Was this a right instruction, if not can you please let me know where I
>>>> can find hive-conf.xml file.
>>>>
>>>> Should we really have to build hive as mentioned in the below link
>>>> during installation? Just untarring the hive tar ball doesn't lead to its
>>>> installation?
>>>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted
>>>>
>>>> Thanks,
>>>> Baahu
>>>>
>>>>
>>>>  On Sat, Oct 5, 2013 at 3:26 PM, Mikhail Antonov <olorinbant@gmail.com>wrote:
>>>>
>>>>> Bases on original question I decided the author does not need hcatalog
>>>>> specifically, but got sample hive-site.xml suggesting storage authenticator
>>>>> somewhere. So why use it?:)
>>>>>
>>>>> On Oct 5, 2013, at 2:51, Lefty Leverenz <leftyleverenz@gmail.com>
>>>>> wrote:
>>>>>
>>>>> P.S.  I found StorageDelegationAuthorizationProvider mentioned in the
>>>>> Known Issues section of the HCatalog authorization doc:  "A different
>>>>> authorization provider (StorageDelegationAuthorizationProvider) needs
to be
>>>>> used for working with HBase tables as well. But that is not well tested
>>>>> ."
>>>>>
>>>>> -- Lefty
>>>>>
>>>>>
>>>>> On Sat, Oct 5, 2013 at 2:44 AM, Lefty Leverenz <
>>>>> leftyleverenz@gmail.com> wrote:
>>>>>
>>>>>> I'm out of my depth, but based on existing documentation I'll make
>>>>>> some suggestions you could try while waiting for replies from people
who
>>>>>> actually know about installation.
>>>>>>
>>>>>> Following Mikhail's lead, since this is an HCatalog security problem
>>>>>> you should start by seeing if HCatalog is installed.  Then you could
try
>>>>>> changing the value of hive.security.authorization.manager -- I found
it
>>>>>> documented in two places:
>>>>>>
>>>>>>    - Hive Configuration Properties<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Authentication%2FAuthorization>
>>>>>>    - HCatalog Storage Based Authorization<https://cwiki.apache.org/confluence/display/Hive/HCatalog+Authorization#HCatalogAuthorization-ConfiguringStorageSystemBasedAuthorization>
>>>>>>
>>>>>> The HCatalog authorization doc says you need to add the HCatalog
jar
>>>>>> to the Hive classpath (add "export HIVE_AUX_JARS_PATH=<*path to
>>>>>> hcatalog jar*>" to hive-env.sh), but I'm not sure if that's needed
>>>>>> now that HCatalog has been integrated with Hive.
>>>>>>
>>>>>> Try changing hive.security.authorization.manager to its default
>>>>>> value, either the one listed in the Hive doc or the one in the HCatalog
>>>>>> doc:
>>>>>>
>>>>>>    -
>>>>>>    org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider
>>>>>>    - org.apache.hcatalog.security.HdfsAuthorizationProvider
>>>>>>
>>>>>> Or you could turn off hive.security.authorization.enabled by setting
>>>>>> it to false (see the HCatalog authorization doc).  There's more information
>>>>>> about this property in the Hive Authorization doc's Prerequisites<https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Authorization#LanguageManualAuthorization-Prerequisites>section.
>>>>>>
>>>>>> Changing configuration property values is documented in Getting
>>>>>> Started (Configuration management overview<https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-Configurationmanagementoverview>)
>>>>>> as well as Configuring Hive<https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration#AdminManualConfiguration-ConfiguringHive>
>>>>>> .
>>>>>>
>>>>>> Agreed, the documentation is confusing.  We're trying to improve
it.
>>>>>>
>>>>>> -- Lefty
>>>>>>
>>>>>>
>>>>>> On Sat, Oct 5, 2013 at 12:53 AM, Baahu <bahubali@gmail.com>
wrote:
>>>>>>
>>>>>>> Lefty,
>>>>>>> Can the install instructions given at
>>>>>>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted
be
>>>>>>> followed as is?
>>>>>>> I see instructions to build from source code just after "Installing
>>>>>>> Hive from a Stable Release" section, which is kinda confusing.I
bump in to
>>>>>>> the error mentioned above while I execute the instructions in
"Running
>>>>>>> Hive" section.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Baahu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Oct 5, 2013 at 1:17 PM, Lefty Leverenz <
>>>>>>> leftyleverenz@gmail.com> wrote:
>>>>>>>
>>>>>>>> Do you have any suggestions for improving the Hive wiki
>>>>>>>> instructions?
>>>>>>>>
>>>>>>>> -- Lefty
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 4, 2013 at 7:30 PM, Baahu <bahubali@gmail.com>
wrote:
>>>>>>>>
>>>>>>>>> Apache hive.
>>>>>>>>> On Oct 5, 2013 1:06 AM, "Mikhail Antonov" <olorinbant@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Are you using apache hive or one from cloudera/Horton
works?
>>>>>>>>>>
>>>>>>>>>> On Oct 4, 2013, at 11:52, Baahu <bahubali@gmail.com>
wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Mikhail,
>>>>>>>>>> I followed the steps mentioned in the Apache hive
website, found
>>>>>>>>>> the instructions not so helpful... Is there any good
link which provides
>>>>>>>>>> proper installation steps.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> On Oct 5, 2013 12:07 AM, "Mikhail Antonov" <olorinbant@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Looks like you have StorageDelegationAuthorizationProvider
>>>>>>>>>>> enabled in hive-site.xml (like below), but hcatalog
isn't installed somehow?
>>>>>>>>>>>
>>>>>>>>>>> <property>
>>>>>>>>>>>   <name>hive.security.authorization.manager</name>
>>>>>>>>>>>   <value>org.apache.hcatalog.security.StorageDelegationAuthorizationProvider</value>
>>>>>>>>>>>   <description>the hive client authorization
manager class name.
>>>>>>>>>>>   The user defined authorization class should
implement interface org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProvider.
>>>>>>>>>>>   HCatalog uses a model, where authorization
checks are delegated to the storage layer (hdfs, hbase, ...).
>>>>>>>>>>>   </description>
>>>>>>>>>>> </property>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/10/4 Baahu <bahubali@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> Have installed Hive 0.11 in centos.I see
the below error when I
>>>>>>>>>>>> execute hive.Can you please help me resolving
this.
>>>>>>>>>>>>
>>>>>>>>>>>> [root@centosvm bin]# hive
>>>>>>>>>>>> 13/10/04 09:05:35 WARN conf.HiveConf: DEPRECATED:
Configuration
>>>>>>>>>>>> property hive.metastore.local no longer has
any effect. Make sure to
>>>>>>>>>>>> provide a valid value for hive.metastore.uris
if you are connecting to a
>>>>>>>>>>>> remote metastore.
>>>>>>>>>>>>
>>>>>>>>>>>> Logging initialized using configuration in
>>>>>>>>>>>> jar:file:/opt/hive-0.11.0-bin/lib/hive-common-0.11.0.jar!/hive-log4j.properties
>>>>>>>>>>>> Hive history
>>>>>>>>>>>> file=/tmp/root/hive_job_log_root_8553@centosvm_201310040905_1657489660.txt
>>>>>>>>>>>> Exception in thread "main" java.lang.RuntimeException:
>>>>>>>>>>>> org.apache.hadoop.hive.ql.metadata.HiveException:
>>>>>>>>>>>> java.lang.ClassNotFoundException:
>>>>>>>>>>>> org.apache.hcatalog.security.StorageDelegationAuthorizationProvider
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:278)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
>>>>>>>>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>>>> Method)
>>>>>>>>>>>>         at
>>>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>>>>>>         at
>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
>>>>>>>>>>>> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
>>>>>>>>>>>> java.lang.ClassNotFoundException:
>>>>>>>>>>>> org.apache.hcatalog.security.StorageDelegationAuthorizationProvider
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:342)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:272)
>>>>>>>>>>>>         ... 7 more
>>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>>>> org.apache.hcatalog.security.StorageDelegationAuthorizationProvider
>>>>>>>>>>>>         at
>>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>>>>>>>>>>         at
>>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>>>>>> Method)
>>>>>>>>>>>>         at
>>>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>>>         at java.lang.Class.forName0(Native
Method)
>>>>>>>>>>>>         at java.lang.Class.forName(Class.java:270)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:335)
>>>>>>>>>>>>         ... 8 more
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Michael Antonov
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Twitter:http://twitter.com/Baahu
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Twitter:http://twitter.com/Baahu
>>>>
>>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
> Twitter:http://twitter.com/Baahu
>
>


-- 
Nitin Pawar

Mime
View raw message