hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Pawar <nitinpawar...@gmail.com>
Subject Re: Need help with Installation hive 0.11
Date Mon, 07 Oct 2013 09:59:30 GMT
So here is what I did quickly. (I did not follow the step by step guide
from hive wiki)
1) I already have a running hadoop cluster
2) Downloaded the bin-tar.gz from hive.apache.org links
3) Uncompressed the binaries
4) Inside the conf folder there are few .template files (I don't know there
is no hive-site.xml there. In core hadoop projects the core-site.xml is
empty but its present)
5) I just copied all the template files non template files (eg.
hive-defaults.xml.template to hive-defaults.xml and so on)
6) I created a a new hive-site.xml file
7) added following properties to it (which are kind of missing from
hive-defaults but we will talk on dev forums for that)

   - javax.jdo.option.ConnectionURL - I use mysql as my meta store so i
   have pointed this to
   jdbc:mysql://<myhost>:<myport>/<mydbname>?createDatabaseIfNotExist=true
   - javax.jdo.option.ConnectionDriverName - this is set to
   com.mysql.jdbc.driver  I have downloaded mysql driver from oracle and
   stored in hive lib folder.
   - javax.jdo.option.ConnectionUserName - this is my db username
   - javax.jdo.option.ConnectionPassword - this is password for the
   username i used above

8) export HADOOP_HOME variable to where my hadoop installation is present
9) export HIVE_HOME to the directory where I have extracted the tar ball
10) Granted access to new machine on mysql db
11) started hive metastore and verified show tables.

This worked. Of course this does not take care of security on hive yet.

if run into any issues after this do let me know.


On Sat, Oct 5, 2013 at 7:50 PM, Nitin Pawar <nitinpawar432@gmail.com> wrote:

> Baahu,
>
> hive has two conf files. One is hive-site.xml and other is hive-env.sh
>
> From hive-0.11 hcat is bundled with hive so you specially need not do
> anything special to setup hcat.
> you just need to setup few environment variables like HIVE_HOME,
> HIVE_CONF_DIR, HADOOP_HOME.
>
> and then if you have setup the hive-site.xml correctly then it should work
> out of the box.
>
> If you have downloaded the already built tarball. you need not compile it.
> Plus its really a long build if you run it with unit test cases etc.
>
> If possible tonight I will  setup a vm and test out the steps mentioned on
> hive wiki.
>
>
> On Sat, Oct 5, 2013 at 7:21 PM, Baahu <bahubali@gmail.com> wrote:
>
>> Mikhail,
>> The hive-site.xml was missing and while googling I found some suggestion
>> of copying hive-0.11.0-bin/hcatalog/etc/hcatalog/proto-hive-site.xml as
>> hive-site.xml to the directory hive-0.11.0-bin/conf .
>> Was this a right instruction, if not can you please let me know where I
>> can find hive-conf.xml file.
>>
>> Should we really have to build hive as mentioned in the below link during
>> installation? Just untarring the hive tar ball doesn't lead to its
>> installation?
>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted
>>
>> Thanks,
>> Baahu
>>
>>
>>  On Sat, Oct 5, 2013 at 3:26 PM, Mikhail Antonov <olorinbant@gmail.com>wrote:
>>
>>> Bases on original question I decided the author does not need hcatalog
>>> specifically, but got sample hive-site.xml suggesting storage authenticator
>>> somewhere. So why use it?:)
>>>
>>> On Oct 5, 2013, at 2:51, Lefty Leverenz <leftyleverenz@gmail.com> wrote:
>>>
>>> P.S.  I found StorageDelegationAuthorizationProvider mentioned in the
>>> Known Issues section of the HCatalog authorization doc:  "A different
>>> authorization provider (StorageDelegationAuthorizationProvider) needs to be
>>> used for working with HBase tables as well. But that is not well tested
>>> ."
>>>
>>> -- Lefty
>>>
>>>
>>> On Sat, Oct 5, 2013 at 2:44 AM, Lefty Leverenz <leftyleverenz@gmail.com>wrote:
>>>
>>>> I'm out of my depth, but based on existing documentation I'll make some
>>>> suggestions you could try while waiting for replies from people who
>>>> actually know about installation.
>>>>
>>>> Following Mikhail's lead, since this is an HCatalog security problem
>>>> you should start by seeing if HCatalog is installed.  Then you could try
>>>> changing the value of hive.security.authorization.manager -- I found it
>>>> documented in two places:
>>>>
>>>>    - Hive Configuration Properties<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Authentication%2FAuthorization>
>>>>    - HCatalog Storage Based Authorization<https://cwiki.apache.org/confluence/display/Hive/HCatalog+Authorization#HCatalogAuthorization-ConfiguringStorageSystemBasedAuthorization>
>>>>
>>>> The HCatalog authorization doc says you need to add the HCatalog jar to
>>>> the Hive classpath (add "export HIVE_AUX_JARS_PATH=<*path to hcatalog
>>>> jar*>" to hive-env.sh), but I'm not sure if that's needed now that
>>>> HCatalog has been integrated with Hive.
>>>>
>>>> Try changing hive.security.authorization.manager to its default value,
>>>> either the one listed in the Hive doc or the one in the HCatalog doc:
>>>>
>>>>    -
>>>>    org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider
>>>>    - org.apache.hcatalog.security.HdfsAuthorizationProvider
>>>>
>>>> Or you could turn off hive.security.authorization.enabled by setting it
>>>> to false (see the HCatalog authorization doc).  There's more information
>>>> about this property in the Hive Authorization doc's Prerequisites<https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Authorization#LanguageManualAuthorization-Prerequisites>section.
>>>>
>>>> Changing configuration property values is documented in Getting Started
>>>> (Configuration management overview<https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-Configurationmanagementoverview>)
>>>> as well as Configuring Hive<https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration#AdminManualConfiguration-ConfiguringHive>
>>>> .
>>>>
>>>> Agreed, the documentation is confusing.  We're trying to improve it.
>>>>
>>>> -- Lefty
>>>>
>>>>
>>>> On Sat, Oct 5, 2013 at 12:53 AM, Baahu <bahubali@gmail.com> wrote:
>>>>
>>>>> Lefty,
>>>>> Can the install instructions given at
>>>>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted be
>>>>> followed as is?
>>>>> I see instructions to build from source code just after "Installing
>>>>> Hive from a Stable Release" section, which is kinda confusing.I bump
in to
>>>>> the error mentioned above while I execute the instructions in "Running
>>>>> Hive" section.
>>>>>
>>>>> Thanks,
>>>>> Baahu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Oct 5, 2013 at 1:17 PM, Lefty Leverenz <
>>>>> leftyleverenz@gmail.com> wrote:
>>>>>
>>>>>> Do you have any suggestions for improving the Hive wiki instructions?
>>>>>>
>>>>>> -- Lefty
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 4, 2013 at 7:30 PM, Baahu <bahubali@gmail.com>
wrote:
>>>>>>
>>>>>>> Apache hive.
>>>>>>> On Oct 5, 2013 1:06 AM, "Mikhail Antonov" <olorinbant@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Are you using apache hive or one from cloudera/Horton works?
>>>>>>>>
>>>>>>>> On Oct 4, 2013, at 11:52, Baahu <bahubali@gmail.com>
wrote:
>>>>>>>>
>>>>>>>> Hi Mikhail,
>>>>>>>> I followed the steps mentioned in the Apache hive website,
found
>>>>>>>> the instructions not so helpful... Is there any good link
which provides
>>>>>>>> proper installation steps.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> On Oct 5, 2013 12:07 AM, "Mikhail Antonov" <olorinbant@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Looks like you have StorageDelegationAuthorizationProvider
>>>>>>>>> enabled in hive-site.xml (like below), but hcatalog isn't
installed somehow?
>>>>>>>>>
>>>>>>>>> <property>
>>>>>>>>>   <name>hive.security.authorization.manager</name>
>>>>>>>>>   <value>org.apache.hcatalog.security.StorageDelegationAuthorizationProvider</value>
>>>>>>>>>   <description>the hive client authorization manager
class name.
>>>>>>>>>   The user defined authorization class should implement
interface org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProvider.
>>>>>>>>>   HCatalog uses a model, where authorization checks are
delegated to the storage layer (hdfs, hbase, ...).
>>>>>>>>>   </description>
>>>>>>>>> </property>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/10/4 Baahu <bahubali@gmail.com>
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> Have installed Hive 0.11 in centos.I see the below
error when I
>>>>>>>>>> execute hive.Can you please help me resolving this.
>>>>>>>>>>
>>>>>>>>>> [root@centosvm bin]# hive
>>>>>>>>>> 13/10/04 09:05:35 WARN conf.HiveConf: DEPRECATED:
Configuration
>>>>>>>>>> property hive.metastore.local no longer has any effect.
Make sure to
>>>>>>>>>> provide a valid value for hive.metastore.uris if
you are connecting to a
>>>>>>>>>> remote metastore.
>>>>>>>>>>
>>>>>>>>>> Logging initialized using configuration in
>>>>>>>>>> jar:file:/opt/hive-0.11.0-bin/lib/hive-common-0.11.0.jar!/hive-log4j.properties
>>>>>>>>>> Hive history
>>>>>>>>>> file=/tmp/root/hive_job_log_root_8553@centosvm_201310040905_1657489660.txt
>>>>>>>>>> Exception in thread "main" java.lang.RuntimeException:
>>>>>>>>>> org.apache.hadoop.hive.ql.metadata.HiveException:
>>>>>>>>>> java.lang.ClassNotFoundException:
>>>>>>>>>> org.apache.hcatalog.security.StorageDelegationAuthorizationProvider
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:278)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
>>>>>>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>> Method)
>>>>>>>>>>         at
>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>>>>         at
>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
>>>>>>>>>> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
>>>>>>>>>> java.lang.ClassNotFoundException:
>>>>>>>>>> org.apache.hcatalog.security.StorageDelegationAuthorizationProvider
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:342)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:272)
>>>>>>>>>>         ... 7 more
>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>> org.apache.hcatalog.security.StorageDelegationAuthorizationProvider
>>>>>>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>>>>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>>>> Method)
>>>>>>>>>>         at
>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>         at java.lang.Class.forName0(Native Method)
>>>>>>>>>>         at java.lang.Class.forName(Class.java:270)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:335)
>>>>>>>>>>         ... 8 more
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks,
>>>>>>>>> Michael Antonov
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Twitter:http://twitter.com/Baahu
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Twitter:http://twitter.com/Baahu
>>
>>
>
>
> --
> Nitin Pawar
>



-- 
Nitin Pawar

Mime
View raw message