hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Vyas <jayunit...@gmail.com>
Subject Re: Hive, datanucleus, jdbc, localmode.
Date Sun, 29 Dec 2013 02:18:38 GMT
Anyone try this yet :

http://hadoop-pig-hive-thejas.blogspot.com/2013/04/running-hive-in-local-mode.html

?


On Sat, Dec 28, 2013 at 8:09 PM, Jay Vyas <jayunit100@gmail.com> wrote:

> -Local mode should have clear instructions on how to run fully local hive
> jobs, with no hadoop installation.
>
> -I like the hive_test repo but I'm not yet sure hive_test is 100% up to
> date with the simplest strategy for testing hive workflows on the JVM.
>
>
> On Dec 28, 2013, at 4:19 PM, Lefty Leverenz <leftyleverenz@gmail.com>
> wrote:
>
> This sounds like something the documentation should cover.
>
>    - What information should be added to the Local Mode<https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-LocalMetastore>section?
>    - Should the wiki have a link to hive_test<https://github.com/edwardcapriolo/hive_test>
(for
>    example, in Hive Developer FAQ<https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ>
>    )?
>
>
> -- Lefty
>
>
> On Sat, Dec 28, 2013 at 8:02 AM, Edward Capriolo <edlinuxguru@gmail.com>wrote:
>
>> I do not think so. Local mode, just implies the job tracker is local (and
>> some of the temp storage directories) it does not imply hive will use
>> hadoop without forking.
>>
>>
>> On Sat, Dec 28, 2013 at 10:43 AM, Jay Vyas <jayunit100@gmail.com> wrote:
>>
>>> Thanks... But are you sure this is the only way? Or is there some magic
>>> way to run hive in local mode that we both are missing out on ?:)...
>>>
>>> - isn't hive in local mode supposed to be run simply via the jdbc://hive
>>> URL which runs local mode... Or maybe by the fork config parameter?
>>>
>>> - For example see the parameters in this file:
>>>
>>>
>>> https://github.com/riptano/brisk/blob/master/resources/hive/conf/hive-site.xml
>>>
>>>
>>> On Dec 28, 2013, at 10:22 AM, Edward Capriolo <edlinuxguru@gmail.com>
>>> wrote:
>>>
>>> You can follow along to what I do here.
>>>
>>> https://github.com/edwardcapriolo/hive_test
>>>
>>> Essentially hive requires a HADOOP_HOME because it always wants to fork
>>> a bin/hadoop process. Hive-test helps you unpack hadoop inside target and
>>> change your hadoop_home to some other directory.
>>>
>>> It would be nice if there was some other way to do this.
>>>
>>>
>>> On Fri, Dec 27, 2013 at 10:27 PM, Jay Vyas <jayunit100@gmail.com> wrote:
>>>
>>>> Hi Hive:
>>>>
>>>> I'm attempting to create a robust eclipse based dev environment for
>>>> testing my hive jobs in localmode however I run into classnotfound errors
>>>> depending on which maven dependencies I use. Also, it seems when I change
>>>> these dependencies from hive 0.12 to hive 0.11, I get other errors related
>>>> to hive trying to launch jobs via calling /usr/bin/hadoop.
>>>>
>>>> This I am stuck: I can't run hive 12 in local java mode because of
>>>> subtle datanucleus class and API inconsistencies which are tough to
>>>> resolve, and when going to hive 11, it seems local mode is not natively
>>>> detected via the jdbc URL...
>>>>
>>>> So I have 2 questions:
>>>>
>>>> 0) how does hive 12 versus 11 implement local mode differently ?
>>>>
>>>> And
>>>>
>>>> 1) What is the right way to in hive in pure java/ local environments?
>>>>
>>>> The hive book suggests modifying configuration properties, for local
>>>> mode..
>>>>
>>>> but I also have found  that in hive 0.12 , using the jdbc://hive
>>>> connection URL automagically launches jobs in local mode..
>>>>
>>>> However in 0.11 , I see calls to /usr/bin/hadoop when running java
>>>> classes in local eclipse environment.
>>>>
>>>> Thanks!
>>>>
>>>> FYI to see an example of my pom.xml, you can checkout the
>>>> github://jayunit100/bigpetstore pom.xml file.
>>>>
>>>
>>>
>>
>


-- 
Jay Vyas
http://jayunit100.blogspot.com

Mime
View raw message