Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Parag Sarda <PSarda@walmartlabs.com>
To: "user@hive.apache.org" <user@hive.apache.org>
Date: Tue, 12 Feb 2013 09:31:07 -0800
Subject: Re: How to load hive metadata from conf dir
Thread-Topic: How to load hive metadata from conf dir
Thread-Index: Ac4JRr6Xohq76yLnT46+8Ui9f6dP7A==
Message-ID: <CD4076FE.566D%PSarda@walmartlabs.com>
In-Reply-To: 
 <CAENxBwwhn91FrTB2GCQvdhK3RNdV8QtqT3CxUGSya0zLhPJKsQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.3.1.130117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hive-thrift is definitely best option till now. That said, I am wondering
if its possible to load megastore in local mode[1] to avoid dependency on
external service. Can I read the HIVE_CONF_DIR for javax.jdo.option.*
parameters and talk to sql server hosting hive metadata?

[1] https://cwiki.apache.org/Hive/adminmanual-metastoreadmin.html

Thanks,=20
Parag


On 12/02/13 10:23 PM, "Edward Capriolo" <edlinuxguru@gmail.com> wrote:

>If you use hive-thrift/hive-service you can get the location of a
>table through the Table API (instead of Dean's horrid bash-isms)
>
>http://hive.apache.org/docs/r0.7.0/api/org/apache/hadoop/hive/metastore/ap
>i/ThriftHiveMetastore.Client.html#get_table(java.lang.String,
>java.lang.String)
>
>Table t =3D ....
>t.getSd().getLocation()
>
>
>On Tue, Feb 12, 2013 at 9:41 AM, Dean Wampler
><dean.wampler@thinkbiganalytics.com> wrote:
>> I'll mention another bash hack that I use all the time:
>>
>> hive -e 'some_command' | grep for_what_i_want |
>> sed_command_to_remove_just_i_dont_want
>>
>> For example, the following command will print just the value of
>> hive.metastore.warehouse.dir, sending all the logging junk written to
>>stderr
>> to /dev/null and stripping off the leading
>>"hive.metastore.warehouse.dir=3D"
>> from the stdout output:
>>
>> hive -e 'set hive.metastore.warehouse.dir;' 2> /dev/null | sed -e
>> 's/hive.metastore.warehouse.dir=3D//'
>>
>> (No grep subcommand required in this case...)
>>
>> You could do something similar with DESCRIBE EXTENDED table PARTION(...)
>> Suppose you want a script that works for any property. Put the
>>following in
>> a script file, say hive-prop.sh:
>>
>> #!/bin/sh
>> hive -e "set $1;" 2> /dev/null | sed -e "s/$1=3D//"
>>
>> Make it executable (chmod +x /path/to/hive-prop.sh), then run it this
>>way:
>>
>> /path/to/hive-prop.sh hive.metastore.warehouse.dir
>>
>> Back to asking for for metadata for a table. The following script will
>> determine the location of a particular partition for an external
>> "mydatabase.stocks" table:
>>
>> #!/bin/sh
>> hive -e "describe formatted mydatabase.stocks
>>partition(exchange=3D'NASDAQ',
>> symbol=3D'AAPL');" 2> /dev/null | grep Location | sed -e "s/Location:[
>>\t]*//"
>>
>> dean
>>
>> On Mon, Feb 11, 2013 at 4:59 PM, Parag Sarda <PSarda@walmartlabs.com>
>>wrote:
>>>
>>> Hello Hive Users,
>>>
>>> I am writing a program in java which is bundled as JAR and executed
>>>using
>>> hadoop jar command. I would like to access hive metadata (read
>>>partitions
>>> informations) in this program. I can ask user to set HIVE_CONF_DIR
>>> environment variable before calling my program or ask for any
>>>reasonable
>>> parameters to be passed. I do not want to force user to run hive
>>>megastore
>>> service if possible to increase reliability of program by avoiding
>>> external dependencies.
>>>
>>> What is the recommended way to get partitions information? Here is my
>>> understanding
>>> 1. Make sure my jar is bundled with hive-metastore[1] library.
>>> 2. Use HiveMetastoreClient[2]
>>>
>>> Is this correct? If yes, how to read the hive configuration[3] from
>>> HIVE_CONF_DIR?
>>>
>>> [1] http://mvnrepository.com/artifact/org.apache.hive/hive-metastore
>>> [2]
>>>
>>>=20
>>>http://hive.apache.org/docs/r0.7.1/api/org/apache/hadoop/hive/metastore/
>>>Hiv
>>> eMetaStoreClient.html
>>> [3]
>>>
>>>=20
>>>http://hive.apache.org/docs/r0.7.1/api/org/apache/hadoop/hive/conf/HiveC
>>>onf
>>> .html
>>>
>>> Thanks in advance,
>>> Parag
>>>
>>
>>
>>
>> --
>> Dean Wampler, Ph.D.
>> thinkbiganalytics.com
>> +1-312-339-1330
>>