incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Jurney <russell.jur...@gmail.com>
Subject Re: Booting the metastore?
Date Wed, 20 Jun 2012 21:24:49 GMT
I am able to run hcatalog from sbin/hcat_server.sh, however my hive
metastore won't startup.  I can't make root user's environment get used, no
matter what I set.  I can't bind to the port as user hadoop.  I'm not sure
what to do.

https://gist.github.com/2962304

Googling around, this sounds like a common issue for Ubuntu.  Should I bug
the Hive list?

On Tue, Jun 12, 2012 at 6:24 PM, Russell Jurney <russell.jurney@gmail.com>wrote:

> I've started Hive's metastore before, so I'm not sure why it is crashing -
> other than my symlink?
>
> hadoop@ip-10-4-115-51:~$ sudo bash -x
> /usr/local/hcat/share/hcatalog/scripts/hcat_server_start.sh
> + this=/usr/local/hcat/share/hcatalog/scripts/hcat_server_start.sh
> + '[' -h /usr/local/hcat/share/hcatalog/scripts/hcat_server_start.sh ']'
> ++ dirname /usr/local/hcat/share/hcatalog/scripts/hcat_server_start.sh
> + bin=/usr/local/hcat/share/hcatalog/scripts
> ++ basename /usr/local/hcat/share/hcatalog/scripts/hcat_server_start.sh
> + script=hcat_server_start.sh
> ++ unset CDPATH
> ++ cd /usr/local/hcat/share/hcatalog/scripts
> ++ pwd
> + bin=/usr/local/hcat/share/hcatalog/scripts
> + this=/usr/local/hcat/share/hcatalog/scripts/hcat_server_start.sh
> ++ dirname /usr/local/hcat/share/hcatalog/scripts/hcat_server_start.sh
> + export HCAT_HOME=/usr/local/hcat/share/hcatalog/scripts/..
> + HCAT_HOME=/usr/local/hcat/share/hcatalog/scripts/..
> + . /usr/local/hcat/share/hcatalog/scripts/../conf/hcat-env.sh
> ++ ROOT=/usr/local/hcat
> ++ DBROOT=/home/hadoop/mysql-connector-java-3.1.14
> ++ USER=root
> ++ HADOOP_HOME=/home/hadoop
> ++ export METASTORE_PORT=9933
> ++ METASTORE_PORT=9933
> + PID_FILE=/usr/local/hcat/var/log/hcat.pid
> + SLEEP_TIME_AFTER_START=15
> + '[' -s /usr/local/hcat/var/log/hcat.pid ']'
> + for f in '${DBROOT}/mysql-connector-java-*-bin.jar'
> +
> AUX_CLASSPATH=:/home/hadoop/mysql-connector-java-3.1.14/mysql-connector-java-3.1.14-bin.jar
> + for f in '${ROOT}/lib/*.jar'
> +
> AUX_CLASSPATH=':/home/hadoop/mysql-connector-java-3.1.14/mysql-connector-java-3.1.14-bin.jar:/usr/local/hcat/lib/*.jar'
> + export
> 'AUX_CLASSPATH=:/home/hadoop/mysql-connector-java-3.1.14/mysql-connector-java-3.1.14-bin.jar:/usr/local/hcat/lib/*.jar'
> +
> AUX_CLASSPATH=':/home/hadoop/mysql-connector-java-3.1.14/mysql-connector-java-3.1.14-bin.jar:/usr/local/hcat/lib/*.jar'
> + export HADOOP_HOME=/home/hadoop
> + HADOOP_HOME=/home/hadoop
> ++ date +%Y%m%d%H%M
> + export 'HADOOP_OPTS= -server -XX:+UseConcMarkSweepGC
> -XX:ErrorFile=/usr/local/hcat/var/log/hcat_err_pid%p.log
> -Xloggc:/usr/local/hcat/var/log/hcat_gc.log-201206130121 -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps'
> + HADOOP_OPTS=' -server -XX:+UseConcMarkSweepGC
> -XX:ErrorFile=/usr/local/hcat/var/log/hcat_err_pid%p.log
> -Xloggc:/usr/local/hcat/var/log/hcat_gc.log-201206130121 -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps'
> + export HADOOP_HEAPSIZE=2048
> + HADOOP_HEAPSIZE=2048
> + PID=15287
> + '[' 15287x == x ']'
> + echo Started metastore server init, testing if initialized correctly...
> Started metastore server init, testing if initialized correctly...
> + sleep 15
> + nohup /usr/local/hcat/bin/hive --service metastore
> + ps -p 15287
> + echo 'Metastore startup failed, see /usr/local/hcat/var/log/hcat.err'
> Metastore startup failed, see /usr/local/hcat/var/log/hcat.err
> + exit 1
>
>
> On Tue, Jun 12, 2012 at 4:33 PM, Aniket Mokashi <aniket486@gmail.com>wrote:
>
>> Can you try sudo bash -x /usr/local/hcat/share/
>> hcatalog/scripts/hcat_server_start.sh and paste me stderr/stdout.
>>
>> Thanks,
>> Aniket
>>
>> On Tue, Jun 12, 2012 at 3:48 PM, Russell Jurney <russell.jurney@gmail.com
>> > wrote:
>>
>>> Thanks for your help, I've gotten a bit further.
>>>
>>> I did this:
>>>
>>> [bash]$ wget
>>> http://download.nextag.com/apache/incubator/hcatalog/hcatalog-0.4.0-incubating/hcatalog-src-0.4.0-incubating.tar.gz
>>> [bash]$ tar -xvzf hcatalog-src-0.4.0-incubating.tar.gz
>>> [bash]$ cd hcatalog-src-0.4.0-incubating
>>> [bash]$ mkdir var/log
>>> [bash]$ export HCAT_HOME=/home/hadoop/hcatalog-src-0.4.0-incubating
>>> [bash]$ ant tar -v #will take a long time, be patient
>>> [bash]$ cd build/hcatalog-0.4.0-dev
>>> [bash]$ cp ../../NOTICE.txt .
>>> [bash]$ sudo bash share/hcatalog/scripts/hcat_server_install.sh -r
>>> /usr/local/hcat -d /home/hadoop/mysql-connector-java-3.1.14 -h $HADOOP_HOME
>>> -p 9933
>>> [bash]$ cd
>>> [bash]$ sudo cp hive-0.9.0/conf/hive-site.xml
>>>  /usr/local/hcat/etc/hcatalog/
>>> [bash]$ sudo ln -s /home/hadoop/hive-0.9.0/bin/hive
>>> /usr/local/hcat/bin/hive
>>> [bash]$ sudo bash
>>> /usr/local/hcat/share/hcatalog/scripts/hcat_server_start.sh
>>>
>>>
>>> And I now get this:
>>>
>>> hadoop@ip-10-4-115-51:~$ sudo bash
>>> /usr/local/hcat/share/hcatalog/scripts/hcat_server_start.sh
>>> Started metastore server init, testing if initialized correctly...
>>> Metastore startup failed, see /usr/local/hcat/var/log/hcat.err
>>> hadoop@ip-10-4-115-51:~$ cat /usr/local/hcat/var/log/hcat.err
>>> <empty>
>>>
>>>
>>> I'm going to circle around with the directions again and see if I missed
>>> anything.
>>>
>>>
>>> On Mon, Jun 11, 2012 at 8:28 PM, Aniket Mokashi <aniket486@gmail.com>wrote:
>>>
>>>> The steps are little unclear on the document
>>>>
>>>> Here is what I would suggest you to try-
>>>>
>>>>  1. from share location under build/hcatalog-$version, run
>>>> share/hcatalog/scripts/hcat_server_install.sh -r /usr/local/hcat -d
>>>> /home/hadoop/mysql-connector-java-3.1.14 -h $HADOOP_HOME -p 9933 (I think
>>>> it somehow mandates you to run it with that path).
>>>> 2. copy your hive-site.xml under /usr/local/hcat/etc/hcatalog/ (or
>>>> wherever the hive-default.xml is).
>>>> 3. run /usr/local/hcat/share/hcatalog/scripts/hcat_server_start.sh
>>>>
>>>> In general, /usr/local/hcat can be any directory. install script is
>>>> essentially a set of copy commands (along with generating a config.sh file
>>>> from template) and you need to configure installation under
>>>> /usr/local/hcat.
>>>>
>>>> (I may have mispelled shell script names)
>>>>
>>>> Thanks,
>>>> Aniket
>>>>
>>>> On Mon, Jun 11, 2012 at 7:58 PM, Russell Jurney <
>>>> russell.jurney@gmail.com> wrote:
>>>>
>>>>> I've gotten a little further after reading the installation
>>>>> instructions at
>>>>> http://incubator.apache.org/hcatalog/docs/r0.4.0/install.html and
>>>>> patching https://issues.apache.org/jira/browse/HCATALOG-429
>>>>>
>>>>> Now I am stuck at:
>>>>>
>>>>> [bash]$ sudo mkdir /usr/local/hcat
>>>>> [bash]$ sudo ./scripts/hcat_server_install.sh -r /usr/local/hcat -d
>>>>> /home/hadoop/mysql-connector-java-3.1.14 -h $HADOOP_HOME -p 9933
>>>>>
>>>>> Installing into [/usr/local/hcat]
>>>>> cp: cannot stat `./etc/*': No such file or directory
>>>>> cp: cannot stat `./libexec/*': No such file or directory
>>>>> cp: cannot stat `./sbin/*': No such file or directory
>>>>> cp: cannot stat `./share/*': No such file or directory
>>>>> ln: creating symbolic link `/usr/local/hcat/share/hcatalog/conf': No
>>>>> such file or directory
>>>>> ./scripts/hcat_server_install.sh: line 146:
>>>>> /usr/local/hcat/etc/hcatalog/hcat-env.sh: No such file or directory
>>>>> Installation successful
>>>>>
>>>>>
>>>>> Will patch as needed.
>>>>>
>>>>> On Mon, Jun 11, 2012 at 6:31 PM, Russell Jurney <
>>>>> russell.jurney@gmail.com> wrote:
>>>>>
>>>>>> I am running HCatalog for the first time.  My first issue was
>>>>>> https://issues.apache.org/jira/browse/HCATALOG-428
>>>>>>
>>>>>> Once I create var/log, and re-run bin/hcat_server.sh start I get:
>>>>>>
>>>>>> Exception in thread "main" java.lang.NumberFormatException: For input
>>>>>> string: ""
>>>>>> at
>>>>>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
>>>>>>  at java.lang.Integer.parseInt(Integer.java:470)
>>>>>> at java.lang.Integer.parseInt(Integer.java:499)
>>>>>>  at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HiveMetastoreCli.parse(HiveMetaStore.java:2902)
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:2913)
>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>  at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>  at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>>
>>>>>>
>>>>>> My relevant ENV follows (had to set HIVE_HOME):
>>>>>>
>>>>>> TERM=xterm-color
>>>>>> SHELL=/bin/bash
>>>>>> HADOOP_HOME=/home/hadoop
>>>>>> OLDPWD=/home/hadoop
>>>>>> SSH_TTY=/dev/pts/0
>>>>>> USER=hadoop
>>>>>> LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/lib:
>>>>>> MAIL=/var/mail/hadoop
>>>>>>
>>>>>> PATH=/usr/local/cuda/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/hadoop/bin
>>>>>> HIVE_HOME=/home/hadoop/hive-0.9.0
>>>>>> PWD=/home/hadoop/hcatalog-src-0.4.0-incubating
>>>>>> JAVA_HOME=/usr/lib/jvm/java-6-sun
>>>>>> LANG=en_US.UTF-8
>>>>>> SHLVL=1
>>>>>> HOME=/home/hadoop
>>>>>> LOGNAME=hadoop
>>>>>> HADOOP_HOME_WARN_SUPPRESS=true
>>>>>>
>>>>>> What do I do? :)
>>>>>>
>>>>>> --
>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>> datasyndrome.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>> datasyndrome.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> "...:::Aniket:::... Quetzalco@tl"
>>>>
>>>
>>>
>>>
>>> --
>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome
>>> .com
>>>
>>
>>
>>
>> --
>> "...:::Aniket:::... Quetzalco@tl"
>>
>
>
>
> --
> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.
> com
>



-- 
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com

Mime
View raw message