hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastien FLAESCH ...@4js.com>
Subject Re: Compiling Hive ODBC
Date Tue, 30 Apr 2013 15:51:41 GMT
So... could finally compile Hive ODBC by patching the Thrift.h header...

I think there is some configure header issue with the HAVE_* constants
the are defined in the <THRIFT_HOME>/include/thrift/config.h header.

It will only include the required headers if HAVE_CONFIG_H is defined:

#define HAVE_CONFIG_H 1      <--------- added line
#ifdef HAVE_CONFIG_H
#include "config.h"
#endif

Or is there a way to pass this define to ant ?

Seb

On 04/30/2013 04:39 PM, Sebastien FLAESCH wrote:
> FYI:
>
> hive@orca:~/hive-0.10.0/src$ g++ --version
> g++ (Debian 4.4.5-8) 4.4.5
>
> Seb
>
> On 04/30/2013 04:13 PM, Sebastien FLAESCH wrote:
>> Making some progress...
>>
>> After disabling some options to build thrift - because it installs some
>> files in the system directory (/usr/lib/php), and I - do not - want that,
>> I did following configure for thrift:
>>
>> ./configure --prefix=/home/hive/thrift-0.9.0 \
>> --with-qt4=no \
>> --with-csharp=no \
>> --with-erlang=no \
>> --with-python=no \
>> --with-perl=no \
>> --with-php=no \
>> --with-php_extension=no \
>> --with-ruby=no \
>> --with-haskell=no \
>> --with-go=no \
>> --with-d=no
>>
>> Then I could build thrift and fb303, but not when I try to build Hive
>> ODBC:
>>
>> hive@orca:~/hive-0.10.0/src$ ant compile-cpp -Dthrift.home=$THRIFT_HOME
>> ...
>> ...
>> ...
>> [exec] g++ -Wall -g -fPIC -m32 -DARCH32
>> -I/home/hive/thrift-0.9.0/include/thrift
>> -I/home/hive/thrift-0.9.0/include/thrift/fb303 -I/include
>> -I/home/hive/hive-0.10.0/src/service/src/gen/thrift/gen-cpp
>> -I/home/hive/hive-0.10.0/src/ql/src/gen/thrift/gen-cpp
>> -I/home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp
>> -I/home/hive/hive-0.10.0/src/odbc/src/cpp -c
>> /home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
>>
>> -o /home/hive/hive-0.10.0/src/build/metastore/objs/ThriftHiveMetastore.o
>> [exec] In file included from
>> /usr/local/include/thrift/protocol/TProtocol.h:23,
>> [exec] from /usr/local/include/thrift/TProcessor.h:24,
>> [exec] from /usr/local/include/thrift/TDispatchProcessor.h:22,
>> [exec] from
>> /home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h:10,
>>
>>
>> [exec] from
>> /home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp:7:
>>
>>
>> [exec] /usr/local/include/thrift/transport/TTransport.h:34: error:
>> expected constructor, destructor, or type conversion before ‘readAll’
>> [exec] /usr/local/include/thrift/transport/TTransport.h:107: error:
>> ‘uint32_t’ does not name a type
>> [exec] /usr/local/include/thrift/transport/TTransport.h:111: error:
>> ‘uint32_t’ does not name a type
>> [exec] /usr/local/include/thrift/transport/TTransport.h:124: error:
>> ‘uint32_t’ does not name a type
>> [exec] /usr/local/include/thrift/transport/TTransport.h:128: error:
>> ‘uint32_t’ does not name a type
>> [exec] /usr/local/include/thrift/transport/TTransport.h:139: error:
>> ‘uint32_t’ does not name a type
>>
>>
>> Any idea?
>>
>> Thanks
>> Seb
>>
>> On 04/30/2013 11:00 AM, Sebastien FLAESCH wrote:
>>> Thank you Carl, but I still have problems to build Hive ODBC:
>>>
>>> I followed the instruction from this page (I believe the doc is wrong):
>>>
>>> https://cwiki.apache.org/Hive/hiveodbc.html
>>>
>>> Where it says:
>>>
>>> "Build the Hive client by running the following command from
>>> HIVE_HOME..."
>>>
>>> Here is my env:
>>>
>>> ----------------------------------------------------------
>>> hive@orca:~$ cat env-apache.sh
>>> JAVA_HOME="/usr/lib/jvm/java-6-sun"
>>> export JAVA_HOME
>>> JRE_HOME="$JAVA_HOME/jre"
>>> export JRE_HOME
>>> PATH=$JAVA_HOME/bin:$PATH
>>>
>>> HADOOP_HOME=/home/hive/hadoop-0.23.7
>>> export HADOOP_HOME
>>> PATH=$HADOOP_HOME/bin:$PATH
>>> export PATH
>>>
>>> HIVE_HOME=/home/hive/hive-0.10.0
>>> export HIVE_HOME
>>> PATH=$HIVE_HOME/bin:$PATH
>>> export PATH
>>>
>>> THRIFT_HOME=/home/hive/thrift-0.9.0
>>> export THRIFT_HOME
>>> ----------------------------------------------------------
>>>
>>> But:
>>>
>>> ----------------------------------------------------------
>>> hive@orca:~$ cd $HIVE_HOME
>>> hive@orca:~/hive-0.10.0$ ant compile-cpp -Dthrift.home=$THRIFT_HOME
>>> Buildfile: build.xml does not exist!
>>> Build failed
>>> ----------------------------------------------------------
>>>
>>>
>>> And when I go to the src directory, it goes further, but then
>>> I have missing Thrift header files:
>>>
>>> ----------------------------------------------------------
>>> hive@orca:~/hive-0.10.0$ cd src
>>> hive@orca:~/hive-0.10.0/src$ ant compile-cpp -Dthrift.home=$THRIFT_HOME
>>> ...
>>> compile-cpp:
>>> [echo] Project: odbc
>>> [exec] mkdir -p /home/hive/hive-0.10.0/src/build/metastore/objs
>>> [exec] g++ -Wall -g -fPIC -m32 -DARCH32
>>> -I/home/hive/thrift-0.9.0/include/thrift
>>> -I/home/hive/thrift-0.9.0/include/thrift/fb303 -I/include
>>> -I/home/hive/hive-0.10.0/src/service/src/gen/thrift/gen-cpp
>>> -I/home/hive/hive-0.10.0/src/ql/src/gen/thrift/gen-cpp
>>> -I/home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp
>>> -I/home/hive/hive-0.10.0/src/odbc/src/cpp -c
>>> /home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
>>>
>>>
>>> -o /home/hive/hive-0.10.0/src/build/metastore/objs/ThriftHiveMetastore.o
>>> [exec] In file included from
>>> /home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp:7:
>>>
>>>
>>>
>>> [exec]
>>> /home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h:10:39:
>>>
>>>
>>> error: thrift/TDispatchProcessor.h: No such file or directory
>>> [exec] In file included from
>>> /home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h:11,
>>>
>>>
>>>
>>> [exec] from
>>> /home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp:7:
>>>
>>>
>>>
>>> [exec]
>>> /home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h:10:27:
>>>
>>>
>>> error: thrift/Thrift.h: No such file or directory
>>> [exec]
>>> /home/hive/hive-0.10.0/src/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h:11:42:
>>>
>>>
>>> error: thrift/TApplicationException.h: No such file or directory
>>> ...
>>> ----------------------------------------------------------
>>>
>>> In fact there is no such directory:
>>>
>>> /home/hive/thrift-0.9.0/include/thrift/fb303
>>>
>>> I see only:
>>>
>>> ----------------------------------------------------------
>>> hive@orca:~/hive-0.10.0/src$ ls /home/hive/thrift-0.9.0/include/thrift
>>> c_glib
>>> ----------------------------------------------------------
>>>
>>> According to:
>>>
>>> http://mail-archives.apache.org/mod_mbox/hive-user/201110.mbox/%3C556325346CA26341B6F0530E07F90D960169F0FD464D@GBGH-EXCH-CMS.sig.ads%3E
>>>
>>>
>>>
>>>
>>> I need to build fb303 component of thrift...
>>>
>>> So I tried to build "fb303" in the thrift sources.
>>>
>>> Note that I ran thrift's configure with a non-system installation target
>>> directory:
>>>
>>> ----------------------------------------------------------
>>> hive@orca:~$ cd tmp/thrift-0.9.0/contrib/fb303
>>> hive@orca:~/tmp/thrift-0.9.0/contrib/fb303$ ./configure
>>> --prefix=/home/hive/thrift-0.9.0
>>> ----------------------------------------------------------
>>>
>>> But it could not find /usr/local/bin/thrift and
>>> /usr/local/include/thrift...
>>>
>>> So I created links:
>>>
>>> ----------------------------------------------------------
>>> /usr/local/bin/thrift -> /home/hive/thrift-0.9.0/bin/thrift
>>> /usr/local/include/thrift -> /home/hive/thrift-0.9.0/include/thrift
>>> ----------------------------------------------------------
>>>
>>> But:
>>>
>>> ----------------------------------------------------------
>>> hive@orca:~/tmp/thrift-0.9.0/contrib/fb303$ make
>>>
>>> make all-recursive
>>> make[1]: Entering directory `/home/hive/tmp/thrift-0.9.0/contrib/fb303'
>>> Making all in .
>>> make[2]: Entering directory `/home/hive/tmp/thrift-0.9.0/contrib/fb303'
>>> make[2]: Nothing to be done for `all-am'.
>>> make[2]: Leaving directory `/home/hive/tmp/thrift-0.9.0/contrib/fb303'
>>> Making all in cpp
>>> make[2]: Entering directory
>>> `/home/hive/tmp/thrift-0.9.0/contrib/fb303/cpp'
>>> make all-am
>>> make[3]: Entering directory
>>> `/home/hive/tmp/thrift-0.9.0/contrib/fb303/cpp'
>>> g++ -DPACKAGE_NAME=\"libfb303\" -DPACKAGE_TARNAME=\"libfb303\"
>>> -DPACKAGE_VERSION=\"20080209\" -DPACKAGE_STRING=\"libfb303\ 20080209\"
>>> -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DHAVE_BOOST=/\*\*/
>>> -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1
>>> -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1
>>> -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1
>>> -DHAVE_INTTYPES_H=1 -DHAVE_NETINET_IN_H=1 -I. -I.. -Igen-cpp
>>> -I/usr/local/include/thrift -I/usr/include -Wall -O3 -MT
>>> FacebookService.o -MD -MP -MF .deps/FacebookService.Tpo -c -o
>>> FacebookService.o `test -f 'gen-cpp/FacebookService.cpp' || echo
>>> './'`gen-cpp/FacebookService.cpp
>>> In file included from gen-cpp/FacebookService.cpp:7:
>>> gen-cpp/FacebookService.h:10:39: error: thrift/TDispatchProcessor.h: No
>>> such file or directory
>>> In file included from gen-cpp/FacebookService.h:11,
>>> from gen-cpp/FacebookService.cpp:7:
>>>
>>> ----------------------------------------------------------
>>>
>>> ...
>>>
>>> Seb
>>>
>>>
>>> On 04/29/2013 08:48 PM, Carl Steinbach wrote:
>>>> Hi Sebastien,
>>>>
>>>> Most (if not all) of the ant targets will fail if run from the
>>>> submodule
>>>> directories. You must run the targets from the root source directory
>>>> instead.
>>>>
>>>> Thanks.
>>>>
>>>> Carl
>>>>
>>>>
>>>> On Fri, Apr 26, 2013 at 6:09 AM, Sebastien FLAESCH <sf@4js.com
>>>> <mailto:sf@4js.com>> wrote:
>>>>
>>>> I have now installed Thrift, but I still get the error with the
>>>> "osfamily" task:
>>>>
>>>> Please help!
>>>>
>>>> hive@orca:~/hive-0.10.0/src/__odbc$ ant compile-cpp
>>>> -Dthrift.home=/home/hive/__thrift-0.9.0
>>>>
>>>>
>>>> Buildfile: /home/hive/hive-0.10.0/src/__odbc/build.xml
>>>>
>>>> BUILD FAILED
>>>> /home/hive/hive-0.10.0/src/__odbc/build.xml:30: The following error
>>>> occurred while executing this line:
>>>> /home/hive/hive-0.10.0/src/__build-common.xml:117: Problem: failed
>>>> to create task or type osfamily
>>>> Cause: The name is undefined.
>>>> Action: Check the spelling.
>>>> Action: Check that any custom tasks/types have been declared.
>>>> Action: Check that any <presetdef>/<macrodef> declarations have
>>>> taken place.
>>>>
>>>>
>>>> Total time: 0 seconds
>>>>
>>>> Seb
>>>>
>>>>
>>>> On 04/26/2013 01:40 PM, Sebastien FLAESCH wrote:
>>>>
>>>> Go it, need to install Apache Thrift first ... sorry for this mail.
>>>> Seb
>>>>
>>>> On 04/26/2013 01:33 PM, Sebastien FLAESCH wrote:
>>>>
>>>> Hi all,
>>>>
>>>> Just started with an Hadoop / Hive POC to write a
>>>>
>>>> So far, I have installed:
>>>>
>>>> So far I have downloaded the Hadoop and Hive packages from:
>>>>
>>>> http://hadoop.apache.org/__releases.html
>>>> <http://hadoop.apache.org/releases.html>
>>>> http://hive.apache.org/__releases.html
>>>> <http://hive.apache.org/releases.html>
>>>>
>>>> I took:
>>>>
>>>> http://hadoop.apache.org/__releases.html#18+April%2C+__2013%3A+Release+0.23.7+__available
>>>>
>>>>
>>>>
>>>> <http://hadoop.apache.org/releases.html#18+April%2C+2013%3A+Release+0.23.7+available>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> and
>>>>
>>>> http://hive.apache.org/__releases.html#11+January%2C+__2013%3A+release+0.10.0+__available
>>>>
>>>>
>>>>
>>>> <http://hive.apache.org/releases.html#11+January%2C+2013%3A+release+0.10.0+available>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> After setting the env, it seems that hive is working:
>>>>
>>>> hive@orca:~$ $HIVE_HOME/bin/hive
>>>> ...
>>>> hive> create table t1 ( k int, s string );
>>>> OK
>>>>
>>>> hive> select count(*) from t1;
>>>> Total MapReduce jobs = 1
>>>> ...
>>>> OK
>>>> 0
>>>> Time taken: 7.634 seconds
>>>>
>>>>
>>>> No I want to connect through ODBC...
>>>>
>>>> Following the instructions of this page:
>>>>
>>>> https://cwiki.apache.org/__confluence/display/Hive/__HiveODBC
>>>> <https://cwiki.apache.org/confluence/display/Hive/HiveODBC>
>>>>
>>>> Section "Hive Client Build/Setup", I tried:
>>>>
>>>> hive@orca:~/hive-0.10.0$ cd $HIVE_HOME
>>>> hive@orca:~/hive-0.10.0$ ant compile-cpp
>>>> -Dthrift.home=/home/hive/hive-__0.10.0/odbc
>>>> Buildfile: build.xml does not exist!
>>>> Build failed
>>>>
>>>> Then, from the sources directory src/odbc:
>>>>
>>>> hive@orca:~/hive-0.10.0$ cd src/odbc/
>>>> hive@orca:~/hive-0.10.0/src/__odbc$ ant compile-cpp
>>>> -Dthrift.home=/home/hive/hive-__0.10.0/odbc
>>>> Buildfile: /home/hive/hive-0.10.0/src/__odbc/build.xml
>>>>
>>>> BUILD FAILED
>>>> /home/hive/hive-0.10.0/src/__odbc/build.xml:30: The
>>>> following error
>>>> occurred while executing this line:
>>>> /home/hive/hive-0.10.0/src/__build-common.xml:117: Problem:
>>>> failed to
>>>> create task or type osfamily
>>>> Cause: The name is undefined.
>>>> Action: Check the spelling.
>>>> Action: Check that any custom tasks/types have been declared.
>>>> Action: Check that any <presetdef>/<macrodef> declarations
>>>> have taken
>>>> place.
>>>>
>>>>
>>>> Total time: 0 seconds
>>>>
>>>>
>>>> Can someone help or point me to an up to date documentation?
>>>>
>>>> Thanks!
>>>> Seb
>>>>
>>>>
>>>>
>>>>
>>>
>>
>


Mime
View raw message