hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: isSplitable() deprecated
Date Fri, 15 Jan 2010 21:21:09 GMT
When you are seeing errors early in the configure phase it is possible
that you do not have a complete development environment.  gcc alone is
not enough.

Give this a whirl:

yum groupinstall 'Development Tools'

This is going to install many things but at least you wont be chasing
down dependencies 1 by 1.

Edward

On Fri, Jan 15, 2010 at 4:17 PM, Todd Lipcon <todd@cloudera.com> wrote:
> Are you starting from a clean tarball of the lzo stuff?
>
> Can you make sure your /tmp/ partition isn't full?
>
> There should be a config.log file hanging around somewhere also - perhaps
> this would be helpful.
>
> -Todd
>
> On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> Todd:
>> Thanks for the continued support.
>>
>> I installed lzo-devel:
>> [root@tyu-linux batchclient]# rpm -ivh
>> /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
>> warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
>> signature: NOKEY, key ID 217521f6
>> Preparing...                ###########################################
>> [100%]
>>    1:lzo-devel              ###########################################
>> [100%]
>>
>> But I still see:
>>     [exec] Try `/usr/bin/install --help' for more information.
>>     [exec] checking for gcc... gcc
>>      [exec] checking for C compiler default output file name...
>>     [exec] configure: error: C compiler cannot create executables
>>
>>
>> On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>
>> > Hi Ted,
>> >
>> > Did you also install liblzo-devel?
>> >
>> > Here are the packages I install for LZO:
>> >
>> > lzo-2.02-2.el5.1
>> > lzo-devel-2.02-2.el5.1
>> >
>> > -Todd
>> >
>> > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>> >
>> > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest
>> version
>> > > Nothing to do
>> > >
>> > > [root@tyu-linux batchclient]# yum install gcc-c++
>> > > Loaded plugins: fastestmirror
>> > > Loading mirror speeds from cached hostfile
>> > >  * base: mirrors.kernel.org
>> > >  * updates: updates.interworx.info
>> > >  * addons: mirrors.kernel.org
>> > >  * extras: mirrors.netdna.com
>> > > Setting up Install Process
>> > > Parsing package install arguments
>> > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
>> > > version
>> > >
>> > >
>> > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <todd@cloudera.com>
>> wrote:
>> > >
>> > > > Make sure you have gcc, g++ installed?
>> > > >
>> > > > -Todd
>> > > >
>> > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yuzhihong@gmail.com>
>> wrote:
>> > > >
>> > > > > I installed
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
>> > > > > .
>> > > > >
>> > > > > [root@tyu-linux software]# rpm -e lzo
>> > > > > [root@tyu-linux software]# rpm -ivh
>> > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
>> > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3
DSA
>> > > > signature:
>> > > > > NOKEY, key ID 217521f6
>> > > > > Preparing...
>> >  ###########################################
>> > > > > [100%]
>> > > > >   1:lzo
>> >  ###########################################
>> > > > > [100%]
>> > > > >
>> > > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
>> > > > > lrwxrwxrwx 1 root root     16 Jan 12 10:29 /usr/lib64/liblzo2.so.2
>> ->
>> > > > > liblzo2.so.2.0.0
>> > > > > -rwxr-xr-x 1 root root 123408 May  6  2007
>> > /usr/lib64/liblzo2.so.2.0.0
>> > > > >
>> > > > > I still see this error:
>> > > > >      [exec] configure: error: C compiler cannot create executables
>> > > > >     [exec] See `config.log' for more details.
>> > > > >
>> > > > > BUILD FAILED
>> > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
>> > returned:
>> > > > 77
>> > > > >
>> > > > >
>> > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <todd@cloudera.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi Ted,
>> > > > > >
>> > > > > > Please make sure you have version 2.02 of liblzo installed.
>> There's
>> > > an
>> > > > > > earlier version in RHEL that doesn't fail properly at the
>> configure
>> > > > stage
>> > > > > > and I've seen it produce the error you pasted below.
>> > > > > >
>> > > > > > -Todd
>> > > > > >
>> > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yuzhihong@gmail.com>
>> > > wrote:
>> > > > > >
>> > > > > > > I followed
>> > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
>> > > > > > >
>> > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed
and
>> > > latest
>> > > > > > > version
>> > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
06:36:37
>> EDT
>> > > > 2009
>> > > > > > > x86_64 x86_64 x86_64 GNU/Linux
>> > > > > > >
>> > > > > > > For 32 bit build, I got:
>> > > > > > >     [exec] checking for C compiler default output
file name...
>> > > > > > >     [exec] configure: error: C compiler cannot create
>> executables
>> > > > > > >     [exec] See `config.log' for more details.
>> > > > > > >
>> > > > > > > BUILD FAILED
>> > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243:
exec
>> > > > returned:
>> > > > > > 77
>> > > > > > >
>> > > > > > > For 64 bit build, I got:
>> > > > > > >     [exec]  gcc -DHAVE_CONFIG_H -I.
>> > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native
-I./impl
>> > > > > > > -I/usr/java/default/include -I/usr/java/default/include/linux
>> > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
>> > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2
-m64 -m64
>> -MT
>> > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
>> > > > impl/lzo/.deps/LzoCompressor.Tpo
>> > > > > -c
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
>> > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
>> > > > > > >     [exec]
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
>> > > > > > > In function
>> > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
>> > > > > > >     [exec]
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
>> > > > > > > error: expected expression before ',' token
>> > > > > > >     [exec] make: *** [impl/lzo/LzoCompressor.lo]
Error 1
>> > > > > > >
>> > > > > > > BUILD FAILED
>> > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251:
exec
>> > > > returned:
>> > > > > 2
>> > > > > > >
>> > > > > > > Has anybody seen the above ?
>> > > > > > >
>> > > > > > > Thanks
>> > > > > > >
>> > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
>> todd@cloudera.com>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Ted,
>> > > > > > > >
>> > > > > > > > You need to install liblzo from EPEL:
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
>> > > > > > > >
>> > > > > > > > -Todd
>> > > > > > > >
>> > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yuzhihong@gmail.com
>> >
>> > > > wrote:
>> > > > > > > >
>> > > > > > > > > Can someone tell me how I can install liblzo
?
>> > > > > > > > >
>> > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
>> > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP
Tue Jul 14
>> 06:36:37
>> > > EDT
>> > > > > > 2009
>> > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
>> > > > > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
>> > > > > > > > > Loaded plugins: fastestmirror
>> > > > > > > > > Loading mirror speeds from cached hostfile
>> > > > > > > > >  * base: mirrors.usc.edu
>> > > > > > > > >  * updates: mirror.san.fastserv.com
>> > > > > > > > >  * addons: centos.promopeddler.com
>> > > > > > > > >  * extras: mirrors.versaweb.com
>> > > > > > > > > Setting up Install Process
>> > > > > > > > > Parsing package install arguments
>> > > > > > > > > No package liblzo-devel available.
>> > > > > > > > > Nothing to do
>> > > > > > > > >
>> > > > > > > > > Thanks
>> > > > > > > > >
>> > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo
<
>> > > kuosenhao@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Ted,
>> > > > > > > > > >
>> > > > > > > > > > You may want to consider LZO compression,
which allows
>> > > > splitting
>> > > > > > for
>> > > > > > > a
>> > > > > > > > > > comporessed file for Map jobs.  On
the other hand, gzip
>> is
>> > > not
>> > > > > > > > > splittable.
>> > > > > > > > > >
>> > > > > > > > > > Check out these links.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
>> > > > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted
Yu <
>> > yuzhihong@gmail.com>
>> > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > The input file is in .gz format
>> > > > > > > > > > > FYI
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM,
Ted Yu <
>> > > yuzhihong@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > My current project processes
input file of size
>> > 333302161
>> > > > > > bytes.
>> > > > > > > > > > > > What I plan to do is to split
the file into equal
>> size
>> > > > pieces
>> > > > > > > (and
>> > > > > > > > on
>> > > > > > > > > > > blank
>> > > > > > > > > > > > line boundary) to improve
performance.
>> > > > > > > > > > > >
>> > > > > > > > > > > > I found 12 classes in 0.20.1
source code which
>> > implement
>> > > > > > > > InputSplit.
>> > > > > > > > > > > >
>> > > > > > > > > > > > If someone has written code
similar to what I plan to
>> > do,
>> > > > > > please
>> > > > > > > > > share
>> > > > > > > > > > > some
>> > > > > > > > > > > > hint.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27
AM, Amogh Vasekar <
>> > > > > > > amogh@yahoo-inc.com
>> > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > >> Hi,
>> > > > > > > > > > > >> The deprecation is due
to the new evolving mapreduce
>> (
>> > > > > > > > > o.a.h.mapreduce
>> > > > > > > > > > )
>> > > > > > > > > > > >> APIs. Old APIs are supported
for available
>> > > distributions.
>> > > > > The
>> > > > > > > > > > equivalent
>> > > > > > > > > > > of
>> > > > > > > > > > > >> TextInputFormat is available
in new API :
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Thanks,
>> > > > > > > > > > > >> Amogh
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted
Yu" <yuzhihong@gmail.com>
>> > > wrote:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> According to:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> isSplitable() is deprecated.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Which method should I
use to replace it ?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Thanks
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Mime
View raw message