hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: isSplitable() deprecated
Date Fri, 15 Jan 2010 21:34:34 GMT
I tried to set LD_LIBRARY_PATH to point to where crt1.o resides:
export LD_LIBRARY_PATH=/usr/lib64

Still the same issue.
This is snippet of config.log:

Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-libgcj-multifile
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
--disable-dssi --enable-plugin
--with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
--host=x86_64-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
configure:2781: $? = 0
configure:2788: gcc -V >&5
gcc: '-V' option must have argument
configure:2791: $? = 1
configure:2814: checking for C compiler default output file name
configure:2841: gcc -m32   conftest.c  >&5
/usr/bin/ld: crt1.o: No such file: No such file or directory
collect2: ld returned 1 exit status
configure:2844: $? = 1
configure:2882: result:
configure: failed program was:
| /* confdefs.h.  */
| #define PACKAGE_NAME "gpl-compression"
| #define PACKAGE_TARNAME "gpl-compression"
| #define PACKAGE_VERSION "0.1.0"
| #define PACKAGE_STRING "gpl-compression 0.1.0"
| #define PACKAGE_BUGREPORT "omalley@apache.org"
| #define PACKAGE "gpl-compression"
| #define VERSION "0.1.0"
| #define _GNU_SOURCE 1
| /* end confdefs.h.  */
|
| int
| main ()
| {
|
|   ;
|   return 0;
| }
configure:2889: error: C compiler cannot create executables
See `config.log' for more details.


On Fri, Jan 15, 2010 at 1:17 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Are you starting from a clean tarball of the lzo stuff?
>
> Can you make sure your /tmp/ partition isn't full?
>
> There should be a config.log file hanging around somewhere also - perhaps
> this would be helpful.
>
> -Todd
>
> On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Todd:
> > Thanks for the continued support.
> >
> > I installed lzo-devel:
> > [root@tyu-linux batchclient]# rpm -ivh
> > /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
> > warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
> > signature: NOKEY, key ID 217521f6
> > Preparing...                ###########################################
> > [100%]
> >    1:lzo-devel              ###########################################
> > [100%]
> >
> > But I still see:
> >     [exec] Try `/usr/bin/install --help' for more information.
> >     [exec] checking for gcc... gcc
> >      [exec] checking for C compiler default output file name...
> >     [exec] configure: error: C compiler cannot create executables
> >
> >
> > On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >
> > > Hi Ted,
> > >
> > > Did you also install liblzo-devel?
> > >
> > > Here are the packages I install for LZO:
> > >
> > > lzo-2.02-2.el5.1
> > > lzo-devel-2.02-2.el5.1
> > >
> > > -Todd
> > >
> > > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > version
> > > > Nothing to do
> > > >
> > > > [root@tyu-linux batchclient]# yum install gcc-c++
> > > > Loaded plugins: fastestmirror
> > > > Loading mirror speeds from cached hostfile
> > > >  * base: mirrors.kernel.org
> > > >  * updates: updates.interworx.info
> > > >  * addons: mirrors.kernel.org
> > > >  * extras: mirrors.netdna.com
> > > > Setting up Install Process
> > > > Parsing package install arguments
> > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > > > version
> > > >
> > > >
> > > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <todd@cloudera.com>
> > wrote:
> > > >
> > > > > Make sure you have gcc, g++ installed?
> > > > >
> > > > > -Todd
> > > > >
> > > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yuzhihong@gmail.com>
> > wrote:
> > > > >
> > > > > > I installed
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > > > > .
> > > > > >
> > > > > > [root@tyu-linux software]# rpm -e lzo
> > > > > > [root@tyu-linux software]# rpm -ivh
> > > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3
DSA
> > > > > signature:
> > > > > > NOKEY, key ID 217521f6
> > > > > > Preparing...
> > >  ###########################################
> > > > > > [100%]
> > > > > >   1:lzo
> > >  ###########################################
> > > > > > [100%]
> > > > > >
> > > > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > > > > > lrwxrwxrwx 1 root root     16 Jan 12 10:29
> /usr/lib64/liblzo2.so.2
> > ->
> > > > > > liblzo2.so.2.0.0
> > > > > > -rwxr-xr-x 1 root root 123408 May  6  2007
> > > /usr/lib64/liblzo2.so.2.0.0
> > > > > >
> > > > > > I still see this error:
> > > > > >      [exec] configure: error: C compiler cannot create
> executables
> > > > > >     [exec] See `config.log' for more details.
> > > > > >
> > > > > > BUILD FAILED
> > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > > returned:
> > > > > 77
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <todd@cloudera.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hi Ted,
> > > > > > >
> > > > > > > Please make sure you have version 2.02 of liblzo installed.
> > There's
> > > > an
> > > > > > > earlier version in RHEL that doesn't fail properly at the
> > configure
> > > > > stage
> > > > > > > and I've seen it produce the error you pasted below.
> > > > > > >
> > > > > > > -Todd
> > > > > > >
> > > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yuzhihong@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > I followed
> > > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > > > > >
> > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed
and
> > > > latest
> > > > > > > > version
> > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul
14 06:36:37
> > EDT
> > > > > 2009
> > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > >
> > > > > > > > For 32 bit build, I got:
> > > > > > > >     [exec] checking for C compiler default output
file
> name...
> > > > > > > >     [exec] configure: error: C compiler cannot create
> > executables
> > > > > > > >     [exec] See `config.log' for more details.
> > > > > > > >
> > > > > > > > BUILD FAILED
> > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243:
exec
> > > > > returned:
> > > > > > > 77
> > > > > > > >
> > > > > > > > For 64 bit build, I got:
> > > > > > > >     [exec]  gcc -DHAVE_CONFIG_H -I.
> > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native
> -I./impl
> > > > > > > > -I/usr/java/default/include -I/usr/java/default/include/linux
> > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2
-m64 -m64
> > -MT
> > > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > > > > impl/lzo/.deps/LzoCompressor.Tpo
> > > > > > -c
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > > > >     [exec]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > > > > In function
> > > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > > > >     [exec]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > > > > error: expected expression before ',' token
> > > > > > > >     [exec] make: *** [impl/lzo/LzoCompressor.lo] Error
1
> > > > > > > >
> > > > > > > > BUILD FAILED
> > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251:
exec
> > > > > returned:
> > > > > > 2
> > > > > > > >
> > > > > > > > Has anybody seen the above ?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
> > todd@cloudera.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Ted,
> > > > > > > > >
> > > > > > > > > You need to install liblzo from EPEL:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > > > > >
> > > > > > > > > -Todd
> > > > > > > > >
> > > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <
> yuzhihong@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Can someone tell me how I can install liblzo
?
> > > > > > > > > >
> > > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP
Tue Jul 14
> > 06:36:37
> > > > EDT
> > > > > > > 2009
> > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > > >  * base: mirrors.usc.edu
> > > > > > > > > >  * updates: mirror.san.fastserv.com
> > > > > > > > > >  * addons: centos.promopeddler.com
> > > > > > > > > >  * extras: mirrors.versaweb.com
> > > > > > > > > > Setting up Install Process
> > > > > > > > > > Parsing package install arguments
> > > > > > > > > > No package liblzo-devel available.
> > > > > > > > > > Nothing to do
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve
Kuo <
> > > > kuosenhao@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Ted,
> > > > > > > > > > >
> > > > > > > > > > > You may want to consider LZO compression,
which allows
> > > > > splitting
> > > > > > > for
> > > > > > > > a
> > > > > > > > > > > comporessed file for Map jobs.  On
the other hand, gzip
> > is
> > > > not
> > > > > > > > > > splittable.
> > > > > > > > > > >
> > > > > > > > > > > Check out these links.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted
Yu <
> > > yuzhihong@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > The input file is in .gz format
> > > > > > > > > > > > FYI
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM,
Ted Yu <
> > > > yuzhihong@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > My current project processes
input file of size
> > > 333302161
> > > > > > > bytes.
> > > > > > > > > > > > > What I plan to do is to split
the file into equal
> > size
> > > > > pieces
> > > > > > > > (and
> > > > > > > > > on
> > > > > > > > > > > > blank
> > > > > > > > > > > > > line boundary) to improve
performance.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I found 12 classes in 0.20.1
source code which
> > > implement
> > > > > > > > > InputSplit.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If someone has written code
similar to what I plan
> to
> > > do,
> > > > > > > please
> > > > > > > > > > share
> > > > > > > > > > > > some
> > > > > > > > > > > > > hint.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27
AM, Amogh Vasekar <
> > > > > > > > amogh@yahoo-inc.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > >> The deprecation is due
to the new evolving
> mapreduce
> > (
> > > > > > > > > > o.a.h.mapreduce
> > > > > > > > > > > )
> > > > > > > > > > > > >> APIs. Old APIs are supported
for available
> > > > distributions.
> > > > > > The
> > > > > > > > > > > equivalent
> > > > > > > > > > > > of
> > > > > > > > > > > > >> TextInputFormat is available
in new API :
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >> Amogh
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted
Yu" <yuzhihong@gmail.com>
> > > > wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> According to:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Which method should I
use to replace it ?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message