Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates
 209.85.222.198 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=lu0pvbD7obifF/llIio+74hE0nPv8stQqBKQBFK9g8qBJDq48s9+utcvMQkqcctLsQ
         HMAxj++AycmHfyvblJxnSeRKf+IWzU1EhTEXE+eQLuLilfV/A5wJzz+fO4H3NrV2gBN5
         wTIDi55XQaW6fDEbr1DpqNqUab9x2+DHYD+b8=
MIME-Version: 1.0
In-Reply-To: <45f85f71001151317r6571098ek989cb72298df8b25@mail.gmail.com>
References: <17e273101001071417s674d5258hadb3890b5d71800a@mail.gmail.com>
	 <45f85f71001111534l57476a95u357abe7e9c0f9517@mail.gmail.com>
	 <17e273101001121015j4c91d8a9obba77fcaa08054b3@mail.gmail.com>
	 <45f85f71001121025n6bc01a2fva6cf84a8f7eba9bd@mail.gmail.com>
	 <17e273101001121057q2b06b960h943595dfa1a78f46@mail.gmail.com>
	 <45f85f71001121432j782e2669tb62b00c5ca96d289@mail.gmail.com>
	 <17e273101001121628q4dabfb83hecf56309b1c32726@mail.gmail.com>
	 <45f85f71001151209i68f357a1qf8239dffb16ba71b@mail.gmail.com>
	 <17e273101001151314s556e8d7xe25b07513ab9b9d6@mail.gmail.com>
	 <45f85f71001151317r6571098ek989cb72298df8b25@mail.gmail.com>
Date: Fri, 15 Jan 2010 13:34:34 -0800
Message-ID: <17e273101001151334g6d84ba97y624b2b0d71e2c25b@mail.gmail.com>
Subject: Re: isSplitable() deprecated
From: Ted Yu <yuzhihong@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0a5911225b7047d3ac52e

--001636e0a5911225b7047d3ac52e
Content-Type: text/plain; charset=ISO-8859-1

I tried to set LD_LIBRARY_PATH to point to where crt1.o resides:
export LD_LIBRARY_PATH=/usr/lib64

Still the same issue.
This is snippet of config.log:

Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-libgcj-multifile
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
--disable-dssi --enable-plugin
--with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
--host=x86_64-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
configure:2781: $? = 0
configure:2788: gcc -V >&5
gcc: '-V' option must have argument
configure:2791: $? = 1
configure:2814: checking for C compiler default output file name
configure:2841: gcc -m32   conftest.c  >&5
/usr/bin/ld: crt1.o: No such file: No such file or directory
collect2: ld returned 1 exit status
configure:2844: $? = 1
configure:2882: result:
configure: failed program was:
| /* confdefs.h.  */
| #define PACKAGE_NAME "gpl-compression"
| #define PACKAGE_TARNAME "gpl-compression"
| #define PACKAGE_VERSION "0.1.0"
| #define PACKAGE_STRING "gpl-compression 0.1.0"
| #define PACKAGE_BUGREPORT "omalley@apache.org"
| #define PACKAGE "gpl-compression"
| #define VERSION "0.1.0"
| #define _GNU_SOURCE 1
| /* end confdefs.h.  */
|
| int
| main ()
| {
|
|   ;
|   return 0;
| }
configure:2889: error: C compiler cannot create executables
See `config.log' for more details.


On Fri, Jan 15, 2010 at 1:17 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Are you starting from a clean tarball of the lzo stuff?
>
> Can you make sure your /tmp/ partition isn't full?
>
> There should be a config.log file hanging around somewhere also - perhaps
> this would be helpful.
>
> -Todd
>
> On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Todd:
> > Thanks for the continued support.
> >
> > I installed lzo-devel:
> > [root@tyu-linux batchclient]# rpm -ivh
> > /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
> > warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
> > signature: NOKEY, key ID 217521f6
> > Preparing...                ###########################################
> > [100%]
> >    1:lzo-devel              ###########################################
> > [100%]
> >
> > But I still see:
> >     [exec] Try `/usr/bin/install --help' for more information.
> >     [exec] checking for gcc... gcc
> >      [exec] checking for C compiler default output file name...
> >     [exec] configure: error: C compiler cannot create executables
> >
> >
> > On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >
> > > Hi Ted,
> > >
> > > Did you also install liblzo-devel?
> > >
> > > Here are the packages I install for LZO:
> > >
> > > lzo-2.02-2.el5.1
> > > lzo-devel-2.02-2.el5.1
> > >
> > > -Todd
> > >
> > > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > version
> > > > Nothing to do
> > > >
> > > > [root@tyu-linux batchclient]# yum install gcc-c++
> > > > Loaded plugins: fastestmirror
> > > > Loading mirror speeds from cached hostfile
> > > >  * base: mirrors.kernel.org
> > > >  * updates: updates.interworx.info
> > > >  * addons: mirrors.kernel.org
> > > >  * extras: mirrors.netdna.com
> > > > Setting up Install Process
> > > > Parsing package install arguments
> > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > > > version
> > > >
> > > >
> > > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <todd@cloudera.com>
> > wrote:
> > > >
> > > > > Make sure you have gcc, g++ installed?
> > > > >
> > > > > -Todd
> > > > >
> > > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yuzhihong@gmail.com>
> > wrote:
> > > > >
> > > > > > I installed
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > > > > .
> > > > > >
> > > > > > [root@tyu-linux software]# rpm -e lzo
> > > > > > [root@tyu-linux software]# rpm -ivh
> > > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
> > > > > signature:
> > > > > > NOKEY, key ID 217521f6
> > > > > > Preparing...
> > >  ###########################################
> > > > > > [100%]
> > > > > >   1:lzo
> > >  ###########################################
> > > > > > [100%]
> > > > > >
> > > > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > > > > > lrwxrwxrwx 1 root root     16 Jan 12 10:29
> /usr/lib64/liblzo2.so.2
> > ->
> > > > > > liblzo2.so.2.0.0
> > > > > > -rwxr-xr-x 1 root root 123408 May  6  2007
> > > /usr/lib64/liblzo2.so.2.0.0
> > > > > >
> > > > > > I still see this error:
> > > > > >      [exec] configure: error: C compiler cannot create
> executables
> > > > > >     [exec] See `config.log' for more details.
> > > > > >
> > > > > > BUILD FAILED
> > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > > returned:
> > > > > 77
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <todd@cloudera.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hi Ted,
> > > > > > >
> > > > > > > Please make sure you have version 2.02 of liblzo installed.
> > There's
> > > > an
> > > > > > > earlier version in RHEL that doesn't fail properly at the
> > configure
> > > > > stage
> > > > > > > and I've seen it produce the error you pasted below.
> > > > > > >
> > > > > > > -Todd
> > > > > > >
> > > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yuzhihong@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > I followed
> > > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > > > > >
> > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and
> > > > latest
> > > > > > > > version
> > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37
> > EDT
> > > > > 2009
> > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > >
> > > > > > > > For 32 bit build, I got:
> > > > > > > >     [exec] checking for C compiler default output file
> name...
> > > > > > > >     [exec] configure: error: C compiler cannot create
> > executables
> > > > > > > >     [exec] See `config.log' for more details.
> > > > > > > >
> > > > > > > > BUILD FAILED
> > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > > > > returned:
> > > > > > > 77
> > > > > > > >
> > > > > > > > For 64 bit build, I got:
> > > > > > > >     [exec]  gcc -DHAVE_CONFIG_H -I.
> > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native
> -I./impl
> > > > > > > > -I/usr/java/default/include -I/usr/java/default/include/linux
> > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64
> > -MT
> > > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > > > > impl/lzo/.deps/LzoCompressor.Tpo
> > > > > > -c
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > > > >     [exec]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > > > > In function
> > > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > > > >     [exec]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > > > > error: expected expression before ',' token
> > > > > > > >     [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
> > > > > > > >
> > > > > > > > BUILD FAILED
> > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec
> > > > > returned:
> > > > > > 2
> > > > > > > >
> > > > > > > > Has anybody seen the above ?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
> > todd@cloudera.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Ted,
> > > > > > > > >
> > > > > > > > > You need to install liblzo from EPEL:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > > > > >
> > > > > > > > > -Todd
> > > > > > > > >
> > > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <
> yuzhihong@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Can someone tell me how I can install liblzo ?
> > > > > > > > > >
> > > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
> > 06:36:37
> > > > EDT
> > > > > > > 2009
> > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > > >  * base: mirrors.usc.edu
> > > > > > > > > >  * updates: mirror.san.fastserv.com
> > > > > > > > > >  * addons: centos.promopeddler.com
> > > > > > > > > >  * extras: mirrors.versaweb.com
> > > > > > > > > > Setting up Install Process
> > > > > > > > > > Parsing package install arguments
> > > > > > > > > > No package liblzo-devel available.
> > > > > > > > > > Nothing to do
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
> > > > kuosenhao@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Ted,
> > > > > > > > > > >
> > > > > > > > > > > You may want to consider LZO compression, which allows
> > > > > splitting
> > > > > > > for
> > > > > > > > a
> > > > > > > > > > > comporessed file for Map jobs.  On the other hand, gzip
> > is
> > > > not
> > > > > > > > > > splittable.
> > > > > > > > > > >
> > > > > > > > > > > Check out these links.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <
> > > yuzhihong@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > The input file is in .gz format
> > > > > > > > > > > > FYI
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
> > > > yuzhihong@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > My current project processes input file of size
> > > 333302161
> > > > > > > bytes.
> > > > > > > > > > > > > What I plan to do is to split the file into equal
> > size
> > > > > pieces
> > > > > > > > (and
> > > > > > > > > on
> > > > > > > > > > > > blank
> > > > > > > > > > > > > line boundary) to improve performance.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I found 12 classes in 0.20.1 source code which
> > > implement
> > > > > > > > > InputSplit.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If someone has written code similar to what I plan
> to
> > > do,
> > > > > > > please
> > > > > > > > > > share
> > > > > > > > > > > > some
> > > > > > > > > > > > > hint.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> > > > > > > > amogh@yahoo-inc.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > >> The deprecation is due to the new evolving
> mapreduce
> > (
> > > > > > > > > > o.a.h.mapreduce
> > > > > > > > > > > )
> > > > > > > > > > > > >> APIs. Old APIs are supported for available
> > > > distributions.
> > > > > > The
> > > > > > > > > > > equivalent
> > > > > > > > > > > > of
> > > > > > > > > > > > >> TextInputFormat is available in new API :
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >> Amogh
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yuzhihong@gmail.com>
> > > > wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> According to:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Which method should I use to replace it ?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--001636e0a5911225b7047d3ac52e--