hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yongzhi Wang <wang.yongzhi2...@gmail.com>
Subject Re: No 32-bit taskcontroller on Hadoop 1.0.3
Date Wed, 19 Sep 2012 03:42:26 GMT
Hi, Harsh

Thanks for your suggestion. It works!

However, you will met an error while building the taskcontroller:

[exec] /usr/include/fts.h:41:3: error: #error "<fts.h> cannot be used
with -D_FILE_OFFSET_BITS==64"

This is a known bug, you can find more information in
https://issues.apache.org/jira/browse/MAPREDUCE-2178

The solution is as follows:

    Another amendment: AC_SYS_LARGEFILE in configure.ac causes a build
failure on 32-bit RHEL5:

    [exec] /usr/include/fts.h:41:3: error: #error "<fts.h> cannot be
used with -D_FILE_OFFSET_BITS==64"

    Since the task-controller doesn't need to deal with very large
(>2G) files, I don't think this flag is necessary. (it just copies
job.xml, tokens, and taskjvm.sh)

Thank you very much!
Yongzhi

On Mon, Sep 17, 2012 at 11:38 PM, Harsh J <harsh@cloudera.com> wrote:
> Hey Yongzhi,
>
> You'll need to build it manually on your 32-bit machine. Run the build
> from your $HADOOP_HOME/$HADOOP_PREFIX as:
>
> (Depending on your version you may or may not require jdk5, we had
> removed that dependency recently. The rest of the tools mentioned, you
> should require for a full tar generation.)
>
> ant -Dcompile.native=true -Dcompile.c++=true -Dlibhdfs=true
> -Dlibrecordio=true -Dhadoop.conf.dir=/etc/hadoop
> -Dxercescroot=$HOME/tools/xerces/latest
> -Declipse.home=$HOME/tools/eclipse/latest
> -Djava5.home=$HOME/tools/java5/latest
> -Dforrest.home=$HOME/tools/forrest/apache-forrest-0.8
> -Dfindbugs.home=$HOME/tools/findbugs/latest veryclean clean
> create-c++-configure task-controller tar
>
> This should build a fully formed tarball that should have 32-bit
> natives in it. Use this tarball to deploy your cluster.
>
> Or if you just want the task-controller and the natives for your arch,
> do (Shouldn't require any tool dependencies):
>
> ant -Dcompile.native=true -Dcompile.c++=true -Dlibhdfs=true
> -Dlibrecordio=true veryclean clean compile create-c++-configure
> task-controller
>
> Alternatively, I encourage looking at using packages for your arch,
> and at the Apache Bigtop (incubating) project:
> http://incubator.apache.org/bigtop
>
> On Tue, Sep 18, 2012 at 8:34 AM, Yongzhi Wang
> <wang.yongzhi2009@gmail.com> wrote:
>> Dear All,
>>
>> I am currently deploying hadoop 1.0.3 on my Debian 32-bit Linux. I
>> think need a 32-bit binary file taskcontroller. However, I found the
>> binary
>> files provided in hadoop 1.0.3 is 64 bit. I downloaded the hadoop
>> build file from server jenkins
>> (https://builds.apache.org/job/Hadoop-1.0-Build/ws/trunk/build/c++-build/Linux-i386-32/task-controller/).
>> It's still a 64 bit file.
>>
>> I got the following errors when I start task tracker using the hadoop
>> 64-bit taskcontroller:
>>
>> 12/09/17 11:59:58 ERROR mapred.TaskTracker: Can not start task tracker
>> because java.io.IOException: Task controller setup failed because of
>> invalidpermissions/ownership with exit code 126
>>         at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:143)
>>         at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1452)
>>         at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
>> Caused by: org.apache.hadoop.util.Shell$ExitCodeException:
>> /opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller:
>> /opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller: cannot execute
>> binary file
>>
>>         at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
>>         at org.apache.hadoop.util.Shell.run(Shell.java:182)
>>         at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
>>         at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:137)
>>
>> I am wondering if not providing 32-bit of taskcontroller is a build
>> bug, or 64-bit taskcontroller can be used somehow on the 32-bit
>> platform? If no 32-bit executable is provided in the daily build of
>> hadoop, how can I build one by myself?
>>
>> Thanks!
>
>
>
> --
> Harsh J

Mime
View raw message