hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Burton" <rkevinbur...@charter.net>
Subject RE: Warnings?
Date Mon, 29 Apr 2013 02:21:21 GMT
I looked at the link you provided and found the Ubuntu is one of the
"supported platforms" but it doesn't give any information on how to obtain
it or build it. Any idea why it is not includde as part of the Cloudera CDH4
distribution? I followed the installation instructions (mostly apt-get
install . . . .) but I fail to see the libhadoop.so.  In order to avoid this
warning do I need to download the Apache distribution? Which one?


For the warnings about the configuration I looked in my configuration and
for this specific example I don't see 'session.id' used anywhere. It must be
used by default. If so why is the deprecated default being used? 


As for the two warnings about counters. I know I have not implemented any
code for counters so again this must be something internal. Is there
something I am doing to trigger this?


So I can avoid them what are "hadoop generic options"?


Thanks again.




From: Ted Xu [mailto:txu@gopivotal.com] 
Sent: Friday, April 26, 2013 10:49 PM
To: user@hadoop.apache.org
Subject: Re: Warnings?


Hi Kevin,


Please see my comments inline,


On Sat, Apr 27, 2013 at 11:24 AM, Kevin Burton <rkevinburton@charter.net>

Is the native library not available for Ubuntu? If so how do I load it?

Native libraries usually requires recompile, for more information please
refer Native Libraries
n/NativeLibraries.html> . 



Can I tell which key is off? Since I am just starting I would want to be as
up to date as possible. It is out of date probably because I copied my
examples from books and tutorials.


I think the warning messages are telling it already, "xxx is deprecated, use
xxx instead...". In fact, most of the configure keys are changed from hadoop
1.x to 2.x. The compatibility change may later documented on


The main class does derive from Tool. Should I ignore this warning as it
seems to be in error?

Of course you can ignore this warning as long as you don't use hadoop
generic options.



Thank you.

On Apr 26, 2013, at 7:49 PM, Ted Xu <txu@gopivotal.com> wrote:



First warning is saying hadoop cannot load native library, usually a
compression codec. In that case, hadoop will use java implementation
instead, which is slower.


Second is caused by hadoop 1.x/2.x configuration key change. You're using a
1.x style key under 2.x, yet hadoop still guarantees backward compatibility.


Third is saying that the main class of a hadoop application is recommanded
to implement org.apache.hadoop.util.Tool
, or else generic command line options (e.g., -D options) will not


On Sat, Apr 27, 2013 at 5:51 AM, <rkevinburton@charter.net> wrote:

I am running a simple WordCount m/r job and I get output but I get five
warnings that I am not sure if I should pay attention to: 


13/04/26 16:24:50 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable 


13/04/26 16:24:50 WARN conf.Configuration: session.id is deprecated.
Instead, use dfs.metrics.session-id 


13/04/26 16:24:50 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same. 


13/04/26 16:24:51 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead 


13/04/26 16:24:51 WARN mapreduce.Counters: Counter name MAP_INPUT_BYTES is
deprecated. Use FileInputFormatCounters as group name and  BYTES_READ as
counter name instead


Any ideas on what these mean? The only one that I can see in the code is the
third one. I am using GenericOptionsParser as it is part of an example that
I copied. But I don't know why this is considered bad. 


Thank you. 





Ted Xu




Ted Xu

View raw message