Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5C446F979 for ; Mon, 29 Apr 2013 18:19:48 +0000 (UTC) Received: (qmail 29544 invoked by uid 500); 29 Apr 2013 18:19:43 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 29421 invoked by uid 500); 29 Apr 2013 18:19:43 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 29414 invoked by uid 99); 29 Apr 2013 18:19:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Apr 2013 18:19:43 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rkevinburton@charter.net designates 216.33.127.83 as permitted sender) Received: from [216.33.127.83] (HELO mta41.charter.net) (216.33.127.83) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Apr 2013 18:19:36 +0000 Received: from imp10 ([10.20.200.15]) by mta41.charter.net (InterMail vM.8.01.05.02 201-2260-151-103-20110920) with ESMTP id <20130429181915.HLJM10143.mta41.charter.net@imp10>; Mon, 29 Apr 2013 14:19:15 -0400 Received: from [10.107.3.161] ([166.137.96.168]) by imp10 with smtp.charter.net id VuKE1l00b3dyE9i05uKEAy; Mon, 29 Apr 2013 14:19:15 -0400 X-Authority-Analysis: v=2.0 cv=foeE/yEf c=1 sm=1 a=wx9n5Nr9WPFqVmFBHcCnQg==:17 a=yUnIBFQkZM0A:10 a=hOpmn2quAAAA:8 a=lb7Xh8rC7tgA:10 a=fE3eKMaoAAAA:8 a=mV9VRH-2AAAA:8 a=vX-s-dEDozIhM-ZKGakA:9 a=QEXdDO2ut3YA:10 a=5yKvCNww8xcA:10 a=hUswqBWy9Q8A:10 a=zaHepHhhAAAA:8 a=EZS9vMF_CjOHkP3su9YA:9 a=_W_S_7VecoQA:10 a=tXsnliwV7b4A:10 a=Z9zn5FB1dCcA:10 a=SQ_PktkZ81UA:10 a=d2WjTFMwHWbGhDWy:21 a=wx9n5Nr9WPFqVmFBHcCnQg==:117 X-Auth-id: cmtldmluYnVydG9uQGNoYXJ0ZXIubmV0 References: <131B8C5C-1E0E-4971-AD40-A696645FC8CB@charter.net> <009801ce4480$3de4cab0$b9ae6010$@charter.net> In-Reply-To: Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: 7bit Content-Type: multipart/alternative; boundary=Apple-Mail-90A0832A-5FAD-4C7D-BEAA-CF799E0B4AA5 Message-Id: <684CA899-4F17-473D-A94F-1FAB0CB4F644@charter.net> Cc: "user@hadoop.apache.org" X-Mailer: iPhone Mail (10B329) From: Kevin Burton Subject: Re: Warnings? Date: Mon, 29 Apr 2013 13:19:14 -0500 To: Ted Xu X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-90A0832A-5FAD-4C7D-BEAA-CF799E0B4AA5 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable If it doesn't work what are my options? Is there source that I can download a= nd compile? On Apr 29, 2013, at 10:31 AM, Ted Xu wrote: > Hi Kevin, >=20 > Native libraries are those implemented using C/C++, which only provide cod= e level portability (instead of binary level portability, as Java do). That i= s to say, the binaries provided by CDH4 distribution will in most cases be b= roken in your environment.=20 >=20 > To check if your native libraries are working or not, you can follow the i= nstructions I sent previously. Quote as following. >=20 >
> During runtime, check the hadoop log files for your MapReduce tasks. >=20 > If everything is all right, then: DEBUG util.NativeCodeLoader - Trying to l= oad the custom-built native-hadoop library... INFO util.NativeCodeLoader - L= oaded the native-hadoop library > If something goes wrong, then: INFO util.NativeCodeLoader - Unable to load= native-hadoop library for your platform... using builtin-java classes where= applicable > >=20 >=20 > On Mon, Apr 29, 2013 at 10:21 AM, Kevin Burton w= rote: >> I looked at the link you provided and found the Ubuntu is one of the =E2=80= =9Csupported platforms=E2=80=9D but it doesn=E2=80=99t give any information o= n how to obtain it or build it. Any idea why it is not includde as part of t= he Cloudera CDH4 distribution? I followed the installation instructions (mos= tly apt-get install . . . .) but I fail to see the libhadoop.so. In order t= o avoid this warning do I need to download the Apache distribution? Which on= e? >>=20 >> =20 >>=20 >> For the warnings about the configuration I looked in my configuration and= for this specific example I don=E2=80=99t see =E2=80=98session.id=E2=80=99 u= sed anywhere. It must be used by default. If so why is the deprecated defaul= t being used? >>=20 >> =20 >>=20 >> As for the two warnings about counters. I know I have not implemented any= code for counters so again this must be something internal. Is there someth= ing I am doing to trigger this? >>=20 >> =20 >>=20 >> So I can avoid them what are =E2=80=9Chadoop generic options=E2=80=9D? >>=20 >> =20 >>=20 >> Thanks again. >>=20 >> =20 >>=20 >> Kevin >>=20 >> =20 >>=20 >> From: Ted Xu [mailto:txu@gopivotal.com]=20 >> Sent: Friday, April 26, 2013 10:49 PM >> To: user@hadoop.apache.org >> Subject: Re: Warnings? >>=20 >> =20 >>=20 >> Hi Kevin, >>=20 >> =20 >>=20 >> Please see my comments inline, >>=20 >> =20 >>=20 >> On Sat, Apr 27, 2013 at 11:24 AM, Kevin Burton = wrote: >>=20 >> Is the native library not available for Ubuntu? If so how do I load it? >>=20 >> Native libraries usually requires recompile, for more information please r= efer Native Libraries.=20 >>=20 >> =20 >>=20 >> =20 >>=20 >> Can I tell which key is off? Since I am just starting I would want to be a= s up to date as possible. It is out of date probably because I copied my exa= mples from books and tutorials. >>=20 >> =20 >>=20 >> I think the warning messages are telling it already, "xxx is deprecated, u= se xxx instead...". In fact, most of the configure keys are changed from had= oop 1.x to 2.x. The compatibility change may later documented on http://wiki= .apache.org/hadoop/Compatibility. >>=20 >> =20 >>=20 >> The main class does derive from Tool. Should I ignore this warning as it s= eems to be in error? >>=20 >> Of course you can ignore this warning as long as you don't use hadoop gen= eric options. >>=20 >> =20 >>=20 >> =20 >>=20 >> Thank you. >>=20 >>=20 >> On Apr 26, 2013, at 7:49 PM, Ted Xu wrote: >>=20 >> Hi, >>=20 >> =20 >>=20 >> First warning is saying hadoop cannot load native library, usually a comp= ression codec. In that case, hadoop will use java implementation instead, wh= ich is slower. >>=20 >> =20 >>=20 >> Second is caused by hadoop 1.x/2.x configuration key change. You're using= a 1.x style key under 2.x, yet hadoop still guarantees backward compatibili= ty. >>=20 >> =20 >>=20 >> Third is saying that the main class of a hadoop application is recommande= d to implement org.apache.hadoop.util.Tool, or else generic command line opt= ions (e.g., -D options) will not supported. =20 >>=20 >> =20 >>=20 >> On Sat, Apr 27, 2013 at 5:51 AM, wrote: >>=20 >> I am running a simple WordCount m/r job and I get output but I get five w= arnings that I am not sure if I should pay attention to: >>=20 >> =20 >>=20 >> 13/04/26 16:24:50 WARN util.NativeCodeLoader: Unable to load native-hadoo= p library for your platform... using builtin-java classes where applicable >>=20 >> =20 >>=20 >> 13/04/26 16:24:50 WARN conf.Configuration: session.id is deprecated. Inst= ead, use dfs.metrics.session-id >>=20 >> =20 >>=20 >> 13/04/26 16:24:50 WARN mapred.JobClient: Use GenericOptionsParser for par= sing the arguments. Applications should implement Tool for the same. >>=20 >> =20 >>=20 >> 13/04/26 16:24:51 WARN mapreduce.Counters: Group org.apache.hadoop.mapred= .Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter ins= tead >>=20 >> =20 >>=20 >> 13/04/26 16:24:51 WARN mapreduce.Counters: Counter name MAP_INPUT_BYTES i= s deprecated. Use FileInputFormatCounters as group name and BYTES_READ as c= ounter name instead >>=20 >> =20 >>=20 >> Any ideas on what these mean? The only one that I can see in the code is t= he third one. I am using GenericOptionsParser as it is part of an example th= at I copied. But I don't know why this is considered bad. >>=20 >> =20 >>=20 >> Thank you. >>=20 >> =20 >>=20 >>=20 >>=20 >>=20 >> =20 >>=20 >> -- >>=20 >> Regards, >>=20 >> Ted Xu >>=20 >>=20 >>=20 >>=20 >> =20 >>=20 >> -- >>=20 >> Regards, >>=20 >> Ted Xu >>=20 >=20 >=20 >=20 > --=20 > Regards, > Ted Xu --Apple-Mail-90A0832A-5FAD-4C7D-BEAA-CF799E0B4AA5 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
If it doesn't work what are my options= ? Is there source that I can download and compile?

On Apr 29, 2= 013, at 10:31 AM, Ted Xu <txu@gopivo= tal.com> wrote:

Hi Kevin,

Native libraries are those imp= lemented using C/C++, which only provide code level portability (instead of b= inary level portability, as Java do). That is to say, the binaries provided b= y CDH4 distribution will in most cases be broken in your environment. <= /div>

To check if your native libraries a= re working or not, you can follow the instructions I sent previously. Quote a= s following.

<blockquote&g= t;

During runtime, check the hadoop log files= for your MapReduce tasks.

  • If everything is all right,= then: DEBUG util.NativeCodeLoader - Trying to load the custom-buil= t native-hadoop library... INFO util.NativeCodeLoader - Loaded= the native-hadoop library
  • If something goes wrong, th= en: INFO util.NativeCodeLoader - Unable to load native-hadoop libra= ry for your platform... using builtin-java classes where applicable
</blokquote>


On Mon, Apr 29, 2= 013 at 10:21 AM, Kevin Burton <rkevinburton@charter.net> wrote:

I looked at the l= ink you provided and found the Ubuntu is one of the =E2=80=9Csupported platf= orms=E2=80=9D but it doesn=E2=80=99t give any information on how to obtain i= t or build it. Any idea why it is not includde as part of the Cloudera CDH4 d= istribution? I followed the installation instructions (mostly apt-get instal= l . . . .) but I fail to see the libhadoop.so.  In order to avoid this w= arning do I need to download the Apache distribution? Which one?

 =

For the warnings about t= he configuration I looked in my configuration and for this specific example I= don=E2=80=99t see =E2=80=98= session.id=E2=80=99 used anywhere. It must be used by default. If so why= is the deprecated default being used?

 =

As for the two warnings= about counters. I know I have not implemented any code for counters so agai= n this must be something internal. Is there something I am doing to trigger t= his?

 =

So I can avoid them wha= t are =E2=80=9Chadoop generic options=E2=80=9D?

 =

Thanks again.=

 =

Kevin

 =

From: Ted Xu [= mailto:txu@gopivotal.= com]
Sent: Friday, April 26, 2013 10:49 PM
To: user@hadoop.apache.org
= Subject: Re: Warnings?

 

Hi Kevin,<= /p>

 

Please see my comments inline,

 

On Sat, Apr 27, 2013 at 1= 1:24 AM, Kevin Burton <rkevinburton@charter.net> wrote:

Is the native library not available for Ubuntu? If so= how do I load it?

= Native libraries usually requires recompile, for more information please ref= er Native Libraries.=  

 

 

Can I tell which key is off? Since I am ju= st starting I would want to be as up to date as possible. It is out of date p= robably because I copied my examples from books and tutorials.=

 

I think the warning messages are telling i= t already, "xxx is deprecated, use xxx instead...". In fact, most of the con= figure keys are changed from hadoop 1.x to 2.x. The compatibility change may= later documented on http://wiki.apache.org/hadoop/Compatibility.

 

The m= ain class does derive from Tool. Should I ignore this warning as it seems to= be in error?

Of course you can ignor= e this warning as long as you don't use hadoop generic options.

 

 

Thank you.


On Apr 26, 2013, at 7:49 PM, T= ed Xu <txu@gopivot= al.com> wrote:

<= p class=3D"MsoNormal">Hi,

 

First warning is saying= hadoop cannot load native library, usually a compression codec. In that cas= e, hadoop will use java implementation instead, which is slower.

 

Second is caused by hadoop 1.x/2.x configuration key change.= You're using a 1.x style key under 2.x, yet hadoop still guarantees ba= ckward compatibility.

 

Third is saying that the main class of a hadoop application i= s recommanded to implement org.apache.hadoop.ut= il.Tool, or else generic command line options (e.g., -D option= s) will not supported.   

&nbs= p;

On Sat, Apr 27, 2013 at 5:51 AM, &l= t;rkevinburton= @charter.net> wrote:

I am running a simple WordCount m/r job and I get output b= ut I get five warnings that I am not sure if I should pay attention to:

 

<= b>13/04/26 16:24:50 WARN util.NativeCodeLoader: Unable to load nat= ive-hadoop library for your platform... using builtin-java classes where app= licable

 

13/04/26 16:24:50 WARN conf.Configuration: session.id is deprecated. Instead, use dfs.metrics= .session-id

 

13/04/26 16:24:50 WARN mapred.JobClient: Use GenericOptionsParser for= parsing the arguments. Applications should implement Tool for the same.

 

13/04/26 16:24:51 WARN mapreduce.Counters: Group org.apache.hadoop.ma= pred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter= instead

 

= 13/04/26 16:24:51 WARN mapreduce.Counters: Counter name MAP_INP= UT_BYTES is deprecated. Use FileInputFormatCounters as group name and  B= YTES_READ as counter name instead

 

Any ideas on= what these mean? The only one that I can see in the code is the third one. I= am using GenericOptionsParser as it is part of an example that I copied. Bu= t I don't know why this is considered bad.

 

Thank you. <= /span>

 



&= nbsp;

--

Regards,

Ted Xu



 

--

Regards,

Ted Xu




--
=
Regards,
Ted Xu
= --Apple-Mail-90A0832A-5FAD-4C7D-BEAA-CF799E0B4AA5--