Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C6F06D9E2 for ; Wed, 12 Dec 2012 12:54:30 +0000 (UTC) Received: (qmail 6257 invoked by uid 500); 12 Dec 2012 12:54:29 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 6059 invoked by uid 500); 12 Dec 2012 12:54:28 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 6014 invoked by uid 99); 12 Dec 2012 12:54:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Dec 2012 12:54:27 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dontariq@gmail.com designates 209.85.216.176 as permitted sender) Received: from [209.85.216.176] (HELO mail-qc0-f176.google.com) (209.85.216.176) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Dec 2012 12:54:20 +0000 Received: by mail-qc0-f176.google.com with SMTP id n41so282322qco.35 for ; Wed, 12 Dec 2012 04:53:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=xhbNBUek0zVWwVVYyuVC+mLgWQtotryylbnCwZmZNM4=; b=f48/0PGjNe5uaeltfVmZ3NSPKcj04x+72wAvEETjrPn7zoD1ha0TUdVEAKWprbVoKV Fvp/uqe1Hw6XlUkc0gpQ7Kgww0Jj7aiMGeVHJJo5zR8BcVrGNHOnkE2w477Bv4/r25DU YgIcbJJFE2vLvGzf1HjKB7jKUnyauef+h1nRQnL6YufO+2ieA/n3n/kqkuphI2gMzGfx ANtVG0tsv8OZbI1sXrA4cLzEY8YyO2q8peqOrpbTitMQ9qogzQFQsh0jjh1+wHXmqcpD LG5pXMhWfJpzv5MFCk1HfcBzEf4Oyj5BYJFn9Aqp1AndMTAIa7ZoveMJSSQLJhhS+zLC Zj5w== Received: by 10.229.112.38 with SMTP id u38mr404201qcp.104.1355316839328; Wed, 12 Dec 2012 04:53:59 -0800 (PST) MIME-Version: 1.0 Received: by 10.229.126.165 with HTTP; Wed, 12 Dec 2012 04:53:19 -0800 (PST) In-Reply-To: References: From: Mohammad Tariq Date: Wed, 12 Dec 2012 18:23:19 +0530 Message-ID: Subject: Re: Modify the number of map tasks To: user Content-Type: multipart/alternative; boundary=002354471d8cc42c0e04d0a74acb X-Virus-Checked: Checked by ClamAV on apache.org --002354471d8cc42c0e04d0a74acb Content-Type: text/plain; charset=ISO-8859-1 I would check if all the daemons are running properly or not, before anything else. If some problem is found, next place to track is the log of each daemon. The correct command to check the status of a job from command line is : hadoop job -status jobID. (Mind the 'space' after job and remove 'command' from the statement) HTH Regards, Mohammad Tariq On Wed, Dec 12, 2012 at 6:14 PM, imen Megdiche wrote: > My goal is to analyze the response time of MapReduce depending on the size > of the input files. I need to change the number of map and / or Reduce > tasks and recover the execution time. S it turns out that nothing works locally > on my pc : > neither hadoop job-status command job_local_0001 (which return no job > found ) > nor localhost: 50030 > I will be very grateful if you can help m better understand these problem > > > 2012/12/12 Mohammad Tariq > >> Are you working locally?What exactly is the issue? >> >> Regards, >> Mohammad Tariq >> >> >> >> On Wed, Dec 12, 2012 at 6:00 PM, imen Megdiche wrote: >> >>> no >>> >>> >>> 2012/12/12 Mohammad Tariq >>> >>>> Any luck with "localhost:50030"?? >>>> >>>> Regards, >>>> Mohammad Tariq >>>> >>>> >>>> >>>> On Wed, Dec 12, 2012 at 5:53 PM, imen Megdiche >>> > wrote: >>>> >>>>> i run the job through the command line >>>>> >>>>> >>>>> 2012/12/12 Mohammad Tariq >>>>> >>>>>> You have to replace "JobTrackerHost" in "JobTrackerHost:50030" with >>>>>> the actual name of the machine where JobTracker is running. For >>>>>> example, If you are working on a local cluster, you have to use >>>>>> "localhost:50030". >>>>>> >>>>>> Are you running your job through the command line or some IDE? >>>>>> >>>>>> Regards, >>>>>> Mohammad Tariq >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Dec 12, 2012 at 5:42 PM, imen Megdiche < >>>>>> imen.megdiche@gmail.com> wrote: >>>>>> >>>>>>> excuse me the data size is 98 MB >>>>>>> >>>>>>> >>>>>>> 2012/12/12 imen Megdiche >>>>>>> >>>>>>>> the size of data 49 MB and n of map 4 >>>>>>>> the web UI JobTrackerHost:50030 does not wok, what should i do to >>>>>>>> make this appear , i work on ubuntu >>>>>>>> >>>>>>>> >>>>>>>> 2012/12/12 Mohammad Tariq >>>>>>>> >>>>>>>>> Hi Imen, >>>>>>>>> >>>>>>>>> You can visit the MR web UI at "JobTrackerHost:50030" and see >>>>>>>>> all the useful information like no. of mappers, no of reducers, time taken >>>>>>>>> for the execution etc. >>>>>>>>> >>>>>>>>> One quick question for you, what is the size of your data and what >>>>>>>>> is the no of maps which you are getting right now? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Mohammad Tariq >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Dec 12, 2012 at 5:11 PM, imen Megdiche < >>>>>>>>> imen.megdiche@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Thank you Mohammad but the number of map tasks still the same in >>>>>>>>>> the execution. Do you know how to capture the time spent on execution. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2012/12/12 Mohammad Tariq >>>>>>>>>> >>>>>>>>>>> Hi Imen, >>>>>>>>>>> >>>>>>>>>>> You can add "mapred.map.tasks" property in your >>>>>>>>>>> mapred-site.xml file. >>>>>>>>>>> >>>>>>>>>>> But, it is just a hint for the InputFormat. Actually no. of maps >>>>>>>>>>> is actually determined by the no of InputSplits created by the InputFormat. >>>>>>>>>>> >>>>>>>>>>> HTH >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Mohammad Tariq >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Dec 12, 2012 at 4:11 PM, imen Megdiche < >>>>>>>>>>> imen.megdiche@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I try to force the number of map for the mapreduce job with the >>>>>>>>>>>> command : >>>>>>>>>>>> public static void main(String[] args) throws Exception { >>>>>>>>>>>> >>>>>>>>>>>> JobConf conf = new JobConf(WordCount.class); >>>>>>>>>>>> conf.set("mapred.job.tracker", "local"); >>>>>>>>>>>> conf.set("fs.default.name", "local"); >>>>>>>>>>>> conf.setJobName("wordcount"); >>>>>>>>>>>> >>>>>>>>>>>> conf.setOutputKeyClass(Text.class); >>>>>>>>>>>> conf.setOutputValueClass(IntWritable.class); >>>>>>>>>>>> >>>>>>>>>>>> conf.setNumMapTask(6); >>>>>>>>>>>> conf.setMapperClass(Map.class); >>>>>>>>>>>> conf.setCombinerClass(Reduce.class); >>>>>>>>>>>> conf.setReducerClass(Reduce.class); >>>>>>>>>>>> ... >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> But it doesn t work. >>>>>>>>>>>> What can i do to modify the number of map and reduce tasks. >>>>>>>>>>>> >>>>>>>>>>>> Thank you >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > --002354471d8cc42c0e04d0a74acb Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I would check if all the daemons are running properly or not, before anythi= ng else. If some problem is found, next place to track is the log of each d= aemon.

The correct command to check the status of a job = from command line is :
hadoop job -status jobID.=A0
(Mind the 'space' after= job and remove 'command' from the statement)

HTH

Regards,=
=A0=A0 =A0Mohammad Tariq



On Wed, Dec 12, 2012 at 6:14 PM, imen Me= gdiche <imen.megdiche@gmail.com> wrote:
My goal is to analyze the= response time of MapReduce depending on the size of the input files. I need to change the number of = map and / or Reduce tasks and<= /span> recover the execution t= ime. S it turns out tha= t nothing works locally on my pc :
neither hadoop job-status command job_local_0001 (which return no job fo= und )
nor localhost: 50030
I will be very grateful if you can <= span>help m better understand these = problem


2012/12/12 Mohammad Tariq <dontariq@gm= ail.com>
Are you working locally?What exactly is the issue?

Regards,
=A0=A0 =A0Mohammad Tariq
<= div>



On Wed, Dec 12, 2012 at 6:00 PM, imen Me= gdiche <imen.megdiche@gmail.com> wrote:
no


2012/12/12 Mohammad Tariq <dontariq@gmail.com>
Any luck with "localhost:50030"??

Regards,
=A0=A0 =A0Mohammad Tariq



On Wed, Dec 12, 2012 at 5:53 PM, imen Me= gdiche <imen.megdiche@gmail.com> wrote:
i run the job through the command line


2012/12/12 Mohammad Tariq <dontar= iq@gmail.com>
You have to replace "JobTrackerHost" in "JobTrackerHost:50030= " with the actual name=A0of the machine where JobTracker is running. For examp= le, If you are working on a local cluster, you have to use "localhost:= 50030".

=
Are you running your job through the command line or some= IDE?

Regards,
=A0=A0 =A0Mo= hammad Tariq



On Wed, Dec 12, 2012 at 5:42 PM, imen Me= gdiche <imen.megdiche@gmail.com> wrote:
excuse me the data size is 98 MB
<= br>
2012/12/12 imen Megdiche <imen.meg= diche@gmail.com>
the size of data 49 MB and n of map 4=A0
the web UI Jo= bTrackerHost:50030 does not wok, what should i do to make this appear , i w= ork on ubuntu


201= 2/12/12 Mohammad Tariq <dontariq@gmail.com>
Hi Imen,

=A0 =A0 =A0You can visit the M= R web UI at "JobTrackerHost:50030" and see all the useful informa= tion like no. of mappers, no of reducers, time taken =A0for the execution e= tc.

One quick question for you, what is the size of your data and what is = the no of maps which you are getting right now?

Regards,
=A0=A0 =A0Mohammad Tariq



On Wed, Dec 12, 2012 at 5:11 PM, imen Me= gdiche <imen.megdiche@gmail.com> wrote:
Thank you Mohammad but the number of map tasks still the same in the execut= ion. Do you know how to capture the time spent on execution.
=


2012/12/12 Mohammad Tariq <dontariq@gmail.com>
Hi Imen,

=A0 =A0 You can add "mapr= ed.map.tasks" property in your mapred-site.xml file.=A0

But, it is just a hint for the InputFormat. Actually no. of = maps is actually determined by the no of InputSplits created by the=A0Input= Format.

HTH

<= div>Regards,
=A0=A0 =A0Mohammad Tariq



On Wed, Dec 12, 2012 at 4:11 PM, imen Me= gdiche <imen.megdiche@gmail.com> wrote:
Hi,

I try to force the number of map for the mapreduce job with the= command :
=A0 public static void main(String[] args) throws Exception = {

=A0=A0=A0 =A0=A0=A0 =A0 JobConf conf =3D new JobConf(WordCount.cla= ss);
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 conf.set("mapred.job.trac= ker", "local");
=A0=A0=A0 =A0=A0=A0=A0 conf.set("fs.default.name", "local");=A0=A0=A0 =A0=A0=A0=A0 =A0=A0=A0=A0 conf.setJobName("wordcount");
=A0= =A0=A0=A0
=A0=A0=A0=A0 =A0=A0=A0=A0 conf.setOutputKeyClass(Text.class);=
=A0=A0=A0 =A0=A0=A0=A0 conf.setOutputValueClass(IntWritable.class);
=A0= =A0=A0=A0 =A0=A0=A0=A0
=A0 =A0=A0=A0 =A0=A0=A0=A0 conf.setNumMapTask(6)= ;
=A0=A0=A0=A0 =A0=A0=A0=A0 conf.setMapperClass(Map.class);
=A0=A0=A0= =A0 =A0=A0=A0=A0 conf.setCombinerClass(Reduce.class);
=A0=A0=A0=A0 =A0= =A0=A0=A0 conf.setReducerClass(Reduce.class);
...
}

But it doesn t work.
What can i do to modify the number= of map and reduce tasks.

Thank you












--002354471d8cc42c0e04d0a74acb--