hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: Modify the number of map tasks
Date Wed, 12 Dec 2012 13:07:13 GMT
Can I have a look at your config files?

Regards,
    Mohammad Tariq



On Wed, Dec 12, 2012 at 6:31 PM, imen Megdiche <imen.megdiche@gmail.com>wrote:

> i run the start-all.sh and all daemons starts without problems. But i the
> log of the tasktracker look like this :
>
>
> 2012-12-12 13:53:45,495 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG:   host = megdiche-OptiPlex-GX280/127.0.1.1
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 1.0.4
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct  3 05:13:58 UTC 2012
> ************************************************************/
> 2012-12-12 13:53:47,009 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2012-12-12 13:53:47,331 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2012-12-12 13:53:47,336 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2012-12-12 13:53:47,336 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> system started
> 2012-12-12 13:53:48,165 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2012-12-12 13:53:48,192 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2012-12-12 13:53:48,513 ERROR org.apache.hadoop.mapred.TaskTracker: Can
> not start task tracker because java.lang.IllegalArgumentException: Does not
> contain a valid host:port authority: local
>     at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
>     at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
>     at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:2560)
>     at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1426)
>     at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
>
> 2012-12-12 13:53:48,519 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker at megdiche-OptiPlex-GX280/
> 127.0.1.1
> ************************************************************/
>
>
>
>
> 2012/12/12 Mohammad Tariq <dontariq@gmail.com>
>
>> I would check if all the daemons are running properly or not, before
>> anything else. If some problem is found, next place to track is the log of
>> each daemon.
>>
>> The correct command to check the status of a job from command line is :
>> hadoop job -status jobID.
>> (Mind the 'space' after job and remove 'command' from the statement)
>>
>> HTH
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>>
>> On Wed, Dec 12, 2012 at 6:14 PM, imen Megdiche <imen.megdiche@gmail.com>wrote:
>>
>>> My goal is to analyze the response time of MapReduce depending on the size
>>> of the input files. I need to change the number of map and / or Reduce
>>> tasks and recover the execution time. S it turns out that nothing works locally
>>> on my pc :
>>> neither hadoop job-status command job_local_0001 (which return no job
>>> found )
>>> nor localhost: 50030
>>> I will be very grateful if you can help m better understand these
>>> problem
>>>
>>>
>>> 2012/12/12 Mohammad Tariq <dontariq@gmail.com>
>>>
>>>> Are you working locally?What exactly is the issue?
>>>>
>>>> Regards,
>>>>     Mohammad Tariq
>>>>
>>>>
>>>>
>>>> On Wed, Dec 12, 2012 at 6:00 PM, imen Megdiche <imen.megdiche@gmail.com
>>>> > wrote:
>>>>
>>>>> no
>>>>>
>>>>>
>>>>> 2012/12/12 Mohammad Tariq <dontariq@gmail.com>
>>>>>
>>>>>> Any luck with "localhost:50030"??
>>>>>>
>>>>>> Regards,
>>>>>>     Mohammad Tariq
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 12, 2012 at 5:53 PM, imen Megdiche <
>>>>>> imen.megdiche@gmail.com> wrote:
>>>>>>
>>>>>>> i run the job through the command line
>>>>>>>
>>>>>>>
>>>>>>> 2012/12/12 Mohammad Tariq <dontariq@gmail.com>
>>>>>>>
>>>>>>>> You have to replace "JobTrackerHost" in "JobTrackerHost:50030"
>>>>>>>> with the actual name of the machine where JobTracker is running.
>>>>>>>> For example, If you are working on a local cluster, you have
to use
>>>>>>>> "localhost:50030".
>>>>>>>>
>>>>>>>> Are you running your job through the command line or some
IDE?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>     Mohammad Tariq
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Dec 12, 2012 at 5:42 PM, imen Megdiche <
>>>>>>>> imen.megdiche@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> excuse me the data size is 98 MB
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2012/12/12 imen Megdiche <imen.megdiche@gmail.com>
>>>>>>>>>
>>>>>>>>>> the size of data 49 MB and n of map 4
>>>>>>>>>> the web UI JobTrackerHost:50030 does not wok, what
should i do to
>>>>>>>>>> make this appear , i work on ubuntu
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2012/12/12 Mohammad Tariq <dontariq@gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Hi Imen,
>>>>>>>>>>>
>>>>>>>>>>>      You can visit the MR web UI at "JobTrackerHost:50030"
and
>>>>>>>>>>> see all the useful information like no. of mappers,
no of reducers, time
>>>>>>>>>>> taken  for the execution etc.
>>>>>>>>>>>
>>>>>>>>>>> One quick question for you, what is the size
of your data and
>>>>>>>>>>> what is the no of maps which you are getting
right now?
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>     Mohammad Tariq
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Dec 12, 2012 at 5:11 PM, imen Megdiche
<
>>>>>>>>>>> imen.megdiche@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thank you Mohammad but the number of map
tasks still the same
>>>>>>>>>>>> in the execution. Do you know how to capture
the time spent on execution.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2012/12/12 Mohammad Tariq <dontariq@gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Imen,
>>>>>>>>>>>>>
>>>>>>>>>>>>>     You can add "mapred.map.tasks" property
in your
>>>>>>>>>>>>> mapred-site.xml file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But, it is just a hint for the InputFormat.
Actually no. of
>>>>>>>>>>>>> maps is actually determined by the no
of InputSplits created by
>>>>>>>>>>>>> the InputFormat.
>>>>>>>>>>>>>
>>>>>>>>>>>>> HTH
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>     Mohammad Tariq
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Dec 12, 2012 at 4:11 PM, imen
Megdiche <
>>>>>>>>>>>>> imen.megdiche@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I try to force the number of map
for the mapreduce job with
>>>>>>>>>>>>>> the command :
>>>>>>>>>>>>>>   public static void main(String[]
args) throws Exception {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>           JobConf conf = new JobConf(WordCount.class);
>>>>>>>>>>>>>>              conf.set("mapred.job.tracker",
"local");
>>>>>>>>>>>>>>          conf.set("fs.default.name",
"local");
>>>>>>>>>>>>>>           conf.setJobName("wordcount");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>           conf.setOutputKeyClass(Text.class);
>>>>>>>>>>>>>>          conf.setOutputValueClass(IntWritable.class);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>            conf.setNumMapTask(6);
>>>>>>>>>>>>>>           conf.setMapperClass(Map.class);
>>>>>>>>>>>>>>           conf.setCombinerClass(Reduce.class);
>>>>>>>>>>>>>>           conf.setReducerClass(Reduce.class);
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But it doesn t work.
>>>>>>>>>>>>>> What can i do to modify the number
of map and reduce tasks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message