hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Costa <psdc1...@gmail.com>
Subject Re: cleanup task doesn't run always
Date Thu, 09 Jun 2011 09:14:59 GMT
Sorry, but what's the answer from Aaron? It's not in this mail.
And, I think that the cleanup task has nothing to do with HDFS. As far
as I can see, the hadoop contains a setup and a clean up task for map
and reduce tasks. I don't understand why, but it has. Here's the part
of the code that I'm talking about:

[code]
   // cleanup map tip. This map doesn't use any splits. Just assign an empty
    // split.
    JobClient.RawSplit emptySplit = new JobClient.RawSplit();
    cleanup[0] = new TaskInProgress(jobId, jobFile, emptySplit,
            jobtracker, conf, this, numMapTasks);
    cleanup[0].setJobCleanupTask();

    // cleanup reduce tip.
    cleanup[1] = new TaskInProgress(jobId, jobFile, numMapTasks,
                       numReduceTasks, jobtracker, conf, this);
    cleanup[1].setJobCleanupTask();

    // create two setup tips, one map and one reduce.
    setup = new TaskInProgress[2];

    // setup map tip. This map doesn't use any split. Just assign an empty
    // split.
    setup[0] = new TaskInProgress(jobId, jobFile, emptySplit,
            jobtracker, conf, this, numMapTasks + 1 );
    setup[0].setJobSetupTask();

    // setup reduce tip.
    setup[1] = new TaskInProgress(jobId, jobFile, numMapTasks,
                       numReduceTasks + 1, jobtracker, conf, this);
    setup[1].setJobSetupTask();
[/code]

If anyone could explain also what's the purpose of these tasks I would
be very appreciated?

Thanks,

On Thu, Jun 9, 2011 at 9:05 AM, Laurent Hatier <laurent.hatier@gmail.com> wrote:
> Oh i don't see that it was in the HDFS. Aaron has answered i think
>
> 2011/6/9 Laurent Hatier <laurent.hatier@gmail.com>
>>
>> Have you try to restart your hadoop node ? (or all hadoop node). When you
>> go to restart, the namenode go to format the directory that you have choosen
>>
>> 2011/6/8 Pedro Costa <psdc1978@gmail.com>
>>>
>>> Hi,
>>>
>>> After I run the command "bin/hadoop job -history /temp/history/", I've
>>> got these 2 task summary. In one of them, it run a cleanup task and in
>>> the other hasn't run the cleanup task. This means that a cleanup task
>>> doesn't run always. So, when a cleanup task should run?
>>>
>>> Task Summary
>>> ============================
>>> Kind    Total   Successful      Failed  Killed  StartTime
>>> FinishTime
>>>
>>> Setup   1       1               0       0       8-Jun-2011 15:54:50
>>>  8-Jun-2011 15:54:51 (0sec)
>>> Map     600     600             0       0       8-Jun-2011 15:54:54
>>>  8-Jun-2011 15:57:05 (2mins, 11sec)
>>> Reduce  252     250             0       2       8-Jun-2011 15:55:06
>>>  8-Jun-2011 15:57:48 (2mins, 42sec)
>>> Cleanup 0       0               0       0
>>> ============================
>>>
>>>
>>>
>>> Task Summary
>>> ============================
>>> Kind    Total   Successful      Failed  Killed  StartTime
>>> FinishTime
>>>
>>> Setup   1       1               0       0       8-Jun-2011 16:04:19
>>>  8-Jun-2011 16:04:20 (1sec)
>>> Map     500     383             0       117             8-Jun-2011
>>> 16:04:40
>>> Reduce  0       0               0       0
>>> Cleanup 1       1               0       0       8-Jun-2011 16:04:37
>>>  8-Jun-2011 16:04:38 (1sec)
>>> ============================
>>>
>>> Thanks,
>>
>>
>>
>> --
>> Laurent HATIER
>> Étudiant en 2e année du Cycle Ingénieur à l'EISTI
>
>
>
> --
> Laurent HATIER
> Étudiant en 2e année du Cycle Ingénieur à l'EISTI
>

Mime
View raw message