mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Mackles <pa...@loopr.com>
Subject Re: hadoop task-trackers sticking around
Date Fri, 27 Sep 2013 10:58:34 GMT
I see the following messages in the job-tracker logs which probably explain
why the task-trackers are sticking around:

2013-09-27 03:39:58,808 WARN org.apache.hadoop.mapred.MesosScheduler:
Ignoring TaskTracker: http://vm282.dev.xxx:31001 because it might not have
sent a hearbeat
2013-09-27 03:39:58,808 WARN org.apache.hadoop.mapred.MesosScheduler:
Ignoring TaskTracker: http://vm282.dev.xxx:31000 because it might not have
sent a hearbeat
2013-09-27 03:39:58,809 WARN org.apache.hadoop.mapred.MesosScheduler:
Ignoring TaskTracker: http://vm282.dev.xxx:31001 because it might not have
sent a hearbeat
2013-09-27 03:39:58,809 WARN org.apache.hadoop.mapred.MesosScheduler:
Ignoring TaskTracker: http://vm282.dev.xxx:31000 because it might not have
sent a hearbeat

The source for MesosScheduler.java that is bundled with 0.13 looks quite a
bit different than the version that is currently on git.



On Thu, Sep 26, 2013 at 11:08 PM, Paul Mackles <paulm@loopr.com> wrote:

> I will dig a little further as the behavior is inconsistent. On subsequent
> attempts I have seen the task-trackers go away with the job. They always go
> away when I shutdown the corresponding job-tracker.
>
> The hadoop code I am using was included in the 0.13 tarball that I
> downloaded from here:
>
> http://mirror.nexcess.net/apache/mesos/0.13.0/
>
> I built the jar by running hadoop/TUTORIAL.sh. I wound up integrating with
> hadoop manually since the tutorial script didn't work correctly for me. I
> mostly followed the instructions here: https://github.com/mesos/hadoop
>
> At one point I tried building it from https://github.com/mesos/hadoop but
> I had trouble getting it to build with 0.13.
>
> Should I be working off of a different version?
>
> Thanks,
> Paul
>
>
>
> On Thu, Sep 26, 2013 at 10:34 PM, Dan Colish <dcolish@urbanairship.com>wrote:
>
>>
>>
>>
>> On Thu, Sep 26, 2013 at 6:27 PM, Paul Mackles <paulm@loopr.com> wrote:
>>
>>> Hi - I am using mesos 0.13 with cdh4.2.0 in pseudo-distributed mode.
>>> While I am able to launch and run hadoop jobs through mesos successfully, I
>>> noticed in the Mesos UI (and through 'ps') that the task-trackers launched
>>> by mesos are sticking around long after my job is complete. Is that
>>> expected behavior? I am thinking the answer is no since they are tying up
>>> resources that could be used by other frameworks. On the other hand, mesos
>>> seems to know enough to reuse them when running subsequent hadoop jobs.
>>> Maybe there are using reservations or something by default?
>>>
>>>
>> Are you using the mesos-hadoop project found here,
>> https://github.com/mesos/hadoop? If so, you are correct that idle
>> tasktrackers should be torn down at the end. I wonder what the cluster
>> state when the JobInProgressListener is called with upon your job's
>> completion. Specifically, I would look into tracing this section [1] of code
>> * *where the task trackers job queue is checked for emptiness the
>> tracker is checked for being active. If the tracker was never activated I
>> think it would also be running but not killed.
>>
>>
>> [1]
>> https://github.com/mesos/hadoop/blob/master/src/main/java/org/apache/hadoop/mapred/MesosScheduler.java#L105
>>
>>
>
>
> --
> Thanks,
> Paul
>



-- 
Thanks,
Paul

Mime
View raw message