incubator-mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Klingaman <>
Subject source port exhaustion on rapid task submission/failure
Date Tue, 11 Dec 2012 17:14:15 GMT
Has anyone else seen this behavior? I have a python implemented executor
and framework. Currently using 0.90 from the website. The end application
submits approximately 45K+ tasks to the framework for scheduling. Due to a
bug in my tasks, they fail immediately. It is still in the process of
submitting/failing when mesos-slave crashes and a netstat -tp indicates a
very large number of sockets in TIME_WAIT (between the single node and the
master) that belong to mesos-master. The source port is random (44700 in
the last run). The tasks only last about 1-2 seconds.

I'm assuming mesos-slave is crashing because it can't connect to the master
any more after source port exhaustion. It seems to me that the framework is
opening a new connection to mesos-master fairly frequently for task
status/submission. Maybe slave->master as well.

Fixing my own bug in the task, it works ok because the tasks finish in 1-2
seconds each, but there are still a fairly high number of TIME_WAIT sockets
indicating the problem is still there.

The last relevent mesos-slave crash lines:

F1211 08:41:41.626716 26415 process.cpp:1742] Check failed:
sockets.count(s) > 0
*** Check failure stack trace: ***
    @     0x7fc9e39adebd  google::LogMessage::Fail()
    @     0x7fc9e39b064f  google::LogMessage::SendToLog()
    @     0x7fc9e39adabb  google::LogMessage::Flush()
    @     0x7fc9e39b0edd  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fc9e38e5579  process::SocketManager::next()
    @     0x7fc9e38e0063  process::send_data()
    @     0x7fc9e39eb66f  ev_invoke_pending
    @     0x7fc9e39ef9a4  ev_loop
    @     0x7fc9e38e0fb7  process::serve()
    @     0x7fc9e32f1e9a  start_thread
    @     0x7fc9e2b08cbd  (unknown)

On a side note, I'm anxious to see the changelog for the next release.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message