mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikolaos Ballas neXus <>
Subject Re: Batch/queue frameworks?
Date Wed, 07 Oct 2015 10:28:07 GMT
I think any pub/sub system(name it typical jms / rabbitmq/ kafka) etc would do what you describe.
All of them can be run as containers inside apache mess cluster. Kafka has really good integration
with MEsos and YARN and also is more lightweight than a typical jus implementation.

On 07 Oct 2015, at 12:05, F21 <<>>

I am also interested in something like this, although my requirements are much more simpler.

I am interested in a work queue like beanstalkd that will allow me to push to a queue from
a web app and have workers to do things like send emails, generate pdfs and resize images.

I have thought about running a beanstalkd in a container, but it has some limitations. For
example, if it crashes, it needs to be relaunched manually to recover the binlog (which is
a no go).

Another option I can think of is to use kafka (which has a mesos framework) and have the web
app and other parts push jobs into the kafka broker. Workers listening on the broker would
pop each job off and execute whatever needs to be done.

However, there seems to be a lot of wheel-reinventing what that solution. For example, what
if a job depends on another job? There's also a lot of work that needs to be done at a lower
level when all I am interested in is to write domain specific code to generate the pdf, resize
the image etc.

If there's a work queue solution for mesos, I would love to know too.

On 7/10/2015 8:08 PM, Brian Candler wrote:
On 07/10/2015 09:44, Nikolaos Ballas neXus wrote:
Maybe you need to read a bit  :)
I have read plenty, including those you list, and I didn't find anything which met my requirements.
Again I apologise if I was not clear in my question.

Spark has a very specific data model (RDDs) and applications which write to its API. I want
to run arbitrary compute jobs - think "shell scripts" or "docker containers" which run pre-existing
applications which I can't change.  And I want to fill a queue or pipeline with those jobs.

Hadoop also is for specific workloads, written to run under Hadoop and preferably using HDFS.

The nearest Hadoop gets to general-purpose computing, as far as I can see, is its YARN scheduler.
YARN can in turn run under Mesos. Therefore a job queue which can run on YARN might be acceptable,
although I'd rather not have an additional layer in the stack. (There was an old project for
running Torque under YARN, but this has been abandoned)



Nikolaos Ballas  |  Software Development Manager

Technology Nexus S.a.r.l.
2-4 Rue Eugene Rupert
2453 Luxembourg
Delivery address: 2-3 Rue Eugene Rupert,Vertigo Polaris Building
Tel: + 3522619113580<> |<><> | Twitter<>


View raw message