mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vvshvv <vvs...@gmail.com>
Subject Re: framework for short living tasks (1-15 seconds)
Date Mon, 27 Mar 2017 20:41:36 GMT
<div dir="ltr">Sorry, I replied not to all the mailing list.<br>
</div><div dir="ltr"><br>
</div><div dir="ltr"><br>
</div><div dir="ltr">Regards,<br>
</div><div dir="ltr">Uladzimir</div><div class="wps_quotion">On Mar
27, 2017 11:39 PM, vvshvv &lt;vvshvv@gmail.com&gt; wrote:<br type="attribution"><blockquote
class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div
dir="ltr">When user uploads some image to a REST endpoint, it should be processed in a
separate Docker container. Actually, I can implement my custom mesos framework that will consume
uploading events from a Kafka topic and will run tasks on slaves (in containers), but it is
slightly difficult, because when message arrives, cluster can have no available resources
to process it and it has to wait until mesos offers some resources in a queue. And when messages​
are waiting in a queue in memory, process can crash and I will lose all pending events. <br>
</div><div dir="ltr"><br>
</div><div dir="ltr">At the same time I am aware of Marathon, but I see that it
is suitable only for long tasks and I am not sure that it will handle large amount of jobs.<br>
</div><div dir="ltr"><br>
</div><div dir="ltr"><br>
</div><div dir="ltr">Regards,<br>
</div><div dir="ltr">Uladzimir</div><div class="wps_quotion">On Mar
27, 2017 11:28 PM, vvshvv &lt;vvshvv@gmail.com&gt; wrote:<br type="attribution"><blockquote
class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div
dir="ltr">It seems you did not understand what I meant. My processing includes metadata
extraction and resizing, each job should have isolated environment (limited cpu and memory).
Processing will be either in Golang or even C++.<br>
</div><div dir="ltr"><br>
</div><div dir="ltr"><br>
</div><div dir="ltr"><br>
</div><div class="wps_signature"></div><div class="wps_quotion">On
Vaibhav Khanduja &lt;vaibhavkhanduja@gmail.com&gt;, Mar 27, 2017 11:20 PM wrote:<br
type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"><p></p><div dir="ltr">Try deeplearning4j
with spark extensions ..<br><br><br><br></div><div class="gmail_extra"><br><div
class="gmail_quote">On Mon, Mar 27, 2017 at 1:09 PM, daemeon reiydelle <span dir="ltr">&lt;<a
href="mailto:daemeonr@gmail.com" target="_blank">daemeonr@gmail.com</a>&gt;</span>
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:comic
sans ms,sans-serif;color:rgb(7,55,99)">Spark cached RDD&#39;s<br></div></div><div
class="gmail_extra"><br clear="all"><div><div class="m_8017918056782816101gmail_signature"><div
dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(56,118,29)"><span
style="background-color:rgb(255,255,255)"><b><span style="font-family:comic sans
ms,sans-serif"></span></b></span></span><span style="color:rgb(56,118,29)"><span
style="background-color:rgb(255,255,255)"><b><span style="font-family:comic sans
ms,sans-serif"><br>.......</span></b></span></span><span
style="color:rgb(56,118,29)"><span style="background-color:rgb(255,255,255)"><b><span
style="font-family:comic sans ms,sans-serif"><br><br>Daemeon C.M. Reiydelle<br>USA
<a href="tel:(415)%20501-0198" value="+14155010198" target="_blank">(+1) 415.501.0198</a><br>London
<a href="tel:+44%2020%208144%209872" value="+442081449872" target="_blank">(+44) (0)
20 8144 9872</a></span></b></span></span><font size="1"><i><br></i></font></div></div></div></div></div></div></div><div><div
class="h5">
<br><div class="gmail_quote">On Mon, Mar 27, 2017 at 11:52 AM, vvshvv <span
dir="ltr">&lt;<a href="mailto:vvshvv@gmail.com" target="_blank">vvshvv@gmail.com</a>&gt;</span>
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"><div dir="ltr">Hi,<br>
</div><div dir="ltr"><br>
</div><div dir="ltr">I want to run some image processing tasks on mesos cluster,
but at the same time there will be a lot of tasks (1k-10k).<br>
</div><div dir="ltr"><br>
</div><div dir="ltr">What framework do you suggest to use in such case?<br>
</div><div dir="ltr"><br>
</div><div dir="ltr">Regards,<br>
</div><div dir="ltr">Uladzimir</div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</blockquote></div></blockquote></div></blockquote></div>
Mime
View raw message