spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: Multi-tenancy for Spark (Streaming) Applications
Date Tue, 09 Sep 2014 02:26:19 GMT
Hi,

On Thu, Sep 4, 2014 at 10:33 AM, Tathagata Das <tathagata.das1565@gmail.com>
wrote:

> In the current state of Spark Streaming, creating separate Java processes
> each having a streaming context is probably the best approach to
> dynamically adding and removing of input sources. All of these should be
> able to to use a YARN cluster for resource allocation.
>

So, for example, I would write a server application that accepts a command
like "createNewInstance" and then calls spark-submit, pushing my actual
application to the YARN cluster? Or could I use spark-jobserver?

Thanks
Tobias

Mime
View raw message