Mailing-List: contact dev-help@samza.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@samza.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAPOm=TOhHyt87unfdyuP_wNj_ZN2aBFr3M5es3cCiaCLjJf-gQ@mail.gmail.com>
References: 
 <CAFDmHdOVFqU=uxGwJK5=kqf9KaSvXBTaSVX8m9-k=LcV8iMjuA@mail.gmail.com>
 <CAPOm=TMLWgbV=_V9xJZO0bogPjD_aYOPNGbx-2xRm=bmXPHzTg@mail.gmail.com>
 <CAFDmHdOue43ph1R=fQbeGEk1286VNeKO0rDxYeGaGHzHAZjvHw@mail.gmail.com>
 <CAPOm=TOhHyt87unfdyuP_wNj_ZN2aBFr3M5es3cCiaCLjJf-gQ@mail.gmail.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Mon, 30 Nov 2015 12:31:39 -0800
Message-ID: 
 <CADiKvVtJUZ2q-2532hMZOJuJ3mzVRQ2VCNj=jeQs7vZrEE7K7A@mail.gmail.com>
Subject: Re: Executing Samza jobs natively in Kubernetes
To: dev@samza.apache.org
Content-Type: text/plain; charset=UTF-8

Hey Elias-
  This is awesome work.  Would be interested in opening JIRAs for the
changes you need so we can start to process them?

Thanks,
Jakob

On 30 November 2015 at 12:18, Roger Hoover <roger.hoover@gmail.com> wrote:
> Awesome.  Thanks.
>
> On Sun, Nov 29, 2015 at 3:25 PM, Elias Levy <fearsome.lucidity@gmail.com>
> wrote:
>
>> Roger,
>>
>> You are welcomed.  If you want to experiment, you can use my hello samza
>> <https://hub.docker.com/r/elevy/hello-samza/> Docker image.
>>
>> On Sun, Nov 29, 2015 at 12:19 PM, Roger Hoover <roger.hoover@gmail.com>
>> wrote:
>>
>> > Elias,
>> >
>> > I would also love to be able to deploy Samza on Kubernetes with dynamic
>> > task management.  Thanks for sharing this.  It may be a good interim
>> > solution.
>> >
>> > Roger
>> >
>> > On Sun, Nov 29, 2015 at 11:18 AM, Elias Levy <
>> fearsome.lucidity@gmail.com>
>> > wrote:
>> >
>> > > I've been exploring Samza for stream processing as well as Kubernetes
>> as
>> > a
>> > > container orchestration system and I wanted to be able to use one with
>> > the
>> > > other.  The prospect of having to execute YARN either along side or on
>> > top
>> > > of Kubernetes did not appeal to me, so I developed a KubernetesJob
>> > > implementation of SamzaJob.
>> > >
>> > > You can find the details at
>> > https://github.com/eliaslevy/samza_kubernetes,
>> > > but in summary KubernetesJob executes and generates a serialized
>> > JobModel.
>> > > Instead of interacting with Kubernetes directly to create the
>> > > SamzaContainers (as the YarnJob's SamzaApplicationMaster may do with
>> the
>> > > YARN RM), it output a config YAML file that can be used to create the
>> > > SamzaContainers in Kubernetes by using Resource Controllers.  For this
>> > you
>> > > require to package your job as a Docker image.  You can reach the
>> README
>> > at
>> > > the above repo for details.
>> > >
>> > > A few observations:
>> > >
>> > > It would be useful if SamzaContainer accepted the JobModel via an
>> > > environment variable.  Right not it expects a URL to download it
>> from.  I
>> > > get around this by using a entry point script that copies the model
>> from
>> > an
>> > > environment variable into a file, then passes a file URL to
>> > SamzaContainer.
>> > >
>> > > SamzaContainer doesn't allow you to configure the JMX port.  It
>> selects a
>> > > port at random from the ephemeral range as it expects to execute in
>> YARN
>> > > where a static port could result in a conflict.  This is not the case
>> in
>> > > Kubernetes where each Pod (i.e. SamzaContainer) is given its own IP
>> > > address.
>> > >
>> > > This implementation doesn't provide a Samza dashboard, which in the
>> YARN
>> > > implementation is hosted in the Application Master.  There didn't seem
>> to
>> > > be much value provided by the dashboard that is not already provided by
>> > the
>> > > Kubernetes tools for monitoring pods.
>> > >
>> > > I've successfully executed the hello-samza jobs in Kubernetes:
>> > >
>> > > $ kubectl get po
>> > > NAME                       READY     STATUS    RESTARTS   AGE
>> > > kafka-1-jjh8n              1/1       Running   0          2d
>> > > kafka-2-buycp              1/1       Running   0          2d
>> > > kafka-3-tghkp              1/1       Running   0          2d
>> > > wikipedia-feed-0-4its2     1/1       Running   0          1d
>> > > wikipedia-parser-0-l0onv   1/1       Running   0          17h
>> > > wikipedia-parser-1-crrxh   1/1       Running   0          17h
>> > > wikipedia-parser-2-1c5nn   1/1       Running   0          17h
>> > > wikipedia-stats-0-3gaiu    1/1       Running   0          16h
>> > > wikipedia-stats-1-j5qlk    1/1       Running   0          16h
>> > > wikipedia-stats-2-2laos    1/1       Running   0          16h
>> > > zookeeper-1-1sb4a          1/1       Running   0          2d
>> > > zookeeper-2-dndk7          1/1       Running   0          2d
>> > > zookeeper-3-46n09          1/1       Running   0          2d
>> > >
>> > >
>> > > Finally, accessing services within the Kubernetes cluster from the
>> > outside
>> > > is quite cumbersome unless one uses an external load balancer.  This
>> > makes
>> > > it difficult to bootstrap a job, as SamzaJob must connect to Zookeeper
>> > and
>> > > Kafka to find out the number of partitions on the topics it will
>> > subscribe
>> > > to, so it can assign them statically among the number of containers
>> > > requested.
>> > >
>> > > Ideally Samza would operate along the lines of the Kafka high-level
>> > > consumer, which dynamically coordinate to allocate work among members
>> of
>> > a
>> > > consumer group.  This would do away with the new to execute SamzaJob a
>> > > priori to generate the JobModel to pass to the SamzaContainers.  It
>> would
>> > > also allow for dynamically changing the number of containers without
>> > having
>> > > the shutdown the job.
>> > >
>> >
>>