spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@databricks.com>
Subject Re: Jenkins networking / port contention
Date Fri, 01 Jul 2016 16:45:07 GMT
Multiple instances of test runs are usually running in parallel, so they
would need to bind to different ports.

On Friday, July 1, 2016, Cody Koeninger <cody@koeninger.org> wrote:

> Thanks for the response.  I'm talking about test code that starts up
> embedded network services for integration testing.
>
> KafkaTestUtils in particular always attempts to start a kafka broker
> on the standard port, 9092.  Util.startServiceInPort is intended to
> pick a higher port if the starting one has a bind collision... but in
> my local testing multiple KafkaTestUtils instances running at the same
> time on the same machine don't actually behave correctly.
>
> I already updated the kafka 0.10 consumer tests to use a random port,
> and can do the same for the 0.8 consumer tests, but wanted to make
> sure I understood what was happening in the Jenkins environment.
>
> On Fri, Jul 1, 2016 at 11:18 AM, shane knapp <sknapp@berkeley.edu
> <javascript:;>> wrote:
> > i assume you're talking about zinc ports?
> >
> > the tests are designed to run one at a time on randomized ports -- no
> > containerization.  we're on bare metal.
> >
> > the test launch code executes this for each build:
> > # Generate random point for Zinc
> > export ZINC_PORT
> > ZINC_PORT=$(python -S -c "import random; print
> random.randrange(3030,4030)")
> >
> > On Fri, Jul 1, 2016 at 6:02 AM, Cody Koeninger <cody@koeninger.org
> <javascript:;>> wrote:
> >> Can someone familiar with amplab's jenkins setup clarify whether all
> tests
> >> running at a given time are competing for network ports, or whether
> there's
> >> some sort of containerization being done?
> >>
> >> Based on the use of Utils.startServiceOnPort in the tests, I'd assume
> the
> >> former.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <javascript:;>
>
>

Mime
View raw message