aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Isaac Councill <is...@hioscar.com>
Subject Re: api vs apibeta
Date Mon, 29 Dec 2014 21:32:01 GMT
Ok, here's code for the pure json apibeta bench. I realized it was a bit
unfair because I was making concurrent requests (max 3). Stripping out all
concurrency, it's coming in at 17.2sec.

drop in src/aurora/, no dependencies required.

go build o build -o dist/aurora2 aurora/client2/bench
dist/aurora2 -url="http://<aurora_master>:8081/apibeta"


On Mon, Dec 29, 2014 at 3:36 PM, Isaac Councill <isaac@hioscar.com> wrote:

> Here's source from go thrift (warning: very ugly). I had to make a few
> modifications to the ttypes and client libraries to get it working. It
> requires git.apache.org/thrift.git from the 0.9.2 tree (0.9.1 generates
> code that is much farther from correct).
>
> after putting it in src/aurora/client:
> go build -o dist/aurora aurora/client/thrift/read_only_scheduler-remote
>
> then:
> time dist/aurora -u="http://<aurora_master>:8081/api" -http=true -P=json
> doBench
>
> Perhaps my general unhappiness with thrift right now has to due with
> immature go support. Just take a look at that generated code, and it didn't
> actually work, at all, without some messing around.
>
> Re: Kevin's question about differences between getTaskStatus requests, the
> contents I was filling in are the same (role, environment, jobName), but
> inspecting network traffic, it did appear that the taskStatusRequest from
> go thrift is adding empty values for nil list fields, which could impact
> the results.
>
> As for the apibeta client, it will take me a little extra time to rip that
> out into a sharable form.
>
> On Mon, Dec 29, 2014 at 2:17 PM, Jake Farrell <jfarrell@apache.org> wrote:
>
>> Hey Isaac
>> Would love to hear your pain points with Thrift and also can you share
>> your
>> source for the test clients
>>
>> -Jake
>>
>> On Mon, Dec 29, 2014 at 1:27 AM, Isaac Councill <isaac@hioscar.com>
>> wrote:
>>
>> > tl;dr;
>> > apibeta seems way faster (and arguably better) than thrift api. What are
>> > the long term objectives for apibeta?
>> >
>> >
>> > Hi,
>> >
>> > I've been working on some aurora integrations, primarily a blackbox
>> > monitoring tool at present, and was looking for the best way to
>> communicate
>> > with the scheduler.
>> >
>> > For a large read-only example, I wanted to dump the latest scheduler
>> status
>> > info for all our prod jobs, basically:
>> >
>> > for all roles:
>> >   for all jobs in role:
>> >     get scheduler status
>> >
>> > We have about 120 prod jobs in aurora right now (growing fast). I
>> > benchmarked 3 strategies against our prod cluster (mean of 5 tries each
>> > from remote vpn, variance was small in each case):
>> >
>> > 1) aurora2 client: ./aurora2 job status cluster/<role>/prod > /dev/null
>> > 126.0sec
>> >
>> > 2) golang thrift API
>> > 584.3sec (I might be able make a better task query, but still... this is
>> > for only ~120 calls)
>> >
>> > 3) Pure json apibeta client in golang
>> > 13.4sec (again, might be able to optimize query strategy)
>> >
>> > As a side note, getting the golang thrift client to work was a very
>> painful
>> > and illuminating experience.
>> >
>> > I'm inclined to stick with apibeta. It's fast and the documentation is
>> > great. If api changes become a concern, well after today I'd honestly
>> > prefer rolling my own binding generator.
>> >
>> > Are there plans for /apibeta wrt /api?
>> >
>>
>
>

Mime
View raw message