asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wail Alkowaileet <>
Subject Re: New Asterix REST API design
Date Sat, 16 Apr 2016 10:41:11 GMT
Hi Till,

Just for a clarification, I have implemented a remote result iterator on
Asterix-Spark connector. The problem I had was when the result partitions
are in between Asterix and Spark, there are no disk-spilling mechanism. So
when the partitions are too big I get OOM. Now the result are delivered in
a limited number of frames (as you explained) and pull more frames when a
threshold is consumed.

The same technique can adopted in the GUI mode. However, this way can
introduce dangling results which would live for 24 hours. Probably, we
should have an "abort" mechanism when the session is ended/closed.

On Sat, Apr 16, 2016 at 8:35 AM, Mike Carey <> wrote:

> +1 for the separation and also for providing a place where we can deliver
> warnings - I remember longing for a few recently (we might have one or more
> JIRA issues mentioning that).....
> On 4/15/16 8:16 PM, Till Westmann wrote:
>> Yes, indeed the warning are feature-creep and not essential. We have been
>> talking about compiler warnings for along time, but we don’t have them so
>> far. The warnings that we could show for the HTTP API are warnings about
>> unused or ignored parameters but, again, they are not essential. It'll be
>> perfectly fine to add that at a later time when there's a concrete
>> need/use
>> for them.
>> Also, thinking about it a bit more, I'm even more convinced that it's the
>> better approach to have different fields for results and result handles as
>> the client can know if it needs another hop just by looking a the
>> response,
>> without knowing what the request was.
>> Cheers,
>> Till
>> On 15 Apr 2016, at 17:32, Ildar Absalyamov wrote:
>> Till,
>>> All the comments make sense to me except for warnings. I was struggling
>>> to remember anything in Asterix, which would resemble the warnings
>>> described by you.
>>> As a future extension we could support that, but in this document I was
>>> trying to reflect current state of Asterix functionality + some extensions
>>> which were on horizon for a long time and are really needed.
>>> On Apr 15, 2016, at 14:37, Till Westmann <> wrote:
>>>> Hi Ildar,
>>>> thanks for writing all of this up!
>>>> A few comments/proposals:
>>>> - Request:
>>>>  - For the "format" parameter, I think that it would be nice to support
>>>>    both, the parameter and the Accept header, as it’s sometimes much
>>>> more
>>>>    convenient to pass a parameter than a HTTP header. However, I think
>>>> that
>>>>    the HTTP header should override the parameter if they conflict.
>>>>  - "execute-query" could be renamed to "execute-statement" to be
>>>> consistent
>>>>    with the "statement" parameter.
>>>> - Response:
>>>>  - I'm wondering if "results" should be able to get a URI for the
>>>> handle or
>>>>    if we should have another field in the response for that (e.g.
>>>>    "handle"). The advantage of 2 fields would be that the consumer knows
>>>>    how to parse each field (either as a URI or as an array).
>>>>  - Inside of the "metrics" object I would only expect simple numbers.
>>>> For
>>>>    the plans we could have another top-level field ("plan"?) an object
>>>> that
>>>>    contains the different plans ..
>>>>  - It would also be nice to add a new top-level field for warnings. That
>>>>    could be used to report warnings from the engine that evaluates the
>>>>    statement. And it could also be used to report unused parameters
>>>>    (assuming that the default behavior for a parameter that is passed
>>>> in,
>>>>    but not understood by the server is simply ignored).
>>>> Thoughts?
>>>> Cheers,
>>>> Till
>>>> On 14 Apr 2016, at 15:23, Ildar Absalyamov wrote:
>>>> Hi Devs,
>>>>> Recently there have been a number of conversations about the future of
>>>>> our REST (aka HTTP) API. I summarized these discussions in an outline
>>>>> the new API design:
>>>>> <
>>>>> The need to refactor existing API came from different directions (and
>>>>> from different people), and is explained in motivation section. Thus
>>>>> believe it’s about the time to take an effort and improve existing
API, so
>>>>> that it will not drag us down in the future. However during the transition
>>>>> step I believe it would be better to keep exiting API endpoints, so that
>>>>> would not break people’s current experimental setup.
>>>>> It would be good to know feedback from the folks, who have been
>>>>> contributing to that part of the systems recently.
>>>>> Best regards,
>>>>> Ildar
>>> Best regards,
>>> Ildar


Wail Alkowaileet

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message