asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Till Westmann" <ti...@apache.org>
Subject Re: New Asterix REST API design
Date Mon, 18 Apr 2016 22:36:29 GMT
Hi Wail,

so you are actually not looking for for a way to limit the size, but 
really
for a way to paginate the whole result, is that right?

IIRC the question of GC for the results materialized by the result
distribution is not nicely solved anyway, so that might not bring 
additional
issues for now ... but we also shouldn't introduce new heavy 
requirements on
this.

Cheers,
Till

On 16 Apr 2016, at 3:41, Wail Alkowaileet wrote:

> Hi Till,
>
> Just for a clarification, I have implemented a remote result iterator 
> on
> Asterix-Spark connector. The problem I had was when the result 
> partitions
> are in between Asterix and Spark, there are no disk-spilling 
> mechanism. So
> when the partitions are too big I get OOM. Now the result are 
> delivered in
> a limited number of frames (as you explained) and pull more frames 
> when a
> threshold is consumed.
>
> The same technique can adopted in the GUI mode. However, this way can
> introduce dangling results which would live for 24 hours. Probably, we
> should have an "abort" mechanism when the session is ended/closed.
>
> On Sat, Apr 16, 2016 at 8:35 AM, Mike Carey <dtabass@gmail.com> wrote:
>
>> +1 for the separation and also for providing a place where we can 
>> deliver
>> warnings - I remember longing for a few recently (we might have one 
>> or more
>> JIRA issues mentioning that).....
>>
>>
>> On 4/15/16 8:16 PM, Till Westmann wrote:
>>
>>> Yes, indeed the warning are feature-creep and not essential. We have 
>>> been
>>> talking about compiler warnings for along time, but we don’t have 
>>> them so
>>> far. The warnings that we could show for the HTTP API are warnings 
>>> about
>>> unused or ignored parameters but, again, they are not essential. 
>>> It'll be
>>> perfectly fine to add that at a later time when there's a concrete
>>> need/use
>>> for them.
>>>
>>> Also, thinking about it a bit more, I'm even more convinced that 
>>> it's the
>>> better approach to have different fields for results and result 
>>> handles as
>>> the client can know if it needs another hop just by looking a the
>>> response,
>>> without knowing what the request was.
>>>
>>> Cheers,
>>> Till
>>>
>>> On 15 Apr 2016, at 17:32, Ildar Absalyamov wrote:
>>>
>>> Till,
>>>>
>>>> All the comments make sense to me except for warnings. I was 
>>>> struggling
>>>> to remember anything in Asterix, which would resemble the warnings
>>>> described by you.
>>>> As a future extension we could support that, but in this document I 
>>>> was
>>>> trying to reflect current state of Asterix functionality + some 
>>>> extensions
>>>> which were on horizon for a long time and are really needed.
>>>>
>>>> On Apr 15, 2016, at 14:37, Till Westmann <tillw@apache.org> wrote:
>>>>>
>>>>> Hi Ildar,
>>>>>
>>>>> thanks for writing all of this up!
>>>>>
>>>>> A few comments/proposals:
>>>>>
>>>>> - Request:
>>>>>  - For the "format" parameter, I think that it would be nice to 
>>>>> support
>>>>>    both, the parameter and the Accept header, as it’s sometimes 
>>>>> much
>>>>> more
>>>>>    convenient to pass a parameter than a HTTP header. However, I 
>>>>> think
>>>>> that
>>>>>    the HTTP header should override the parameter if they conflict.
>>>>>  - "execute-query" could be renamed to "execute-statement" to be
>>>>> consistent
>>>>>    with the "statement" parameter.
>>>>> - Response:
>>>>>  - I'm wondering if "results" should be able to get a URI for the
>>>>> handle or
>>>>>    if we should have another field in the response for that (e.g.
>>>>>    "handle"). The advantage of 2 fields would be that the consumer 
>>>>> knows
>>>>>    how to parse each field (either as a URI or as an array).
>>>>>  - Inside of the "metrics" object I would only expect simple 
>>>>> numbers.
>>>>> For
>>>>>    the plans we could have another top-level field ("plan"?) an 
>>>>> object
>>>>> that
>>>>>    contains the different plans ..
>>>>>  - It would also be nice to add a new top-level field for 
>>>>> warnings. That
>>>>>    could be used to report warnings from the engine that evaluates 
>>>>> the
>>>>>    statement. And it could also be used to report unused 
>>>>> parameters
>>>>>    (assuming that the default behavior for a parameter that is 
>>>>> passed
>>>>> in,
>>>>>    but not understood by the server is simply ignored).
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On 14 Apr 2016, at 15:23, Ildar Absalyamov wrote:
>>>>>
>>>>> Hi Devs,
>>>>>>
>>>>>> Recently there have been a number of conversations about the 
>>>>>> future of
>>>>>> our REST (aka HTTP) API. I summarized these discussions in an 
>>>>>> outline of
>>>>>> the new API design:
>>>>>> https://cwiki.apache.org/confluence/display/ASTERIXDB/New+HTTP+API+Design
>>>>>> <
>>>>>> https://cwiki.apache.org/confluence/display/ASTERIXDB/New+HTTP+API+Design>.
>>>>>>
>>>>>> The need to refactor existing API came from different directions

>>>>>> (and
>>>>>> from different people), and is explained in motivation section. 
>>>>>> Thus I
>>>>>> believe it’s about the time to take an effort and improve 
>>>>>> existing API, so
>>>>>> that it will not drag us down in the future. However during the 
>>>>>> transition
>>>>>> step I believe it would be better to keep exiting API endpoints,

>>>>>> so that we
>>>>>> would not break people’s current experimental setup.
>>>>>>
>>>>>> It would be good to know feedback from the folks, who have been
>>>>>> contributing to that part of the systems recently.
>>>>>>
>>>>>> Best regards,
>>>>>> Ildar
>>>>>>
>>>>>
>>>> Best regards,
>>>> Ildar
>>>>
>>>
>>
>
>
> -- 
>
> *Regards,*
> Wail Alkowaileet

Mime
View raw message