Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CABvqBf=9tju36Ci4084iTL5Wa_dbhZZFN=O1vV9z+e6uMHxvpw@mail.gmail.com>
References: <CACTd3c8=AOmrD+y7y4n5Tm_Kz_scyULsPXbNRGLa0dD2o4A2Fg@mail.gmail.com>
 <CACTd3c-8GrutYDpvjqy9f778tBzZ5iDM_uBBQ=ajmiUH2nrK-g@mail.gmail.com> <CABvqBf=9tju36Ci4084iTL5Wa_dbhZZFN=O1vV9z+e6uMHxvpw@mail.gmail.com>
From: Stavros Kontopoulos <st.kontopoulos@gmail.com>
Date: Fri, 8 Dec 2017 20:47:21 +0200
Message-ID: <CACTd3c9smx3YQWmU_QQch04oP+L4y440EmSdEKt01NLDMdxS=w@mail.gmail.com>
Subject: Re: queryable state & streaming
To: Burak Yavuz <brkyvz@gmail.com>
Cc: dev@spark.apache.org
Content-Type: multipart/alternative; boundary="f4030436158262addd055fd89d58"
archived-at: Fri, 08 Dec 2017 18:47:33 -0000

--f4030436158262addd055fd89d58
Content-Type: text/plain; charset="UTF-8"

Cool Burak do you have a pointer, should I take the initiative for a first
design document or Databricks is working on it?

Best,
Stavros

On Fri, Dec 8, 2017 at 8:40 PM, Burak Yavuz <brkyvz@gmail.com> wrote:

> Hi Stavros,
>
> Queryable state is definitely on the roadmap! We will revamp the
> StateStore API a bit, and a queryable StateStore is definitely one of the
> things we are thinking about during that revamp.
>
> Best,
> Burak
>
> On Dec 8, 2017 9:57 AM, "Stavros Kontopoulos" <st.kontopoulos@gmail.com>
> wrote:
>
>> Just to re-phrase my question: Would query-able state make a viable SPIP?
>>
>> Regards,
>> Stavros
>>
>> On Thu, Dec 7, 2017 at 1:34 PM, Stavros Kontopoulos <
>> st.kontopoulos@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Maybe this has been discussed before. Given the fact that many streaming
>>> apps out there use state extensively, could be a good idea to make Spark
>>> expose streaming state with an external API like other systems do (Kafka
>>> streams, Flink etc), in order to facilitate interactive queries?
>>>
>>> Regards,
>>> Stavros
>>>
>>
>>

--f4030436158262addd055fd89d58
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Cool Burak do you have a pointer, should I take the initia=
tive for a first design document or Databricks is working on it?<div><br></=
div><div>Best,</div><div>Stavros</div></div><div class=3D"gmail_extra"><br>=
<div class=3D"gmail_quote">On Fri, Dec 8, 2017 at 8:40 PM, Burak Yavuz <spa=
n dir=3D"ltr">&lt;<a href=3D"mailto:brkyvz@gmail.com" target=3D"_blank">brk=
yvz@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div =
dir=3D"auto">Hi Stavros,<div dir=3D"auto"><br></div><div dir=3D"auto">Query=
able state is definitely on the roadmap! We will revamp the StateStore API =
a bit, and a queryable StateStore is definitely one of the things we are th=
inking about during that revamp.</div><div dir=3D"auto"><br></div><div dir=
=3D"auto">Best,</div><div dir=3D"auto">Burak</div></div><div class=3D"HOEnZ=
b"><div class=3D"h5"><div class=3D"gmail_extra"><br><div class=3D"gmail_quo=
te">On Dec 8, 2017 9:57 AM, &quot;Stavros Kontopoulos&quot; &lt;<a href=3D"=
mailto:st.kontopoulos@gmail.com" target=3D"_blank">st.kontopoulos@gmail.com=
</a>&gt; wrote:<br type=3D"attribution"><blockquote class=3D"gmail_quote" s=
tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div=
 dir=3D"ltr">Just to re-phrase my question: Would=C2=A0<span style=3D"font-=
size:12.8px">query-able state make a viable SPIP?</span><br><div><span styl=
e=3D"font-size:12.8px"><br></span></div><div><span style=3D"font-size:12.8p=
x">Regards,</span></div><div><span style=3D"font-size:12.8px">Stavros</span=
></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On T=
hu, Dec 7, 2017 at 1:34 PM, Stavros Kontopoulos <span dir=3D"ltr">&lt;<a hr=
ef=3D"mailto:st.kontopoulos@gmail.com" target=3D"_blank">st.kontopoulos@gma=
il.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"=
margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"=
ltr"><div><span style=3D"font-size:12.8px">Hi,</span></div><div><span style=
=3D"font-size:12.8px"><br></span></div><div><span style=3D"font-size:12.8px=
">Maybe this has been discussed before. Given the fact that many streaming =
apps out there use state extensively, could be a good idea to make Spark ex=
pose streaming state=C2=A0</span><span style=3D"font-size:12.8px">with an e=
xternal API like other systems do (</span><span style=3D"font-size:12.8px">=
Kafka streams, Flink etc),</span><span style=3D"font-size:12.8px">=C2=A0in =
order to facilitate interactive queries?</span></div><div><br></div><div><s=
pan style=3D"font-size:12.8px">Regards,</span><br></div><div><span style=3D=
"font-size:12.8px">Stavros</span></div></div>
</blockquote></div><br></div>
</blockquote></div></div>
</div></div></blockquote></div><br></div>

--f4030436158262addd055fd89d58--