drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Altekruse <altekruseja...@gmail.com>
Subject Re: setting session params in a rest request
Date Thu, 04 Feb 2016 21:27:01 GMT
Venki,

Even for authenticated sessions, do we impose any kind of upper limit on
the number of concurrent connections? I seems like we should issue a
reasonable error message about an over-used Drillbit before the process
dies due to hitting the system file handle limit.

- Jason

On Thu, Feb 4, 2016 at 11:00 AM, Josh Schlesser <josh@spoutable.com> wrote:

> Thanks Venki,
> Let me know how I can help further.
>
> > On Feb 2, 2016, at 10:23 PM, Venki Korukanti <venki.korukanti@gmail.com>
> wrote:
> >
> > I think you made valid points. It makes sense to have session less REST
> > calls in both auth enabled and disabled cases.
> >
> > In case of auth enabled:
> > 1) Session-less calls can be authenticated using Basic auth (this was
> > already asked on mailing list sometime back) as a start and move onto
> token
> > based auth later. These requests usually come from non-browsers. The only
> > issue is setting options before the query. For this we can implement your
> > suggestion of enhancing the query REST API to accept the options.
> > 2) Session-based call using form auth for browser based access. If we
> > enhance the UI to enter options in the query form, we don't need any
> > session on server actually.
> >
> > I will get a fix ASAP to remove the sessions in anonymous calls as they
> the
> > session are not reused in non-browser cases.
> >
> > Thanks
> > Venki
> >
> > On Tue, Feb 2, 2016 at 9:20 PM, Josh Schlesser <josh@spoutable.com>
> wrote:
> >
> >> No, it wasn’t logging out, it was just stopping, obviously that caused
> >> dangling sessions for the authenticated scenario.
> >>
> >> I don’t think that a short timeout for anonymous sessions is a good way
> to
> >> go for anonymous api calls.  Session management isn’t what anybody would
> >> expect when using a REST api that is anonymous in a server to server
> >> context.   I would expect to use a token for authorization for a server
> to
> >> server REST api as well.  I’m not saying that is what it should be here,
> >> but that is what my general expectation is based on using other apis.
>  In
> >> the case of browser to server REST apis, I have run into authentication
> for
> >> a browser session and subsequent REST calls leaning on a browser cookie
> for
> >> persistent authentication.
> >>
> >> Removing sessions for anonymous calls seems like the right path and
> >> possibly easy and I think would be the expected behavior from most
> >> developers.  I would advocate for sessionless and token authenticated
> REST
> >> apis for when using authentication for the server to server case and
> cookie
> >> based with a session for the browser to server scenario, but its really
> the
> >> browser that has a session, not the api per se, its  just piggybacking
> on a
> >> regular authenticated web session for the REST api calls.
> >>
> >> This would actually leave me in a quandary for what I am trying to do
> >> which is set a session configuration option ’store.format', but I cant
> >> think of any reason that those types of settings shouldn’t just be set
> on a
> >> per request basis for a REST api.  In a server to server context for a
> rest
> >> api, keeping it sessionless means you could front a cluster of drillbits
> >> with a load balancer and not worry about dying nodes and sticky sessions
> >> etc...
> >>
> >> I have to get something up and running quickly right now so im
> versioning
> >> back to 1.4 and just spinning up a separate drillbit that will have the
> >> store.format system variable set to ‘json’ . it will be ok for me until
> a
> >> good long term solution arrives in drill.
> >>
> >> I’ll run the test on short session_max_idle_secs to 30 seconds on
> >> 1.5.0-SNAPSHOT to see if that gets rid of the file handle starvation
> >> problem, but keep in mind that means that users of the web console will
> >> have 30 seconds between pages or they have to authenticate again, which
> >> will probably be very annoying.  It doesnt seem like a good long term
> >> solution either.
> >>
> >> How do you think all of this should work?  I look forward to staying
> >> involved.
> >>
> >> Cheers,
> >> Josh
> >>
> >>> On Feb 2, 2016, at 4:40 PM, Venki Korukanti <venki.korukanti@gmail.com
> >
> >> wrote:
> >>>
> >>> When auth is *enabled*, is the worker process logging out after queries
> >> are
> >>> done? When auth is *disabled* can you set session_max_idle_secs in
> >>> drill.exec.http block in drill-override.conf to something like 30
> (secs)
> >>> and try? This way anonymous sessions are closed quickly and not kept
> for
> >>> 1hr (default value). I think we may need to avoid creating sessions in
> >>> anonymous mode (when auth is disabled).
> >>>
> >>> Thanks
> >>> Venki
> >>>
> >>> On Tue, Feb 2, 2016 at 4:02 PM, Josh Schlesser <josh@spoutable.com>
> >> wrote:
> >>>
> >>>> I have a background worker process (on a server, not a browser) that
> >> kicks
> >>>> off every minute or so and issues some queries sequentially to the
> rest
> >>>> query endpoint.    In 1.4 with no authentication this worked fine
> except
> >>>> that in 1 instance I need to issue a CTAS query with a different
> format
> >>>> (json).
> >>>>
> >>>> I upgraded to 1.5-SNAPSHOT commit
> >> bb3fc15216d9cab804fc9a6f0e5bd34597dd4394
> >>>>
> >>>> Since the upgrade I am getting a resource starvation problem with or
> >>>> without authentication
> >>>> The drillbit process stays up for a an hour or less and then becomes
> >>>> unresponsive and eats up the cpu.
> >>>>
> >>>> It is definitely a resource starvation issue, not sure if its a
> resource
> >>>> leak.
> >>>> Below is a stack trace.
> >>>> Also when i lsof on the pid there are a lot (more than a thousand) of
> >>>> files like this listed which are used by NIO selectors.  so it smells
> >> like
> >>>> a resource leak.
> >>>>
> >>>> COMMAND  PID USER   FD   TYPE             DEVICE  SIZE/OFF    NODE
> NAME
> >>>> java    2931 root  288u  0000               0,11         0    7705
> >>>> anon_inode
> >>>>
> >>>> 2016-02-02 21:56:26,520 [qtp1250890858-11590] ERROR
> >>>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> >>>> java.lang.IllegalStateException: failed to create a child event loop
> >>>>       at
> >>>>
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>>>       at
> >>>>
> >>
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> >>>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> >>>>       at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>>       at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>>       at
> >>>>
> >>
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> >>>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>>
> >>
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> >>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> >>>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> >>>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at org.eclipse.jetty.server.Server.handle(Server.java:462)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> >>>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> >>>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too
> many
> >>>> open files
> >>>>       at io.netty.channel.epoll.Native.epollCreate(Native Method)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>>       at
> >>>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>>       at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>>       at
> >>>>
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>>>       ... 25 common frames omitted
> >>>> 2016-02-02 21:56:30,130 [qtp1250890858-11591] ERROR
> >>>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> >>>> java.lang.IllegalStateException: failed to create a child event loop
> >>>>       at
> >>>>
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>>>       at
> >>>>
> >>
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> >>>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> >>>>       at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>>       at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>>       at
> >>>>
> >>
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> >>>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>>
> >>
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> >>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> >>>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> >>>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at org.eclipse.jetty.server.Server.handle(Server.java:462)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> >>>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at
> >>>>
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>>       at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> >>>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too
> many
> >>>> open files
> >>>>       at io.netty.channel.epoll.Native.epollCreate(Native Method)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>>       at
> >>>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>>       at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>>       at
> >>>>
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>>>       ... 25 common frames omitted
> >>>>
> >>>>
> >>>>
> >>>>> On Feb 2, 2016, at 7:40 AM, Venki Korukanti <
> venki.korukanti@gmail.com
> >>>
> >>>> wrote:
> >>>>>
> >>>>> Currently we keep the DrillClient per session. All the state is
in
> >> Server
> >>>>> and DrillClient is the reference to reuse the state. DrillClient
is
> >>>>> automatically closed when the session expires (default value is
1hr
> >> after
> >>>>> the last activity on session) or user explicitly logs out. I am
> trying
> >> to
> >>>>> understand if there is a resource leak. Do you have too many sessions
> >>>> open
> >>>>> when the system load is max or just few sessions but you have already
> >> ran
> >>>>> many queries using the existing sessions? If it is the former it
is
> >>>>> understandable to have per connection per session life. Also are
the
> >>>>> resources not freeing up after logout?
> >>>>>
> >>>>> If you need to have multiple simultaneous sessions, it is better
to
> >>>> connect
> >>>>> to different Drillbits (may be in a round-robin fashion) than always
> >>>>> connecting to a single Drillbit.
> >>>>>
> >>>>> Thanks
> >>>>> Venki
> >>>>>
> >>>>> On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser <josh@spoutable.com
> >>>> <mailto:josh@spoutable.com>> wrote:
> >>>>>
> >>>>>> First: Im a total newb at contributing to apache projects so
please
> >>>> excuse
> >>>>>> any indiscretions, feel free to give comments on style or whatever,
> i
> >>>> take
> >>>>>> feedback well.  Thick skin too.
> >>>>>>
> >>>>>>
> >>>>>> Ill give some background next and then a proposal.
> >>>>>>
> >>>>>> Background:
> >>>>>> I recently changed over to using authentication in the 1.5 snapshot
> >>>>>> because I need to have a session via the REST api so that I
can set
> >> the
> >>>>>> session storage options in an initial query for a subsequent
CTAS
> >> query.
> >>>>>> Previously all rest calls seemed to be completely independent.
> >>>>>>
> >>>>>> Since the change I have started seeing ‘too many files open’
errors
> in
> >>>> my
> >>>>>> drillbit.log and the drillbit java process becomes effectively
hung
> >>>> waiting
> >>>>>> for open file descriptor slots.  When running the top command
the
> >>>> machine
> >>>>>> is running at max load due to the drillbit process and the drillbit
> >>>> becomes
> >>>>>> effectively unresponsive, even the simple pages in the web console
> >> don’t
> >>>>>> respond.   Investigating further it seems that there might be
a file
> >>>> kept
> >>>>>> open per session by the drillbit process for the life of the
> session.
> >>>> I
> >>>>>> used the lsof unix command on the drillbit process and found
a lot
> of
> >>>> unix
> >>>>>> pipes.  Looking at the code it looks like these pipes could
be for
> the
> >>>>>> communication between the web process and the rpc server, with
one
> >> being
> >>>>>> allocated per session.  I haven’t validated this, its just
a guess
> >> after
> >>>>>> scanning the code.   I had 1.4 running without this requirement
and
> >>>> without
> >>>>>> ever seeing the error.  It seems without authentication the
number
> of
> >>>> open
> >>>>>> files is a non-issue for me, possibly due to sessions.
> >>>>>>
> >>>>>> I'm wondering if my guess about what is causing the ‘too many
open
> >>>> files’
> >>>>>> error is plausible?   Does anybody with a deeper understanding
of
> the
> >>>>>> architecture have any comments on this?
> >>>>>>
> >>>>>> Proposal:
> >>>>>> Assuming sessions are the issue, I am making some changes to
my rest
> >>>>>> client so that sessions are more effectively used and I can
up the
> >>>> ulimit
> >>>>>> for the drillbit process for the linux user in hopes of mitigating
> >>>> this.  I
> >>>>>> am effectively creating a rest client based session pool that
resets
> >>>>>> session variables to defaults  when the session gets reused.
> >> However,
> >>>> it
> >>>>>> seems hacky.
> >>>>>>
> >>>>>> Below is an idea for getting per request based settings which
seems
> >> less
> >>>>>> hacky in the long term.
> >>>>>>
> >>>>>> Can I add a new array member to the query.json REST method in
a
> >>>> backwards
> >>>>>> compatible way to set session level parameters in a single request?
> >>>>>> Currently a rest request via the api has a body like so:
> >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”}
> >>>>>>
> >>>>>> id like to do the following
> >>>>>>
> >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”,
“sessionSettings”:
> >>>>>> [“option_1_name”:”option_1_value”,
> “option_2_name”:”option_2_value”]}
> >>>>>>
> >>>>>> or even
> >>>>>>
> >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”,
“sessionSettings”:
> >>>> [“SET
> >>>>>> `option_name` = value”, “SET `option_name1` = value1”,“SET
> >>>> `option_name2` =
> >>>>>> value2”, “SET `option_name3` = value3”]}
> >>>>>>
> >>>>>> As far as I can tell drill is essentially stateless between
queries
> >>>> right
> >>>>>> now except for session level system parameters and authentication.
> >>>> There
> >>>>>> aren’t any in memory temp tables or cursors or variables like
PL/SQL
> >> or
> >>>>>> PSQL or other SQLs that would make it stateful.
> >>>>>>
> >>>>>> Given the stateless assumption, being able to set session level
> params
> >>>> on
> >>>>>> a per request basis would cover all of the cases that I might
need.
> >> It
> >>>>>> looks relatively straight forward to add something to QueryWrapper
> to
> >>>>>> accept an optional query session settings section of the json
packet
> >> and
> >>>>>> execute those ’SET' commands before the final query.    This
will
> work
> >>>> for
> >>>>>> me, as I can run without authentication in an ’secure' backend
> >>>> environment
> >>>>>> which will remove sessions and hence file descriptors, assuming
my
> >>>>>> assumptions about file descriptors and sessions are correct.
> >>>>>>
> >>>>>>
> >>>>>> My java is rusty (circa 2003) but some casual googling implies
that
> if
> >>>>>> this were added as a 3rd @FormParam to submitQuery in QueryResources
> >> it
> >>>>>> would be magically be null if it werent present and could easily
be
> >>>>>> ignored. If its present then an alternative constructor of
> >> QueryWrapper
> >>>>>> could be called with the extra param and it would be easy to
alter
> its
> >>>> run
> >>>>>> method to execute the SET commands.  There would need to be
some
> error
> >>>>>> handling of course if the SET commands were illegal or failed
to run
> >> for
> >>>>>> some reason.
> >>>>>>
> >>>>>> If this seems reasonable, how do I go about contributing?  I
looked
> >>>>>> through the links in the docs to apache foundation incubator
> projects
> >>>> but
> >>>>>> the links to drill were broken :(
> http://drill.apache.org/team.html
> >> <
> >>>>>> http://drill.apache.org/team.html <
> http://drill.apache.org/team.html
> >>>>
> >>>> I read this
> >>>>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/
> <
> >>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/>
<
> >>>>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/>
> >>>> and
> >>>>>> i have subscribed to the dev mailing list (obvious since you
are
> >> getting
> >>>>>> this).    It said to post here before creating a JIRA.  Am I
missing
> >>>>>> anything in my assumptions?  Comments?  Should I just submit
a JIRA
> >> and
> >>>> a
> >>>>>> patch or submit a JIRA and a comment or wait for comments before
> >> coding
> >>>>>> stuff up as an example?
> >>>>>>
> >>>>>> Thanks for taking the time to read and respond.
> >>>>>>
> >>>>>> Josh
> >>>>
> >>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message