drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venki Korukanti <venki.koruka...@gmail.com>
Subject Re: setting session params in a rest request
Date Wed, 03 Feb 2016 06:23:22 GMT
I think you made valid points. It makes sense to have session less REST
calls in both auth enabled and disabled cases.

In case of auth enabled:
1) Session-less calls can be authenticated using Basic auth (this was
already asked on mailing list sometime back) as a start and move onto token
based auth later. These requests usually come from non-browsers. The only
issue is setting options before the query. For this we can implement your
suggestion of enhancing the query REST API to accept the options.
2) Session-based call using form auth for browser based access. If we
enhance the UI to enter options in the query form, we don't need any
session on server actually.

I will get a fix ASAP to remove the sessions in anonymous calls as they the
session are not reused in non-browser cases.

Thanks
Venki

On Tue, Feb 2, 2016 at 9:20 PM, Josh Schlesser <josh@spoutable.com> wrote:

> No, it wasn’t logging out, it was just stopping, obviously that caused
> dangling sessions for the authenticated scenario.
>
> I don’t think that a short timeout for anonymous sessions is a good way to
> go for anonymous api calls.  Session management isn’t what anybody would
> expect when using a REST api that is anonymous in a server to server
> context.   I would expect to use a token for authorization for a server to
> server REST api as well.  I’m not saying that is what it should be here,
> but that is what my general expectation is based on using other apis.   In
> the case of browser to server REST apis, I have run into authentication for
> a browser session and subsequent REST calls leaning on a browser cookie for
> persistent authentication.
>
> Removing sessions for anonymous calls seems like the right path and
> possibly easy and I think would be the expected behavior from most
> developers.  I would advocate for sessionless and token authenticated REST
> apis for when using authentication for the server to server case and cookie
> based with a session for the browser to server scenario, but its really the
> browser that has a session, not the api per se, its  just piggybacking on a
> regular authenticated web session for the REST api calls.
>
> This would actually leave me in a quandary for what I am trying to do
> which is set a session configuration option ’store.format', but I cant
> think of any reason that those types of settings shouldn’t just be set on a
> per request basis for a REST api.  In a server to server context for a rest
> api, keeping it sessionless means you could front a cluster of drillbits
> with a load balancer and not worry about dying nodes and sticky sessions
> etc...
>
> I have to get something up and running quickly right now so im versioning
> back to 1.4 and just spinning up a separate drillbit that will have the
> store.format system variable set to ‘json’ . it will be ok for me until a
> good long term solution arrives in drill.
>
> I’ll run the test on short session_max_idle_secs to 30 seconds on
> 1.5.0-SNAPSHOT to see if that gets rid of the file handle starvation
> problem, but keep in mind that means that users of the web console will
> have 30 seconds between pages or they have to authenticate again, which
> will probably be very annoying.  It doesnt seem like a good long term
> solution either.
>
> How do you think all of this should work?  I look forward to staying
> involved.
>
> Cheers,
> Josh
>
> > On Feb 2, 2016, at 4:40 PM, Venki Korukanti <venki.korukanti@gmail.com>
> wrote:
> >
> > When auth is *enabled*, is the worker process logging out after queries
> are
> > done? When auth is *disabled* can you set session_max_idle_secs in
> > drill.exec.http block in drill-override.conf to something like 30 (secs)
> > and try? This way anonymous sessions are closed quickly and not kept for
> > 1hr (default value). I think we may need to avoid creating sessions in
> > anonymous mode (when auth is disabled).
> >
> > Thanks
> > Venki
> >
> > On Tue, Feb 2, 2016 at 4:02 PM, Josh Schlesser <josh@spoutable.com>
> wrote:
> >
> >> I have a background worker process (on a server, not a browser) that
> kicks
> >> off every minute or so and issues some queries sequentially to the rest
> >> query endpoint.    In 1.4 with no authentication this worked fine except
> >> that in 1 instance I need to issue a CTAS query with a different format
> >> (json).
> >>
> >> I upgraded to 1.5-SNAPSHOT commit
> bb3fc15216d9cab804fc9a6f0e5bd34597dd4394
> >>
> >> Since the upgrade I am getting a resource starvation problem with or
> >> without authentication
> >> The drillbit process stays up for a an hour or less and then becomes
> >> unresponsive and eats up the cpu.
> >>
> >> It is definitely a resource starvation issue, not sure if its a resource
> >> leak.
> >> Below is a stack trace.
> >> Also when i lsof on the pid there are a lot (more than a thousand) of
> >> files like this listed which are used by NIO selectors.  so it smells
> like
> >> a resource leak.
> >>
> >> COMMAND  PID USER   FD   TYPE             DEVICE  SIZE/OFF    NODE NAME
> >> java    2931 root  288u  0000               0,11         0    7705
> >> anon_inode
> >>
> >> 2016-02-02 21:56:26,520 [qtp1250890858-11590] ERROR
> >> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> >> java.lang.IllegalStateException: failed to create a child event loop
> >>        at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> >> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>        at
> >>
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> >> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> >>        at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>        at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>        at
> >>
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> >> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >>
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >>
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> >> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> >> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> >> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at org.eclipse.jetty.server.Server.handle(Server.java:462)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> >> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> >> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> >> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> >> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many
> >> open files
> >>        at io.netty.channel.epoll.Native.epollCreate(Native Method)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>        at
> >> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>        at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>        at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> >> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>        ... 25 common frames omitted
> >> 2016-02-02 21:56:30,130 [qtp1250890858-11591] ERROR
> >> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> >> java.lang.IllegalStateException: failed to create a child event loop
> >>        at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> >> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>        at
> >>
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> >> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> >>        at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>        at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>        at
> >>
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> >> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >>
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >>
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> >> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>        at
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> >> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> >> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at org.eclipse.jetty.server.Server.handle(Server.java:462)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> >> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> >> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> >> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> >> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many
> >> open files
> >>        at io.netty.channel.epoll.Native.epollCreate(Native Method)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>        at
> >> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>        at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>        at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> >> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>        ... 25 common frames omitted
> >>
> >>
> >>
> >>> On Feb 2, 2016, at 7:40 AM, Venki Korukanti <venki.korukanti@gmail.com
> >
> >> wrote:
> >>>
> >>> Currently we keep the DrillClient per session. All the state is in
> Server
> >>> and DrillClient is the reference to reuse the state. DrillClient is
> >>> automatically closed when the session expires (default value is 1hr
> after
> >>> the last activity on session) or user explicitly logs out. I am trying
> to
> >>> understand if there is a resource leak. Do you have too many sessions
> >> open
> >>> when the system load is max or just few sessions but you have already
> ran
> >>> many queries using the existing sessions? If it is the former it is
> >>> understandable to have per connection per session life. Also are the
> >>> resources not freeing up after logout?
> >>>
> >>> If you need to have multiple simultaneous sessions, it is better to
> >> connect
> >>> to different Drillbits (may be in a round-robin fashion) than always
> >>> connecting to a single Drillbit.
> >>>
> >>> Thanks
> >>> Venki
> >>>
> >>> On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser <josh@spoutable.com
> >> <mailto:josh@spoutable.com>> wrote:
> >>>
> >>>> First: Im a total newb at contributing to apache projects so please
> >> excuse
> >>>> any indiscretions, feel free to give comments on style or whatever,
i
> >> take
> >>>> feedback well.  Thick skin too.
> >>>>
> >>>>
> >>>> Ill give some background next and then a proposal.
> >>>>
> >>>> Background:
> >>>> I recently changed over to using authentication in the 1.5 snapshot
> >>>> because I need to have a session via the REST api so that I can set
> the
> >>>> session storage options in an initial query for a subsequent CTAS
> query.
> >>>> Previously all rest calls seemed to be completely independent.
> >>>>
> >>>> Since the change I have started seeing ‘too many files open’ errors
in
> >> my
> >>>> drillbit.log and the drillbit java process becomes effectively hung
> >> waiting
> >>>> for open file descriptor slots.  When running the top command the
> >> machine
> >>>> is running at max load due to the drillbit process and the drillbit
> >> becomes
> >>>> effectively unresponsive, even the simple pages in the web console
> don’t
> >>>> respond.   Investigating further it seems that there might be a file
> >> kept
> >>>> open per session by the drillbit process for the life of the session.
> >> I
> >>>> used the lsof unix command on the drillbit process and found a lot of
> >> unix
> >>>> pipes.  Looking at the code it looks like these pipes could be for the
> >>>> communication between the web process and the rpc server, with one
> being
> >>>> allocated per session.  I haven’t validated this, its just a guess
> after
> >>>> scanning the code.   I had 1.4 running without this requirement and
> >> without
> >>>> ever seeing the error.  It seems without authentication the number of
> >> open
> >>>> files is a non-issue for me, possibly due to sessions.
> >>>>
> >>>> I'm wondering if my guess about what is causing the ‘too many open
> >> files’
> >>>> error is plausible?   Does anybody with a deeper understanding of the
> >>>> architecture have any comments on this?
> >>>>
> >>>> Proposal:
> >>>> Assuming sessions are the issue, I am making some changes to my rest
> >>>> client so that sessions are more effectively used and I can up the
> >> ulimit
> >>>> for the drillbit process for the linux user in hopes of mitigating
> >> this.  I
> >>>> am effectively creating a rest client based session pool that resets
> >>>> session variables to defaults  when the session gets reused.
>  However,
> >> it
> >>>> seems hacky.
> >>>>
> >>>> Below is an idea for getting per request based settings which seems
> less
> >>>> hacky in the long term.
> >>>>
> >>>> Can I add a new array member to the query.json REST method in a
> >> backwards
> >>>> compatible way to set session level parameters in a single request?
> >>>> Currently a rest request via the api has a body like so:
> >>>> { “queryType”: “SQL”, “query” : “<drill query>”}
> >>>>
> >>>> id like to do the following
> >>>>
> >>>> { “queryType”: “SQL”, “query” : “<drill query>”,
“sessionSettings”:
> >>>> [“option_1_name”:”option_1_value”, “option_2_name”:”option_2_value”]}
> >>>>
> >>>> or even
> >>>>
> >>>> { “queryType”: “SQL”, “query” : “<drill query>”,
“sessionSettings”:
> >> [“SET
> >>>> `option_name` = value”, “SET `option_name1` = value1”,“SET
> >> `option_name2` =
> >>>> value2”, “SET `option_name3` = value3”]}
> >>>>
> >>>> As far as I can tell drill is essentially stateless between queries
> >> right
> >>>> now except for session level system parameters and authentication.
> >> There
> >>>> aren’t any in memory temp tables or cursors or variables like PL/SQL
> or
> >>>> PSQL or other SQLs that would make it stateful.
> >>>>
> >>>> Given the stateless assumption, being able to set session level params
> >> on
> >>>> a per request basis would cover all of the cases that I might need.
> It
> >>>> looks relatively straight forward to add something to QueryWrapper to
> >>>> accept an optional query session settings section of the json packet
> and
> >>>> execute those ’SET' commands before the final query.    This will
work
> >> for
> >>>> me, as I can run without authentication in an ’secure' backend
> >> environment
> >>>> which will remove sessions and hence file descriptors, assuming my
> >>>> assumptions about file descriptors and sessions are correct.
> >>>>
> >>>>
> >>>> My java is rusty (circa 2003) but some casual googling implies that
if
> >>>> this were added as a 3rd @FormParam to submitQuery in QueryResources
> it
> >>>> would be magically be null if it werent present and could easily be
> >>>> ignored. If its present then an alternative constructor of
> QueryWrapper
> >>>> could be called with the extra param and it would be easy to alter its
> >> run
> >>>> method to execute the SET commands.  There would need to be some error
> >>>> handling of course if the SET commands were illegal or failed to run
> for
> >>>> some reason.
> >>>>
> >>>> If this seems reasonable, how do I go about contributing?  I looked
> >>>> through the links in the docs to apache foundation incubator projects
> >> but
> >>>> the links to drill were broken :(   http://drill.apache.org/team.html
> <
> >>>> http://drill.apache.org/team.html <http://drill.apache.org/team.html
> >>
> >> I read this
> >>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/ <
> >> http://drill.apache.org/docs/apache-drill-contribution-guidelines/> <
> >>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/>
> >> and
> >>>> i have subscribed to the dev mailing list (obvious since you are
> getting
> >>>> this).    It said to post here before creating a JIRA.  Am I missing
> >>>> anything in my assumptions?  Comments?  Should I just submit a JIRA
> and
> >> a
> >>>> patch or submit a JIRA and a comment or wait for comments before
> coding
> >>>> stuff up as an example?
> >>>>
> >>>> Thanks for taking the time to read and respond.
> >>>>
> >>>> Josh
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message