accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: Accumulo ProxyInstance available
Date Thu, 03 Sep 2015 17:05:16 GMT
On Thu, Sep 3, 2015 at 12:43 PM, Patrone, Dennis S. <
Dennis.Patrone@jhuapl.edu> wrote:

> I think in almost all cases you’d just get a RuntimeException that wraps
> the TTransportException.  We could certainly do a better job at (automatic)
> fault recovery than we are currently… especially in the “simple” cases
> where, for example, the proxy goes down and come back while you are not
> using the Instance code (a simple “reconnect” should suffice).    In other
> cases, we might be able to provide a more API-friendly exception (e.g.,
> throw a MutationRejectedException if a push of Mutations fails due to a
> network problem).   But our use case was for convenience, not production—so
> we didn’t worry too much about failures other than to make sure we knew
> they happened.
>

Thanks for the explanation.  I don't have an opinion on what it should do
in this case, I was just curious what the behavior was.  I agree that
behavior makes sense for your use case.


>
>
> The proxy currently maintains the TTransport reference for the life of the
> Instance (rather than reconnecting on every request).  One of the (other)
> issues doing this is we need an explicit “close()” on the Instance (not in
> the Instance API) when you are done with it.  We toyed with the idea of
> creating/closing/destroying the transport on every access which would
> remove the need for an explicit close as well as handle many of the
> intermittent connectivity issues you raise.  But that would be at the cost
> of runtime performance in what is (hopefully) the average case.   Perhaps
> that performance hit would be OK (hey, if you really need it to work fast
> just the standard library!) but we tried to make the runtime at least
> tolerable! :-) Maybe someone has a brilliant idea of how we could handle it
> better…
>

Could close the TTransport after X time of inactivity and make this
synchronized w.r.t to getting a TTransport.   A Guava cache might be an
easy way to accomplish this.


>
>
> Thanks,
>
> Dennis
>
>
>
>
>
> *From:* Keith Turner [mailto:keith@deenlo.com]
> *Sent:* Thursday, September 03, 2015 11:18 AM
> *To:* user@accumulo.apache.org
> *Subject:* Re: Accumulo ProxyInstance available
>
>
>
> Thats interesting.  Curious what you do in the case of faults?  I.e. if
> proxy does down for a bit and then comes back up, or if network between
> an Accumulo ProxyInstance client and the client goes down for a bit.
>
>
>
> On Thu, Sep 3, 2015 at 7:33 AM, Patrone, Dennis S. <
> Dennis.Patrone@jhuapl.edu> wrote:
>
> I just wanted to let folks know about the availability of a new project
> called 'Accumulo ProxyInstance'. It is a Java Instance implementation that
> communicates with an Accumulo cluster via the Thrift proxy server.
>
>
>
> Basically, we have an isolated Accumulo cluster only accessible to the
> rest of the network through a single, dual-homed gatekeeper machine.  We
> started the Thrift proxy server on the gatekeeper.  We then created this
> Instance implementation to allow developers to access the cluster from
> their development machines through the gatekeeper/proxy but still using the
> traditional Java Instance APIs.    This encapsulates a majority of the
> client code from needing to know if it is running on the cluster or
> connecting through the proxy server (with the notable exception of the
> actual Instance instantiation).
>
>
>
> Now for convenience we can test, debug, and perform smaller queries using
> the ProxyInstance on our development machines through the proxy server.
> When performance demands it, we can simply change 1 line of code to
> instantiate a ZooKeeperInstance instead, move the code to the isolated
> subnet, and communicate directly with the tablet servers using the
> traditional Java client library.
>
>
>
> You can read more about the project here:
> http://jhuapl.github.io/accumulo-proxy-instance/proxy_instance_user_manual.html
> .  The code is available on github (
> https://github.com/JHUAPL/accumulo-proxy-instance) and the JAR is on
> maven central (
> http://search.maven.org/#artifactdetails|edu.jhuapl.accumulo|proxy-instance|1.0.0|jar).
> It was developed and tested against Accumulo 1.6.2.
>
>
>
> We’ve found it useful in our development environment given our isolated
> Accumulo cluster; perhaps others who have a similar setup might also find
> utility in it as well.
>
>
>
> Thanks,
>
> Dennis
>
>
>
>
>
>
>

Mime
View raw message