geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Blum <jb...@pivotal.io>
Subject Re: Simple Java Client
Date Wed, 26 Apr 2017 03:39:02 GMT
For clarification, the Management REST API
<https://github.com/apache/geode/tree/rel/v1.1.1/geode-core/src/main/java/org/apache/geode/management/internal/web>
[1]
is not the Developer REST API
<http://geode.apache.org/docs/guide/11/rest_apps/book_intro.html> [2].
These are 2 separate/different things.

The later is used by developers to perform normal Geode public API style
operations (e.g. Region ops, Queries, Function execution) while the later
is used by *Gfsh* to tunnel commands over HTTP rather than the default, JMX
RMI.  The purpose was to be able to interact with a cluster in AWS, or
other cloud environments, from localhost.

-j

[1]
https://github.com/apache/geode/tree/rel/v1.1.1/geode-core/src/main/java/org/apache/geode/management/internal/web
[2] http://geode.apache.org/docs/guide/11/rest_apps/book_intro.html


On Tue, Apr 25, 2017 at 8:25 PM, Wes Williams <wwilliams@pivotal.io> wrote:

> A couple of points to interact with John's points.
>
> GFSH as API
>
> We agree that GFSH is a DSL, and a really good and useful one. We agree
> that we don't want our API tied to GFSH syntax. I view the valuable part of
> GemFire admin as the Java code underneath GFSH, or the "Commands."
>
> For example, to create a JAVA API to "Create Region",  why not create a
> Java API around CreateAlterDestroyRegionCommands? In this way, GFSH and
> the
> JAVA API share the same code. It seems too obvious yet I don't see it being
> recommended.  Why not?  (Note: the hard-coded formatting would need to be
> removed).
>
> Once you have the Java/GFSH/REST API as common code, you then refactor it.
> What's the objection to this approach? Once you open Java API's to do
> everything that GFSH does, then you have now unshackled the power of the
> developer (the JAVA API) and the admin (GFSH API).
>
>
> REST API
>
> I've found that most don't want to use the Dev REST API because it's
> attached to a server rather than the cluster. HA?
>
>
> *Wes Williams | Pivotal Advisory **Data Engineer*
> 781.606.0325
> http://pivotal.io/big-data/pivotal-gemfire
>
> On Tue, Apr 25, 2017 at 7:01 PM, Fred Krone <fkrone@pivotal.io> wrote:
>
> > Good feedback.
> >
> > This would use the new protocol.  I should have mentioned that.
> >
> > The original driver for this was the Region API needs either an update or
> > an alternative.  Updating has a few drawbacks: Region wasn't designed
> with
> > open source in-mind and as Swap mentioned it is naturally tightly
> coupled.
> > Members of the community are already working to update Region but
> something
> > gets fixed and it breaks something for someone else.  I think it's much
> > better to provide a new interface that implements the first part of JSR
> 107
> > (javax.cache) and get the ball rolling for the community and, perhaps,
> over
> > time deprecate Region (although that's not a primary objective).
> >
> > A Java driver will probably get built regardless just to give the new
> > protocol some legs. That driver also needs a decent caching interface.
> JSR
> > 107 has already solved that part.  So let's get started on it.  If the
> > community wants to go the whole way and continue JSR 107 implementation
> > after that that's awesome.  Functions can also be added, etc.
> >
> > I intentionally did not mention anything about PCF as this pertains to
> > Geode itself as an open source offering and developer experience.  I'm
> > writing as a member of the community. Ie: I'm a developer who would like
> to
> > add some caching to my application -- I can download either Geode or
> > Hazelcast for free and right now it's a no brainer.  Not that we wouldn't
> > keep PCF in-mind but it's out of scope for this thread.  I do believe
> > getting started on a Java driver for the protocol and a standardized
> > caching API are easily leveraged wins across the board.
> >
> >
> >
> >
> > On Tue, Apr 25, 2017 at 3:20 PM, Swapnil Bawaskar <sbawaskar@pivotal.io>
> > wrote:
> >
> > > I had looked at the JCache in the past and here are some of the things
> I
> > > noted:
> > >
> > > Naming convention: Geode's region is a Cache in JSR-107, and Geode's
> > Cache
> > > is a CacheManager. I think it would be too confusing to change the
> > meaning
> > > of cache. Also, how do you document this given that Cache means
> different
> > > things if you are talking JCache vs Geode.
> > >
> > > The way to create a Cache in JSR-107 is:
> > > Cache<Integer, Date> cache = manager.createCache(cacheName,
> Configuration
> > > c);
> > > Where it is upto the implementation to extend Configuration. Given
> this,
> > > users will not be able to switch from an existing implementation to
> ours;
> > > will have to write new code specially Configuration, making callbacks
> > > serializable etc.
> > >
> > > JSR-107 will not be limited to the client. Server side callbacks like
> > > CacheLoader, CacheListener etc. will need handle on jsr-107 “cache”.
> > >
> > > JSR-107 supports features like an EntryProcessor, which is a function
> > > invoked atomically on an entry operation. We will have to make invasive
> > > changes to Geode to support this.
> > >
> > > Given these, I don't think supporting JSR-107 is trivial.
> > >
> > > On Tue, Apr 25, 2017 at 2:55 PM Dan Smith <dsmith@pivotal.io> wrote:
> > >
> > > > What transport are you planning on using? REST, or the current binary
> > > > protocol? Or is this just a wrapper around the existing java client
> > APIs?
> > > >
> > > > If this about creating a new API, I agree with what John is saying
> that
> > > we
> > > > need reduce the number of APIs we have to do the same thing in java.
> It
> > > > seems especially confusing if we end up with different APIs that
> > support
> > > > distinct features - like being able to create a region on the server
> > with
> > > > this new API but only being able to invoke a function with the old
> API.
> > > >
> > > > The other thing I think we need to avoid is being able to do things
> > from
> > > > the client (or gfsh, or xml, ...) that don't have a java API on the
> > > server.
> > > > I'm assuming your getOrCreateRegion region is supposed behave like
> the
> > > gfsh
> > > > command and update the cluster configuration? +++1 to Wes's
> suggestion
> > > have
> > > > a public API for executing these gfsh operations.
> > > >
> > > > I really think we need to work on having *one* authoritative public
> API
> > > > that contains everything that we support on the server. The protocols
> > we
> > > > support (REST, binary, ...) and the client drivers that use those
> > > protocols
> > > > should just be ways of accessing that API. The Java API on the server
> > > right
> > > > now the closest thing we have to this, but there are things you can
> do
> > > with
> > > > gfsh and things you can do with the current client that have no java
> > API
> > > > equivalent. I think we really need to fix that!
> > > >
> > > > Also, preferably we won't have to hand code a bunch of stuff in every
> > > > client driver and protocol whenever we add a new feature. For example
> > if
> > > > were to add a geode function to invoke each new API we add, that new
> > API
> > > > would be accessible from REST, gfsh, the java client etc. Later we
> > could
> > > > add syntatic sugar to make the new API prettier, but if we had a low
> > > level
> > > > API that automatically exposed all new features that would make
> adding
> > > new
> > > > features much less painful.
> > > >
> > > > I do like the idea of adding a reactive API.
> > > >
> > > >  Supporting JSR-107 might be nice - but that could probably be
> written
> > > just
> > > > as a wrapper around the current API without too much work. I don't
> > think
> > > we
> > > > should do anything for JSR-107 other than provide a JSR-107
> compatible
> > > > wrapper - if someone wants additional geode specific features they
> > should
> > > > drop into the existing API.
> > > >
> > > > I also like the idea of a smaller client jar and dependencies, but I
> > > think
> > > > there is a huge amount of room for improvement in our existing client
> > > just
> > > > by refactoring the code a bit more and shinking geode-core into a
> bare
> > > > minimum.
> > > >
> > > > -Dan
> > > >
> > > > On Mon, Apr 24, 2017 at 8:20 PM, Fred Krone <fkrone@pivotal.io>
> wrote:
> > > >
> > > > > That's great, Wes.  I will take a look.  Thanks.
> > > > >
> > > > > John -- good feedback to consider and I was hoping this would come
> > up.
> > > > >
> > > > > "In my mind, there really are only 2 approaches... *Spring* and
> > > > > non-*Spring*,
> > > > > or rather PCF and non-PCF, and since PCF is primarily based on Boot
> > > > (given
> > > > > Microservices/applications are the new concurrency), then I am
> really
> > > > > saying the same thing."
> > > > >
> > > > > This would be cover non spring and attempt to give the community
> > > > something
> > > > > that had the same standardized experience as JSR 107 -- starting
> with
> > > the
> > > > > Cache interface itself.  Although we don't necessarily have to
> > > implement
> > > > > Cache, it's method signatures are essentially a (non spring)
> caching
> > > > > standard for Java developers at this point.   We considered full
> > blown
> > > > JSR
> > > > > 107 implementation but thought it was too robust for the needs
> > > mentioned
> > > > > (that's not to say we couldn't get there incrementally).  I think
> > those
> > > > > needs still exist open-source outside of PCF and don't cannibalize.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Apr 24, 2017 at 7:44 PM, Real Wes <TheRealWes@outlook.com>
> > > > wrote:
> > > > >
> > > > > >
> > > > > > Here is a simple Java client _for enterprise use_ that I
> developed
> > > for
> > > > > > Geode and distributed to several enterprises. It has similarities
> > and
> > > > > > differences for your goal. This project creates both server
and
> > > client
> > > > > > regions dynamically.  It lists regions, alters regions… really
> > > anything
> > > > > > that GFSH can do. It differs in that it calls GFSH to do its
work
> > > > rather
> > > > > > than a Java interface. You can think of this as a Java API on
top
> > of
> > > > > GFSH.
> > > > > >
> > > > > > You can use this model and keep the similarities while adjusting
> > for
> > > > the
> > > > > > Java interface.
> > > > > >
> > > > > > Similarities
> > > > > > - Client uses SDG and complies with JSR-107
> > > > > >      [the Cache Manager is here: https://github.com/Pivotal-
> > > > > > Data-Engineering/ClusterManagement/tree/master/
> > > > > > cluster-management-client/src/main/java/com/geode/cache/manager
> ]
> > > > > > - After the server creates its region, client creates its region
> > > > > >      [ see createRegions at: https://github.com/Pivotal-
> > > > > Data-Engineering/
> > > > > > ClusterManagement/blob/master/cluster-management-
> > > > > >
> > > > client/src/main/java/com/geode/cache/manager/
> > RegionCreator.java<https://
> > > > > > github.com/Pivotal-Data-Engineering/ClusterManagement/
> > > > > blob/master/cluster-
> > > > > > management-client/src/main/java/com/geode/cache/manager/
> > > > > RegionCreator.java>
> > > > > > ]
> > > > > >
> > > > > > Differences
> > > > > > Server dynamically calls GFSH to execute commands
> > > > > >
> > > > > > How it works
> > > > > > - Whenever the client calls a region that does not exist, the
> > client
> > > > > looks
> > > > > > for a <region name>.properties file. If the properties
file
> exists,
> > > the
> > > > > > region is created with the dynamic properties.
> > > > > > - If <region name>.properties does not exist, default
region
> > > properties
> > > > > > are loaded and used to create the region.
> > > > > > - properties are formed into a GFSH string and passed to a server
> > > > > function
> > > > > > - The server function calls GFSH using internal calls. It calls
> the
> > > > > > GfshParser, executes the command, and then returns the parsed
> > > results.
> > > > > > [see https://github.com/Pivotal-Data-Engineering/
> > > > > > ClusterManagement/blob/master/cluster-management-server/src/
> > > > > > main/java/com/geode/gfsh/GfshCommand.java]
> > > > > >
> > > > > > Limitations
> > > > > > It uses internal calls to call GFSH. I have made requests to
make
> > > this
> > > > a
> > > > > > stable public interface. Andrew Baker had a good idea to return
> > gfsh
> > > > > > results in JSON format with a new —results=json property to
pass
> to
> > > > GFSH.
> > > > > >
> > > > > > Strengths
> > > > > > That is uses GFSH can be viewed as a strength to keep a
> consistent
> > > API,
> > > > > > whether Java or GFSH
> > > > > >
> > > > > > TESTS
> > > > > > You can start by running server tests at:
> > > https://github.com/Pivotal-
> > > > > > Data-Engineering/ClusterManagement/tree/master/
> > > > > > cluster-management-server/src/test/java/com/geode/gfsh
> > > > > >
> > > > > > Client tests are at: https://github.com/Pivotal-
> Data-Engineering/
> > > > > > ClusterManagement/tree/master/cluster-management-client/src/
> > > > > > test/java/com/geode/management/client
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Wes Williams
> > > > > >
> > > > > > On Apr 24, 2017, at 6:51 PM, Kirk Lund <klund@apache.org<mailto:
> > > klund
> > > > > > @apache.org>> wrote:
> > > > > >
> > > > > > A couple things I'd like to see:
> > > > > >
> > > > > > 1) completely new API that doesn't reuse the old API classes
(or
> at
> > > > least
> > > > > > not the giant classes such as Cache and Region interfaces)
> > > > > > 2) separation of API and Impl so that users can compile their
> code
> > > > > against
> > > > > > a dedicated client API jar
> > > > > >
> > > > > > On Mon, Apr 24, 2017 at 3:03 PM, Fred Krone <fkrone@pivotal.io
> > > <mailto:
> > > > > fkro
> > > > > > ne@pivotal.io>> wrote:
> > > > > >
> > > > > > In an effort to improve Java developer ease of use for caching
> with
> > > > > Geode I
> > > > > > am looking for feedback on going forward with creating a Java
> > client.
> > > > > This
> > > > > > client will allow for server-side region creation and distributed
> > > data
> > > > > > caching.  This would allow for a thin client that fits with
> > > > microservice
> > > > > > caching patterns and also abstracts a cleaner client-server
> > > experience
> > > > > > driven interface.
> > > > > >
> > > > > > Initially we were going to update the Region interface but were
> > > > concerned
> > > > > > with breaking existing applications.  We also would like to
> provide
> > > > > Region
> > > > > > creation to a client application and so propose here solving
both
> > of
> > > > > these
> > > > > > areas with a Java client.
> > > > > >
> > > > > > It would have new project repo for the client.
> > > > > >
> > > > > > It would provide new Region interface for clients.  The specifics
> > of
> > > > the
> > > > > > API design are too lengthy for this conversation but
> implementation
> > > > will
> > > > > > resemble JSR 107 Cache interface.
> > > > > >
> > > > > > It would use the new security framework.
> > > > > >
> > > > > >
> > > > > > *An example*,
> > > > > >
> > > > > > The client application simply creates an instance of client
and
> > > points
> > > > it
> > > > > > to the locator:
> > > > > >
> > > > > >
> > > > > > org.apache.geode.client.Client client =
> Client.create(locatorHost,
> > > > > > locatorPort);
> > > > > >
> > > > > >
> > > > > > Client has the following methods:
> > > > > >
> > > > > > package org.apache.geode.client;
> > > > > >
> > > > > >
> > > > > > public interface GeodeClient {
> > > > > >
> > > > > >  /**
> > > > > >
> > > > > >   * creates the region on the servers, or gets the region if
it
> > > exits,
> > > > > > returns a PROXY region
> > > > > >
> > > > > >   */
> > > > > >
> > > > > >  public Region getOrCreateRegion(RegionAttributes attributes,
> > String
> > > > > > name);
> > > > > >
> > > > > >
> > > > > >  /**
> > > > > >
> > > > > >   * Returns a PROXY region if the region exists on the server
> > > > > >
> > > > > >   */
> > > > > >
> > > > > >  public Region getRegion(String name);
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > MVP
> > > > > >
> > > > > > The smallest set of features to test this idea, learn and
> iterate,
> > > and
> > > > > get
> > > > > > the client into the communities hands for future iterations
is:
> > > > > >
> > > > > >
> > > > > > Create a server side Region from a client
> > > > > >
> > > > > > Region interface has CRUD on par with javax.cache.Cache (from
JSR
> > > 107)
> > > > > >
> > > > > > Calls are asynchronous -- futures
> > > > > >
> > > > > >
> > > > > >
> > > > > > Also would like feedback on which future functionality would
be
> > most
> > > > > useful
> > > > > > from a thin client:
> > > > > >
> > > > > > Function execution
> > > > > >
> > > > > > Durable clients
> > > > > >
> > > > > > User defined serialization
> > > > > >
> > > > > > Register interest
> > > > > >
> > > > > > Queries
> > > > > >
> > > > > > CQ
> > > > > >
> > > > > > Near side caching
> > > > > >
> > > > > > Create disk stores from client
> > > > > >
> > > > > > Region group membership
> > > > > >
> > > > > > Client subscription load balancing
> > > > > > Transactions
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > -Fred
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
-John
john.blum10101 (skype)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message