geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fred Krone <fkr...@pivotal.io>
Subject Re: [DISCUSS] easy string based partitioning
Date Mon, 05 Jun 2017 19:54:11 GMT
Cool.  This is what I was thinking too.

Fred

On Jun 5, 2017 12:50 PM, "Barry Oglesby" <boglesby@pivotal.io> wrote:

> Yes, that makes a lot of sense. Thanks for the clarification.
>
> Thanks,
> Barry Oglesby
>
>
> On Mon, Jun 5, 2017 at 12:06 PM, Darrel Schneider <dschneider@pivotal.io>
> wrote:
>
> > I pictured the top level parent region (your customer region) as not
> having
> > the StringPrefixPartitionResolver. Instead it would just use string keys
> > that have no prefix and use the default resolver.
> > It would be each co-located child region (your order region) that would
> > have the StringPrefixPartitionResolver and would format its keys to be
> > "parentKey:childKey". Does that make sense? Will it work?
> >
> >
> > On Mon, Jun 5, 2017 at 10:06 AM, Barry Oglesby <boglesby@pivotal.io>
> > wrote:
> >
> > > The top-level region may not have / need a delimiter. If you have a
> > > customer region and a colocated orders region, the customer region key
> > > could be the customerId, and the orders region key could be the
> > > customerId:orderId. The colocation would be on the customerId. In the
> > > customer region, it doesn't need a delimiter. Is it ok that this idea
> > would
> > > require one? Maybe the top-level region shouldn't require a delimiter?
> If
> > > the StringPrefixPartitionResolver is using key.split(":"), the customer
> > > region would return the entire key.
> > >
> > >
> > > Thanks,
> > > Barry Oglesby
> > >
> > >
> > > On Mon, Jun 5, 2017 at 8:46 AM, Jens Deppe <jdeppe@pivotal.io> wrote:
> > >
> > > > I like the idea of keeping the configuration 'conventional' and thus
> > not
> > > > requiring any server configuration. As soon as you need to define a
> > regex
> > > > on the server you might as well be writing PartitionResolver code.
> > > >
> > > > As an aside I also think that using regexes to parse keys would also
> > > > introduce a measurable performance hit.
> > > >
> > > > --Jens
> > > >
> > > > On Mon, Jun 5, 2017 at 8:21 AM, Udo Kohlmeyer <ukohlmeyer@pivotal.io
> >
> > > > wrote:
> > > >
> > > > > Whilst I like the idea to make something (and yes,
> > > > > DelimitedStringPartitionResolver is the simplest), I feel that we
> > > might
> > > > > be able to do one better.
> > > > >
> > > > > */Could/* one possibly have an /*SPEL*/ <
> > https://docs.spring.io/spring
> > > > > -framework/docs/current/spring-framework-reference/
> > > > html/expressions.html>-based
> > > > > PartitionResolver for this? At least this way, we don't have to
> rely
> > on
> > > > > delimiters or a strong knowledge of REGEX.
> > > > >
> > > > > IMO, this would provide the greatest bang for buck implementation.
> > > > >
> > > > > --Udo
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 6/2/17 19:15, Jacob Barrett wrote:
> > > > >
> > > > >> If you implement as regular expression the user doesn't have
to
> > > reformat
> > > > >> their key to a specific format (akin to making them implement
a
> > > class).
> > > > I
> > > > >> would concat the matching groups for generate the routing key.
> > > > >>
> > > > >> Consider RegEx: .*\bcorrelation=(\d+).*\
> bmaybe-something-else=(\w)
> > > > >> With Keys:
> > > > >> A: my,key;with:any-chars;unique=12345;correlation=678/and,maybe
> > > > >> -something-else=a
> > > > >> B: my,key;unique=876324;correlation=678;and,maybe-
> > > something-else=a,foo
> > > > >> C: somthing;different=988975;correlation=678;then,maybe-
> > > > something-else=ba
> > > > >>
> > > > >> Keys A and B would have routing key '678a'. Key C would have
> routing
> > > key
> > > > >> '678b'.
> > > > >>
> > > > >> -Jake
> > > > >>
> > > > >>
> > > > >>
> > > > >> Consider
> > > > >>
> > > > >> On Fri, Jun 2, 2017 at 4:02 PM Darrel Schneider <
> > > dschneider@pivotal.io>
> > > > >> wrote:
> > > > >>
> > > > >> Geode partitioned regions usually partition the data based on
the
> > > key's
> > > > >>> hashcode.
> > > > >>> You can do your own partitioning by implementing the
> > > PartitionResolver
> > > > >>> interface and configuring it on the partitioned region.
> > > > >>>
> > > > >>> In some use cases needing to deploy your class that implements
> > > > >>> PartitionResolver can be problematic so we would like to
find a
> way
> > > to
> > > > >>> offer partitioning based on a portion of the key (instead
of the
> > > > default
> > > > >>> which uses the entire key) that does not require you to implement
> > > your
> > > > >>> own
> > > > >>> PartitionResolver and does not require you to deploy your
own
> code
> > to
> > > > do
> > > > >>> the custom partitioning.
> > > > >>>
> > > > >>> Another group of users that do not want to implement
> > > PartitionResolver
> > > > >>> are
> > > > >>> non-java clients. So the solution is required to be usable
by
> > > non-java
> > > > >>> geode clients without needing to reimplement the client to
> support
> > a
> > > > new
> > > > >>> feature.
> > > > >>>
> > > > >>> Another constraint on the solution is for it to be both easy
to
> use
> > > and
> > > > >>> easy to implement.
> > > > >>>
> > > > >>> The proposed solution is to provide a class named:
> > > > >>>      org.apache.geode.cache.StringPrefixPartitionResolver
> > > > >>> This class will implement PartitionResolver and have a default
> > > > >>> constructor.
> > > > >>> To use it you need to configure a partitioned region's
> > > > PartitionResolver
> > > > >>> using the already existing mechanism for this (api, gfsh,
or
> xml).
> > > > >>> The StringPrefixPartitionResolver will require all keys on
its
> > region
> > > > to
> > > > >>> be
> > > > >>> of type String.
> > > > >>> It also requires that the string key contains at least one
':'
> > > > character.
> > > > >>> The substring of the key that precedes the first ':' is the
> prefix
> > > that
> > > > >>> will be returned from "getRoutingObject".
> > > > >>>
> > > > >>> An example of doing this in gfsh is:
> > > > >>>      create region --name=r1 --type=PARTITION
> > > > >>> --partition-resolver=org.apache.geode.cache.StringPrefixPart
> > > > >>> itionResolver
> > > > >>>
> > > > >>> Note that attempting to use a key that is not a String or
does
> not
> > > > >>> contain
> > > > >>> a ':' will throw an exception. This is to help developers
realize
> > > they
> > > > >>> made
> > > > >>> a mistake.
> > > > >>>
> > > > >>> Note that the delimiter is always a ':'. It would be easy
to made
> > the
> > > > >>> delimiter configurable when using apis or xml but currently
gfsh
> > does
> > > > not
> > > > >>> provide a way to pass parameters to the --partition-resolver
> create
> > > > >>> region
> > > > >>> option.
> > > > >>>
> > > > >>> The only public api change this proposal makes is the new
> > > > >>> StringPrefixPartitionResolver class.
> > > > >>>
> > > > >>>
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message