commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Sicker <boa...@gmail.com>
Subject Re: Commons sub project for parallel method execution
Date Tue, 13 Jun 2017 01:21:20 GMT
Last time I used Spring, they had an @Async annotation you could use which
would automatically execute in an executor service (all handled via bean
proxies as usual).

On 12 June 2017 at 19:22, Gary Gregory <garydgregory@gmail.com> wrote:

> Hi All,
>
> I think it would be most helpful to note the distinction between the
> parallelism aspect and the bridge to domain classes aspect (currently done
> with reflection in the proposed github repo.)
>
> It seems (to me) that in between the ForkJoin framework already in Java (a
> low-level library) and up to Apache Spark (an lowel-level set of classes
> and high-level application-server-like code base), there are a ton of
> options already out there. I am not sure what yet another framework would
> do that is not already there.
>
> Maybe the distinguishing factor here is the use of reflection? What about
> annotations? That seems to be more modern approach (Java 5! :-) than the
> typing of method names in code (as currently done in the repo) which is a
> nightmare to maintain especially when you are in an evolving code base and
> refactoring all the time.
>
> Maybe an interesting angle would be decorating domain classes with
> annotations and submitting those to fork/join. Just thinkin' aloud...
>
> Gary
>
> On Mon, Jun 12, 2017 at 4:29 PM, Matt Sicker <boards@gmail.com> wrote:
>
> > I'd be interested to see where this leads to. It could end up as a sort
> of
> > Commons Parallel library. Besides providing an execution API, there could
> > be plenty of support utilities that tend to be found in all the
> > *Util(s)/*Helper classes in projects like all the ones I mentioned
> earlier
> > (basically all sorts of Hadoop-related projects and other distributed
> > systems here).
> >
> > Really, there's so many ways that such a project could head, I'd like to
> > hear more ideas on what to focus on.
> >
> > On 12 June 2017 at 18:19, Gary Gregory <garydgregory@gmail.com> wrote:
> >
> > > The upshot is that there has to be a way to do this with some custom
> code
> > > to at least have the ability to 'fast path' the code without
> reflection.
> > > Using lambdas should make this fairly syntactically unobtrusive.
> > >
> > > On Mon, Jun 12, 2017 at 4:02 PM, Arun Mohan <strider90arun@gmail.com>
> > > wrote:
> > >
> > > > Yes, reflection is not very performant but I don't think I have any
> > other
> > > > choice since the library has to inspect the object supplied by the
> > client
> > > > at runtime to pick out the methods to be invoked using
> > CompletableFuture.
> > > > But the performance penalty paid for using reflection will be more
> than
> > > > offset by the savings of parallel method execution, more so as the no
> > of
> > > > methods executed in parallel increases.
> > > >
> > > > On Mon, Jun 12, 2017 at 3:21 PM, Gary Gregory <
> garydgregory@gmail.com>
> > > > wrote:
> > > >
> > > > > On a lower-level, if you want to use this for lower-level services
> > > (where
> > > > > there is no network latency for example), you will need to avoid
> > using
> > > > > reflection to get the best performance.
> > > > >
> > > > > Gary
> > > > >
> > > > > On Mon, Jun 12, 2017 at 3:15 PM, Arun Mohan <
> strider90arun@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Gary,
> > > > > >
> > > > > > Thanks for your response. You have some valid and interesting
> > points
> > > > :-)
> > > > > > Of course you are right that Spark is much more mature. Thanks
> for
> > > your
> > > > > > insight.
> > > > > > It will be interesting indeed to find out if the core
> > parallelization
> > > > > > engine of Spark can be isolated like you suggest.
> > > > > >
> > > > > > I started working on this project because I felt that there
was
> no
> > > good
> > > > > > library for parallelizing method calls which can be plugged
in
> > easily
> > > > > into
> > > > > > an existing java project. Ultimately, if such a solution can
be
> > > > > > incorporated in the Apache Commons, it would be a useful addition
> > to
> > > > the
> > > > > > Commons repository.
> > > > > >
> > > > > > Thanks,
> > > > > > Arun
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 12, 2017 at 3:01 PM, Gary Gregory <
> > > garydgregory@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Arun,
> > > > > > >
> > > > > > > Sure, and that is to be expected, Spark is more mature
than a
> > four
> > > > > class
> > > > > > > prototype. What I am trying to get to is that in order
for the
> > > > library
> > > > > to
> > > > > > > be useful, you will end up with more in a first release,
and
> > after
> > > a
> > > > > > couple
> > > > > > > more releases, there will be more and more. Would Spark
not
> have
> > in
> > > > its
> > > > > > > guts the same kind of code your are proposing here? By
> extension,
> > > > will
> > > > > > you
> > > > > > > not end up with more framework-like (Spark-like) code and
> > solutions
> > > > as
> > > > > > > found in Spark? I am just playing devil's advocate here
;-)
> > > > > > >
> > > > > > >
> > > > > > > What would be interesting would be to find out if there
is a
> core
> > > > part
> > > > > of
> > > > > > > Spark that is separable and ex tractable into a Commons
> > component.
> > > > > Since
> > > > > > > Spark has a proven track record, it is more likely, that
such a
> > > > library
> > > > > > > would be generally useful than one created from scratch
that
> does
> > > not
> > > > > > > integrate with anything else. Again, please do not take
any of
> > this
> > > > > > > personally, I am just playing here :-)
> > > > > > >
> > > > > > > Gary
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jun 12, 2017 at 2:29 PM, Matt Sicker <boards@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > I already see a huge difference here: Spark requires
a bunch
> of
> > > > > > > > infrastructure to be set up, while this library is
just a
> > > library.
> > > > > > > Similar
> > > > > > > > to Kafka Streams versus Spark Streaming or Flink or
Storm or
> > > Samza
> > > > or
> > > > > > the
> > > > > > > > others.
> > > > > > > >
> > > > > > > > On 12 June 2017 at 16:28, Gary Gregory <
> garydgregory@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Mon, Jun 12, 2017 at 2:26 PM, Arun Mohan <
> > > > > strider90arun@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi All,
> > > > > > > > > >
> > > > > > > > > > Good afternoon.
> > > > > > > > > >
> > > > > > > > > > I have been working on a java generic parallel
execution
> > > > library
> > > > > > > which
> > > > > > > > > will
> > > > > > > > > > allow clients to execute methods in parallel
irrespective
> > of
> > > > the
> > > > > > > number
> > > > > > > > > of
> > > > > > > > > > method arguments, type of method arguments,
return type
> of
> > > the
> > > > > > method
> > > > > > > > > etc.
> > > > > > > > > >
> > > > > > > > > > Here is the link to the source code:
> > > > > > > > > > https://github.com/striderarun/parallel-execution-engine
> > > > > > > > > >
> > > > > > > > > > The project is in a nascent state and I
am the only
> > > contributor
> > > > > so
> > > > > > > > far. I
> > > > > > > > > > am new to the Apache community and I would
like to bring
> > this
> > > > > > project
> > > > > > > > > into
> > > > > > > > > > Apache and improve, expand and build a developer
> community
> > > > around
> > > > > > it.
> > > > > > > > > >
> > > > > > > > > > I think this project can be a sub project
of Apache
> Commons
> > > > since
> > > > > > it
> > > > > > > > > > provides generic components for parallelizing
any kind of
> > > > > methods.
> > > > > > > > > >
> > > > > > > > > > Can somebody please guide me or suggest
what other
> options
> > I
> > > > can
> > > > > > > > explore
> > > > > > > > > ?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Arun,
> > > > > > > > >
> > > > > > > > > Thank you for your proposal.
> > > > > > > > >
> > > > > > > > > How would this be different from Apache Spark?
> > > > > > > > >
> > > > > > > > > Thank you,
> > > > > > > > > Gary
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Arun
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Matt Sicker <boards@gmail.com>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Matt Sicker <boards@gmail.com>
> >
>



-- 
Matt Sicker <boards@gmail.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message