commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Gregory <garydgreg...@gmail.com>
Subject Re: Commons sub project for parallel method execution
Date Mon, 12 Jun 2017 22:21:47 GMT
On a lower-level, if you want to use this for lower-level services (where
there is no network latency for example), you will need to avoid using
reflection to get the best performance.

Gary

On Mon, Jun 12, 2017 at 3:15 PM, Arun Mohan <strider90arun@gmail.com> wrote:

> Hi Gary,
>
> Thanks for your response. You have some valid and interesting points :-)
> Of course you are right that Spark is much more mature. Thanks for your
> insight.
> It will be interesting indeed to find out if the core parallelization
> engine of Spark can be isolated like you suggest.
>
> I started working on this project because I felt that there was no good
> library for parallelizing method calls which can be plugged in easily into
> an existing java project. Ultimately, if such a solution can be
> incorporated in the Apache Commons, it would be a useful addition to the
> Commons repository.
>
> Thanks,
> Arun
>
>
>
> On Mon, Jun 12, 2017 at 3:01 PM, Gary Gregory <garydgregory@gmail.com>
> wrote:
>
> > Hi Arun,
> >
> > Sure, and that is to be expected, Spark is more mature than a four class
> > prototype. What I am trying to get to is that in order for the library to
> > be useful, you will end up with more in a first release, and after a
> couple
> > more releases, there will be more and more. Would Spark not have in its
> > guts the same kind of code your are proposing here? By extension, will
> you
> > not end up with more framework-like (Spark-like) code and solutions as
> > found in Spark? I am just playing devil's advocate here ;-)
> >
> >
> > What would be interesting would be to find out if there is a core part of
> > Spark that is separable and ex tractable into a Commons component. Since
> > Spark has a proven track record, it is more likely, that such a library
> > would be generally useful than one created from scratch that does not
> > integrate with anything else. Again, please do not take any of this
> > personally, I am just playing here :-)
> >
> > Gary
> >
> >
> > On Mon, Jun 12, 2017 at 2:29 PM, Matt Sicker <boards@gmail.com> wrote:
> >
> > > I already see a huge difference here: Spark requires a bunch of
> > > infrastructure to be set up, while this library is just a library.
> > Similar
> > > to Kafka Streams versus Spark Streaming or Flink or Storm or Samza or
> the
> > > others.
> > >
> > > On 12 June 2017 at 16:28, Gary Gregory <garydgregory@gmail.com> wrote:
> > >
> > > > On Mon, Jun 12, 2017 at 2:26 PM, Arun Mohan <strider90arun@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Good afternoon.
> > > > >
> > > > > I have been working on a java generic parallel execution library
> > which
> > > > will
> > > > > allow clients to execute methods in parallel irrespective of the
> > number
> > > > of
> > > > > method arguments, type of method arguments, return type of the
> method
> > > > etc.
> > > > >
> > > > > Here is the link to the source code:
> > > > > https://github.com/striderarun/parallel-execution-engine
> > > > >
> > > > > The project is in a nascent state and I am the only contributor so
> > > far. I
> > > > > am new to the Apache community and I would like to bring this
> project
> > > > into
> > > > > Apache and improve, expand and build a developer community around
> it.
> > > > >
> > > > > I think this project can be a sub project of Apache Commons since
> it
> > > > > provides generic components for parallelizing any kind of methods.
> > > > >
> > > > > Can somebody please guide me or suggest what other options I can
> > > explore
> > > > ?
> > > > >
> > > >
> > > > Hi Arun,
> > > >
> > > > Thank you for your proposal.
> > > >
> > > > How would this be different from Apache Spark?
> > > >
> > > > Thank you,
> > > > Gary
> > > >
> > > >
> > > > >
> > > > > Thanks,
> > > > > Arun
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Matt Sicker <boards@gmail.com>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message