Mailing-List: contact dev-help@commons.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Commons Developers List" <dev@commons.apache.org>
MIME-Version: 1.0
In-Reply-To: <CACZkXPzmhNp_9UKV1cSCwAeNd9gY0r3sgo7j5v8rT4i2Tx0asA@mail.gmail.com>
References: <CAHYLG1jS5=ZNbTgy3YBgsAOrPfCMyBnE+46=v2-YYoB4_8NVtg@mail.gmail.com>
 <CACZkXPyMCJok2MKjaKAYpm4gdKc86mtx+Ktpa-eGmV2G6_vepw@mail.gmail.com>
 <CACmp6kpSaUpBfzRFpizGWNi2wU97PuCUJWwOcuuiGwBRmiAk8A@mail.gmail.com>
 <CACZkXPzKfof8y9mJzL7TTbnJ=6GVeLDwbPaCnb0MKBTAZ9PJog@mail.gmail.com>
 <CAHYLG1gLY8DvLXnNwvJLuF5nnEgqyhvVCzEKn_bwE3D9NQOaTA@mail.gmail.com>
 <CACZkXPxeDrgYyvyzBiKFXZoQphxdY6y-sLqA+sgm5xNhEcdicg@mail.gmail.com>
 <CAHYLG1hbAEL1uuQV7C3wr=0ikHvUFa1qxuOi856nkAWTosTsbw@mail.gmail.com>
 <CACZkXPx-gwiS+bR8F43gds9cek0wx7UydA_QvpLPEzQc1RRRyw@mail.gmail.com>
 <CACmp6kq+scvjSpZq_DZiTb2kRxWMoyFLWNOaDqwuzFxL0unPBQ@mail.gmail.com>
 <CADRx3PMNj7jw-tG_4gz-eRKJ16QwHzqP61mLqn7UTYrWXTkbaw@mail.gmail.com>
 <CACmp6kr76vjKwdjnGEKFGWTmmNqhW7E4FPcqT13xS4_j3=CzHw@mail.gmail.com> <CACZkXPzmhNp_9UKV1cSCwAeNd9gY0r3sgo7j5v8rT4i2Tx0asA@mail.gmail.com>
From: Arun Mohan <strider90arun@gmail.com>
Date: Mon, 12 Jun 2017 19:23:40 -0700
Message-ID: <CAHYLG1jtgjzMqgVrbpG65wBd4hkSLcK0Ub+v_vcF9rEiq-xcHA@mail.gmail.com>
Subject: Re: Commons sub project for parallel method execution
To: Commons Developers List <dev@commons.apache.org>
Content-Type: multipart/alternative; boundary="94eb2c03643ec305790551ce1f4a"
archived-at: Tue, 13 Jun 2017 02:23:48 -0000

--94eb2c03643ec305790551ce1f4a
Content-Type: text/plain; charset="UTF-8"

A lot of useful and interesting suggestions here.
Usage of annotations instead of hardcoding method names is definitely a
very good idea. Decorating the domain classes with annotations will be much
more clean and maintainable.
And I think we can check out the usage of MethodHandle's interoperability
with Reflection API in java 8 for better performance improvements.
And being pluggable into other frameworks is definitely something to be
kept in mind.

A lot of things to think about.
I will check out the GPars library, haven't used it before :-)


On Mon, Jun 12, 2017 at 7:08 PM, Gary Gregory <garydgregory@gmail.com>
wrote:

> On Mon, Jun 12, 2017 at 6:56 PM, Matt Sicker <boards@gmail.com> wrote:
>
> > So wouldn't something like ASM or Javassist or one of the zillion other
> > bytecode libraries be a better alternative to using reflection for
> > performance? Also, using the Java 7 reflections API improvements helps
> > speed things up quite a bit.
> >
>
> IMO, unless you are doing scripting, reflection should be a used as a
> workaround, but that's just me. For example, like we do in Commons IO's
> Java7Support class.
>
> But I digress ;-)
>
> This is clearly an interesting topic. My concern is that there is a LOT of
> code out there that does stuff like this at the low and high level from the
> JRE's fork/join to Apache Spark and so on as I've stated.
>
> IMO something new would have to be both unique and since this is Commons,
> potentially pluggable into other frameworks.
>
> Gary
>
>
> > On 12 June 2017 at 20:37, Paul King <paul.king.asert@gmail.com> wrote:
> >
> > > My goto library for such tasks would be GPars. It has both Java and
> > > Groovy support for most things (actors/dataflow) but less so for
> > > asynchronous task execution. It's one of the things that would be good
> > > to explore in light of Java 8. Groovy is now Apache, GPars not at this
> > > stage.
> > >
> > > So with adding two jars (GPars + Groovy), you can use Groovy like this:
> > >
> > > @Grab('org.codehaus.gpars:gpars:1.2.1')
> > > import com.arun.student.StudentService
> > > import groovyx.gpars.GParsExecutorsPool
> > >
> > > long startTime = System.nanoTime()
> > > def service = new StudentService()
> > > def bookSeries = ["A Song of Ice and Fire": 7, "Wheel of Time": 14,
> > > "Harry Potter": 7]
> > >
> > > def tasks = [
> > >         { println service.findStudent("john@gmail.com", 11, false) },
> > >         { println service.getStudentMarks(1L) },
> > >         { println service.getStudentsByFirstNames(["John","Alice"]) },
> > >         { println service.getRandomLastName() },
> > >         { println service.findStudentIdByName("Kate", "Williams") },
> > >         { service.printMapValues(bookSeries) }
> > > ]
> > >
> > > GParsExecutorsPool.withPool {
> > >     tasks.collect{ it.callAsync() }.collect{ it.get() }
> > > //    tasks.eachParallel{ it() } // one of numerous alternatives
> > > }
> > >
> > > long executionTime = (System.nanoTime() - startTime) / 1000000
> > > println "\nTotal elapsed time is $executionTime\n\n"
> > >
> > >
> > > Cheers, Paul.
> > >
> > >
> > > On Tue, Jun 13, 2017 at 9:29 AM, Matt Sicker <boards@gmail.com> wrote:
> > > > I'd be interested to see where this leads to. It could end up as a
> sort
> > > of
> > > > Commons Parallel library. Besides providing an execution API, there
> > could
> > > > be plenty of support utilities that tend to be found in all the
> > > > *Util(s)/*Helper classes in projects like all the ones I mentioned
> > > earlier
> > > > (basically all sorts of Hadoop-related projects and other distributed
> > > > systems here).
> > > >
> > > > Really, there's so many ways that such a project could head, I'd like
> > to
> > > > hear more ideas on what to focus on.
> > > >
> > > > On 12 June 2017 at 18:19, Gary Gregory <garydgregory@gmail.com>
> wrote:
> > > >
> > > >> The upshot is that there has to be a way to do this with some custom
> > > code
> > > >> to at least have the ability to 'fast path' the code without
> > reflection.
> > > >> Using lambdas should make this fairly syntactically unobtrusive.
> > > >>
> > > >> On Mon, Jun 12, 2017 at 4:02 PM, Arun Mohan <
> strider90arun@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Yes, reflection is not very performant but I don't think I have
> any
> > > other
> > > >> > choice since the library has to inspect the object supplied by the
> > > client
> > > >> > at runtime to pick out the methods to be invoked using
> > > CompletableFuture.
> > > >> > But the performance penalty paid for using reflection will be more
> > > than
> > > >> > offset by the savings of parallel method execution, more so as the
> > no
> > > of
> > > >> > methods executed in parallel increases.
> > > >> >
> > > >> > On Mon, Jun 12, 2017 at 3:21 PM, Gary Gregory <
> > garydgregory@gmail.com
> > > >
> > > >> > wrote:
> > > >> >
> > > >> > > On a lower-level, if you want to use this for lower-level
> services
> > > >> (where
> > > >> > > there is no network latency for example), you will need to avoid
> > > using
> > > >> > > reflection to get the best performance.
> > > >> > >
> > > >> > > Gary
> > > >> > >
> > > >> > > On Mon, Jun 12, 2017 at 3:15 PM, Arun Mohan <
> > > strider90arun@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi Gary,
> > > >> > > >
> > > >> > > > Thanks for your response. You have some valid and interesting
> > > points
> > > >> > :-)
> > > >> > > > Of course you are right that Spark is much more mature. Thanks
> > for
> > > >> your
> > > >> > > > insight.
> > > >> > > > It will be interesting indeed to find out if the core
> > > parallelization
> > > >> > > > engine of Spark can be isolated like you suggest.
> > > >> > > >
> > > >> > > > I started working on this project because I felt that there
> was
> > no
> > > >> good
> > > >> > > > library for parallelizing method calls which can be plugged in
> > > easily
> > > >> > > into
> > > >> > > > an existing java project. Ultimately, if such a solution can
> be
> > > >> > > > incorporated in the Apache Commons, it would be a useful
> > addition
> > > to
> > > >> > the
> > > >> > > > Commons repository.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Arun
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Mon, Jun 12, 2017 at 3:01 PM, Gary Gregory <
> > > >> garydgregory@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Arun,
> > > >> > > > >
> > > >> > > > > Sure, and that is to be expected, Spark is more mature than
> a
> > > four
> > > >> > > class
> > > >> > > > > prototype. What I am trying to get to is that in order for
> the
> > > >> > library
> > > >> > > to
> > > >> > > > > be useful, you will end up with more in a first release, and
> > > after
> > > >> a
> > > >> > > > couple
> > > >> > > > > more releases, there will be more and more. Would Spark not
> > > have in
> > > >> > its
> > > >> > > > > guts the same kind of code your are proposing here? By
> > > extension,
> > > >> > will
> > > >> > > > you
> > > >> > > > > not end up with more framework-like (Spark-like) code and
> > > solutions
> > > >> > as
> > > >> > > > > found in Spark? I am just playing devil's advocate here ;-)
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > What would be interesting would be to find out if there is a
> > > core
> > > >> > part
> > > >> > > of
> > > >> > > > > Spark that is separable and ex tractable into a Commons
> > > component.
> > > >> > > Since
> > > >> > > > > Spark has a proven track record, it is more likely, that
> such
> > a
> > > >> > library
> > > >> > > > > would be generally useful than one created from scratch that
> > > does
> > > >> not
> > > >> > > > > integrate with anything else. Again, please do not take any
> of
> > > this
> > > >> > > > > personally, I am just playing here :-)
> > > >> > > > >
> > > >> > > > > Gary
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Mon, Jun 12, 2017 at 2:29 PM, Matt Sicker <
> > boards@gmail.com>
> > > >> > wrote:
> > > >> > > > >
> > > >> > > > > > I already see a huge difference here: Spark requires a
> bunch
> > > of
> > > >> > > > > > infrastructure to be set up, while this library is just a
> > > >> library.
> > > >> > > > > Similar
> > > >> > > > > > to Kafka Streams versus Spark Streaming or Flink or Storm
> or
> > > >> Samza
> > > >> > or
> > > >> > > > the
> > > >> > > > > > others.
> > > >> > > > > >
> > > >> > > > > > On 12 June 2017 at 16:28, Gary Gregory <
> > > garydgregory@gmail.com>
> > > >> > > wrote:
> > > >> > > > > >
> > > >> > > > > > > On Mon, Jun 12, 2017 at 2:26 PM, Arun Mohan <
> > > >> > > strider90arun@gmail.com
> > > >> > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi All,
> > > >> > > > > > > >
> > > >> > > > > > > > Good afternoon.
> > > >> > > > > > > >
> > > >> > > > > > > > I have been working on a java generic parallel
> execution
> > > >> > library
> > > >> > > > > which
> > > >> > > > > > > will
> > > >> > > > > > > > allow clients to execute methods in parallel
> > irrespective
> > > of
> > > >> > the
> > > >> > > > > number
> > > >> > > > > > > of
> > > >> > > > > > > > method arguments, type of method arguments, return
> type
> > of
> > > >> the
> > > >> > > > method
> > > >> > > > > > > etc.
> > > >> > > > > > > >
> > > >> > > > > > > > Here is the link to the source code:
> > > >> > > > > > > > https://github.com/striderarun/parallel-
> > execution-engine
> > > >> > > > > > > >
> > > >> > > > > > > > The project is in a nascent state and I am the only
> > > >> contributor
> > > >> > > so
> > > >> > > > > > far. I
> > > >> > > > > > > > am new to the Apache community and I would like to
> bring
> > > this
> > > >> > > > project
> > > >> > > > > > > into
> > > >> > > > > > > > Apache and improve, expand and build a developer
> > community
> > > >> > around
> > > >> > > > it.
> > > >> > > > > > > >
> > > >> > > > > > > > I think this project can be a sub project of Apache
> > > Commons
> > > >> > since
> > > >> > > > it
> > > >> > > > > > > > provides generic components for parallelizing any kind
> > of
> > > >> > > methods.
> > > >> > > > > > > >
> > > >> > > > > > > > Can somebody please guide me or suggest what other
> > > options I
> > > >> > can
> > > >> > > > > > explore
> > > >> > > > > > > ?
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > Hi Arun,
> > > >> > > > > > >
> > > >> > > > > > > Thank you for your proposal.
> > > >> > > > > > >
> > > >> > > > > > > How would this be different from Apache Spark?
> > > >> > > > > > >
> > > >> > > > > > > Thank you,
> > > >> > > > > > > Gary
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks,
> > > >> > > > > > > > Arun
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > --
> > > >> > > > > > Matt Sicker <boards@gmail.com>
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Matt Sicker <boards@gmail.com>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > > For additional commands, e-mail: dev-help@commons.apache.org
> > >
> > >
> >
> >
> > --
> > Matt Sicker <boards@gmail.com>
> >
>

--94eb2c03643ec305790551ce1f4a--