oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Starch <starc...@umich.edu>
Subject Re: Extra Compiler Tools
Date Sat, 08 Nov 2014 06:39:03 GMT
Chris, Et Al.

The solution write-up can be found here.  Turns out it isn't too
ugly....just has a lot of duplication with the classes it is wrapping.

https://reviews.apache.org/r/27773/

Enjoy,

-Michael

On Fri, Nov 7, 2014 at 9:46 AM, Michael Starch <starchmd@umich.edu> wrote:

> Sound fine, expect to see this on review board when I get it done and
> passing my test.
>
> Michael
> On Nov 7, 2014 9:44 AM, "Chris Mattmann" <chris.mattmann@gmail.com> wrote:
>
>> Yep I’d like to see the non changing
>> core structs version, in a patch to
>> evaluate it. You said you know how, so let’s see it
>> and then go from there. We can reserve 0.9 to
>> consider changing structs if the code becomes
>> too unwieldily to maintain.
>>
>> Good?
>>
>> Cheers,
>> Chris
>>
>> ------------------------
>> Chris Mattmann
>> chris.mattmann@gmail.com
>>
>>
>>
>>
>> -----Original Message-----
>> From: Michael Starch <starchmd@umich.edu>
>> Reply-To: <dev@oodt.apache.org>
>> Date: Friday, November 7, 2014 at 9:21 AM
>> To: <dev@oodt.apache.org>
>> Subject: Re: Extra Compiler Tools
>>
>> >Chris,
>> >
>> >Did you ever come to a conclusion on this?
>> >
>> >-Michael
>> >
>> >On Wed, Nov 5, 2014 at 7:14 PM, Chris Mattmann <chris.mattmann@gmail.com
>> >
>> >wrote:
>> >
>> >> OK, let me think about this tonight.
>> >> Maybe we can figure this out tomorrow,
>> >> I won’t hold this up longer than that.
>> >>
>> >> ------------------------
>> >> Chris Mattmann
>> >> chris.mattmann@gmail.com
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Michael Starch <starchmd@umich.edu>
>> >> Reply-To: <dev@oodt.apache.org>
>> >> Date: Wednesday, November 5, 2014 at 8:23 PM
>> >> To: <dev@oodt.apache.org>
>> >> Subject: Re: Extra Compiler Tools
>> >>
>> >> >According to the specs, any subclass that implements Serializable must
>> >> >manually implement the serialization of the parents' members.  I
>> tested
>> >> >this and it fails exactly as expected.  The parent's members aren't
>> >> >serialized.
>> >> >
>> >> >Also, JobInput is an interface.... so I would have no way of catching
>> >>all
>> >> >of the possible implementations that could come at me.
>> >> >
>> >> >Michael
>> >> >On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" <
>> >> >chris.a.mattmann@jpl.nasa.gov> wrote:
>> >> >
>> >> >> Got it, Mike.
>> >> >>
>> >> >> Hmm, how about simply creating SerializableJobSpec and
>> >> >> SerializableJob and SerializableJobInput and then making
>> >> >> them sub-class their parents and implement Serializable.
>> >> >> Then, use these classes in your Mesos implementation.
>> >> >> That seems self-contained, doesn’t change core classes,
>> >> >> and pretty easy, right?
>> >> >>
>> >> >> Cheers,
>> >> >> Chris
>> >> >>
>> >> >>
>> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >> Chris Mattmann, Ph.D.
>> >> >> Chief Architect
>> >> >> Instrument Software and Science Data Systems Section (398)
>> >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> >> Office: 168-519, Mailstop: 168-527
>> >> >> Email: chris.a.mattmann@nasa.gov
>> >> >> WWW:  http://sunset.usc.edu/~mattmann/
>> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >> Adjunct Associate Professor, Computer Science Department
>> >> >> University of Southern California, Los Angeles, CA 90089 USA
>> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: Michael Starch <starchmd@umich.edu>
>> >> >> Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
>> >> >> Date: Wednesday, November 5, 2014 at 8:00 PM
>> >> >> To: "dev@oodt.apache.org" <dev@oodt.apache.org>
>> >> >> Subject: Re: Extra Compiler Tools
>> >> >>
>> >> >> >I need to serialize a JobSpec and children (Job and JobInput)
to a
>> >> >>byte[].
>> >> >> >Java can do this automatically by marking all three as
>> Serializable.
>> >> >> >
>> >> >> >The work around is to manually serialize to a private inner
struct
>> >>and
>> >> >> >back
>> >> >> >out again.  The inner class will have members for each member
in
>> the
>> >> >> >JobSpec and children.  Java can the auto-serialize that without
>> >> >>changing
>> >> >> >the other three.
>> >> >> >
>> >> >> >It is ugly, and essentially a reimplementation of those three
>> >> >> >classes....but it is entirely self-contained.
>> >> >> >
>> >> >> >Michael
>> >> >> >On Nov 5, 2014 6:45 PM, "Chris Mattmann" <chris.mattmann@gmail.com
>> >
>> >> >> wrote:
>> >> >> >
>> >> >> >> Hey Mike,
>> >> >> >>
>> >> >> >> Hmm, what’s the work around just so I know
>> >> >> >> what we’re trading against?
>> >> >> >>
>> >> >> >> Cheers,
>> >> >> >> Chris
>> >> >> >>
>> >> >> >> ------------------------
>> >> >> >> Chris Mattmann
>> >> >> >> chris.mattmann@gmail.com
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> -----Original Message-----
>> >> >> >> From: Michael Starch <starchmd@umich.edu>
>> >> >> >> Reply-To: <dev@oodt.apache.org>
>> >> >> >> Date: Wednesday, November 5, 2014 at 6:31 PM
>> >> >> >> To: <dev@oodt.apache.org>
>> >> >> >> Subject: Re: Extra Compiler Tools
>> >> >> >>
>> >> >> >> >That is basically what I did. Regardless, protobuff
proves to be
>> >> >> >>overkill.
>> >> >> >> >
>> >> >> >> >If I mark those classes as serializable, the correct
solution is
>> >>2
>> >> >> >>lines
>> >> >> >> >of
>> >> >> >> >code.  (protobuff was like 20).  Wrote a test case,
and it works
>> >> >> >> >perfectly.
>> >> >> >> >
>> >> >> >> >If I cannot make JobSpec Job and JonInput implement
Serializable
>> >> >>then
>> >> >> >>the
>> >> >> >> >work around is simple too.
>> >> >> >> >
>> >> >> >> >What do you think?  Should I mark them as Serializable,
or use a
>> >> >> >> >work-around.  Either is a better solution than protobuff.
>> >> >> >> >
>> >> >> >> >Michael
>> >> >> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann"
>> >><chris.mattmann@gmail.com>
>> >> >> >> wrote:
>> >> >> >> >
>> >> >> >> >> Mike, have you looked at this yet?
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>> http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
>> >> >> >> >>co
>> >> >> >> >> l-buffers-from-maven/
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> I’m going to play with it tonight and see if
>> >> >> >> >> I can help here. Do you have some files I can
test
>> >> >> >> >> with? Can you attach them to JIRA or dropbox
them to me
>> >> >> >> >> so I can scope?
>> >> >> >> >>
>> >> >> >> >> Cheers,
>> >> >> >> >> Chris
>> >> >> >> >>
>> >> >> >> >> ------------------------
>> >> >> >> >> Chris Mattmann
>> >> >> >> >> chris.mattmann@gmail.com
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> -----Original Message-----
>> >> >> >> >> From: Michael Starch <starchmd@umich.edu>
>> >> >> >> >> Reply-To: <dev@oodt.apache.org>
>> >> >> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM
>> >> >> >> >> To: <dev@oodt.apache.org>
>> >> >> >> >> Subject: Re: Extra Compiler Tools
>> >> >> >> >>
>> >> >> >> >> >Ok....time for an audible.  Protoc needs
to be built from
>> >> >>source, no
>> >> >> >> >> >binary
>> >> >> >> >> >distributions available.  Thus I am going
to purge
>> >>proto-buffers
>> >> >> >>from
>> >> >> >> >>the
>> >> >> >> >> >new code and be done with it.
>> >> >> >> >> >
>> >> >> >> >> >Any problem making the following classes/interfaces
implement
>> >> >> >> >> >java.io.Serializable:
>> >> >> >> >> >
>> >> >> >> >> >JobSpec
>> >> >> >> >> >Job
>> >> >> >> >> >JobInput
>> >> >> >> >> >
>> >> >> >> >> >Doing so would allow apache and native java
serialization and
>> >> >>thus
>> >> >> >>we
>> >> >> >> >> >wouldn't need something like proto-buffers.
>> >> >> >> >> >
>> >> >> >> >> >-Michael
>> >> >> >> >> >Thanks Mike +1
>> >> >> >> >> >
>> >> >> >> >> >------------------------
>> >> >> >> >> >Chris Mattmann
>> >> >> >> >> >chris.mattmann@gmail.com
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >-----Original Message-----
>> >> >> >> >> >From: Michael Starch <starchmd@umich.edu>
>> >> >> >> >> >Reply-To: <dev@oodt.apache.org>
>> >> >> >> >> >Date: Wednesday, November 5, 2014 at 12:31
PM
>> >> >> >> >> >To: <dev@oodt.apache.org>
>> >> >> >> >> >Subject: Re: Extra Compiler Tools
>> >> >> >> >> >
>> >> >> >> >> >>Looks like you followed the same reasoning
chain that I did.
>> >> >>Yes,
>> >> >> >>I
>> >> >> >> >>came
>> >> >> >> >> >>to the same conclusion that ant-build
was best.
>> >> >> >> >> >>
>> >> >> >> >> >>I wasn't sure how to download protoc,
but you just answered
>> >> >> >> >>that....so I
>> >> >> >> >> >>think this is a great solution!
>> >> >> >> >> >>
>> >> >> >> >> >>Thanks,
>> >> >> >> >> >>
>> >> >> >> >> >>Michael
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris
Mattmann
>> >> >> >> >> >><chris.mattmann@gmail.com>
>> >> >> >> >> >>wrote:
>> >> >> >> >> >>
>> >> >> >> >> >>> Hi Mike,
>> >> >> >> >> >>>
>> >> >> >> >> >>> Thanks for flushing this out.
>> >> >> >> >> >>>
>> >> >> >> >> >>> My thoughts on the below:
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> -----Original Message-----
>> >> >> >> >> >>> From: Michael Starch <starchmd@umich.edu>
>> >> >> >> >> >>> Reply-To: <dev@oodt.apache.org>
>> >> >> >> >> >>> Date: Wednesday, November 5, 2014
at 12:12 PM
>> >> >> >> >> >>> To: <dev@oodt.apache.org>
>> >> >> >> >> >>> Subject: Re: Extra Compiler Tools
>> >> >> >> >> >>>
>> >> >> >> >> >>> >I tried this approach. The plugin
requires a path to the
>> >> >> >>"protoc"
>> >> >> >> >>tool
>> >> >> >> >> >>>and
>> >> >> >> >> >>> >thus a working installation.
 This is what prompted the
>> >> >> >>discussion.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Ah - no worries, what you could
do is:
>> >> >> >> >> >>>
>> >> >> >> >> >>> 1. only enable to plugin if -Pwith-mesos
is enabled; and
>> >> >> >> >> >>>
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >Running the plugin under a profile
works.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Yep.
>> >> >> >> >> >>>
>> >> >> >> >> >>> > However, not running the plugin
>> >> >> >> >> >>> >causes compile errors in dependant
code.  Excluding this
>> >>code
>> >> >> >> >>except
>> >> >> >> >> >>> >within
>> >> >> >> >> >>> >the profile doesn't seem to
work, and is considered by
>> >>some
>> >> >>to
>> >> >> >>be
>> >> >> >> >>bad
>> >> >> >> >> >>>form
>> >> >> >> >> >>> >because there is nothing inside
the jar file that notes
>> >>which
>> >> >> >> >>profiles
>> >> >> >> >> >>> >were
>> >> >> >> >> >>> >used to compile.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Got it. Suggestion here would be:
>> >> >> >> >> >>>
>> >> >> >> >> >>> 2. create a new module, cas-resource-mesos,
and inside of
>> >>that
>> >> >> >> >>module,
>> >> >> >> >> >>> take one of the following approaches,
assuming the module
>> >>is
>> >> >> >> >>activated
>> >> >> >> >> >>> when -Pwith-mesos is enabled:
>> >> >> >> >> >>>
>> >> >> >> >> >>> 2a. Maven Antrun like so (in this
old example):
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
>> >> >> >> >> >>>t
>> >> >> >> >> >>>o-
>> >> >> >> >> >>> maven2-build
>> >> >> >> >> >>>
>> >> >> >> >> >>> (pro: more flexibility in case protoc
isn¹t there; to fail
>> >>on
>> >> >> >> >>error; to
>> >> >> >> >> >>> only compile if
>> >> >> >> >> >>> protoc is available
>> >> >> >> >> >>>
>> >> >> >> >> >>> 2b. Maven protobuf plugin
>> >> >> >> >> >>>
>> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >> >> >> >>>
>> >> >> >> >> >>> Here¹s how to enable a module with
a profile:
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
>> >> >> >> >> >>>a
>> >> >> >> >> >>>ct
>> >> >> >> >> >>> ive-modules-by-profiles-in-maven/
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> It seems like that is a bad idea
though, based on that
>> >> >> >>discussion.
>> >> >> >> >> >>>
>> >> >> >> >> >>> So, here¹s another option:
>> >> >> >> >> >>>
>> >> >> >> >> >>> 1. Inside of cas-resource (no special
new module or
>> >>anything
>> >> >> >>else)
>> >> >> >> >> >>> 2. include some custom Ant magic
via a build.xml file and
>> >>the
>> >> >> >>Maven
>> >> >> >> >> >>> AntRun plugin:
>> >> >> >> >> >>>   2a. test if protoc is on the system
path, and if not,
>> >> >>download
>> >> >> >>it,
>> >> >> >> >> >>>e.g.,
>> >> >> >> >> >>> into the target directory (gets
deleted on clean)
>> >> >> >> >> >>>   2b. call protoc and compile after
2a
>> >> >> >> >> >>>
>> >> >> >> >> >>> I would suggest this solution as
I think it¹s the most
>> >>robust
>> >> >>and
>> >> >> >> >> >>>ensures
>> >> >> >> >> >>> we always have a cas-resource that
includes mesos and
>> >>compiled
>> >> >> >> >> >>>correctly.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Cheers,
>> >> >> >> >> >>> Chris
>> >> >> >> >> >>>
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >Any ideas on how to continue?
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >Michael
>> >> >> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris
Mattmann"
>> >> >> >> >><chris.mattmann@gmail.com>
>> >> >> >> >> >>> >wrote:
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >> Hi Mike,
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Great discussion. It would
be nice if there was
>> >> >> >> >> >>> >> a protoc Maven plugin:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Looks like there is. My
suggestion:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> 1. use a Profile, something
like -Pwith-mesos and
>> >> >> >> >> >>> >> then when activated;
>> >> >> >> >> >>> >> 2. call the above plugin
if -Pwith-mesos is activated
>> >> >> >> >> >>> >> in the resource manager
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Sound good?
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Cheers,
>> >> >> >> >> >>> >> Chris
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> ------------------------
>> >> >> >> >> >>> >> Chris Mattmann
>> >> >> >> >> >>> >> chris.mattmann@gmail.com
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> -----Original Message-----
>> >> >> >> >> >>> >> From: Michael Starch <starchmd@umich.edu>
>> >> >> >> >> >>> >> Reply-To: <dev@oodt.apache.org>
>> >> >> >> >> >>> >> Date: Wednesday, November
5, 2014 at 11:46 AM
>> >> >> >> >> >>> >> To: <dev@oodt.apache.org>
>> >> >> >> >> >>> >> Subject: Extra Compiler
Tools
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> >All,
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >I am trying to integrate
apache-mesos with our
>> resource
>> >> >> >>manager.
>> >> >> >> >> >>> >>However,
>> >> >> >> >> >>> >> >mesos uses a technology
called "protobuff" from Google
>> >>for
>> >> >> >> >> >>> >> >marshaling/unmarshaling
data.
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >This requires running
a tool called "protoc" to
>> >>generate a
>> >> >> >> >>source
>> >> >> >> >> >>>file
>> >> >> >> >> >>> >>in
>> >> >> >> >> >>> >> >java.  What is the
best way to integrate this step
>> into
>> >> >>our
>> >> >> >> >>build
>> >> >> >> >> >>> >>process?
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >Options I can conceive
of:
>> >> >> >> >> >>> >> >   -Check in generated
java file
>> >> >> >> >> >>> >> >   -Require "protoc"
installation to build resource
>> >> >>manager
>> >> >> >> >> >>> >> >   -Separate extra
resource package into new module
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >None of these ideas
are very clean.
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >Any other ideas?  I
tried setting up a profile to only
>> >> >> >>compile
>> >> >> >> >> >>>these
>> >> >> >> >> >>> >> >sources when selected,
but that turned out not to
>> work.
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >-Michael Starch
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>> >>
>>
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message