oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (3980)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Extra Compiler Tools
Date Sat, 08 Nov 2014 15:43:44 GMT
Ship it :) It’s ugly, but will keep the core stable at least through
another revision. 

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++






-----Original Message-----
From: Michael Starch <starchmd@umich.edu>
Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Date: Friday, November 7, 2014 at 10:39 PM
To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Subject: Re: Extra Compiler Tools

>Chris, Et Al.
>
>The solution write-up can be found here.  Turns out it isn't too
>ugly....just has a lot of duplication with the classes it is wrapping.
>
>https://reviews.apache.org/r/27773/
>
>Enjoy,
>
>-Michael
>
>On Fri, Nov 7, 2014 at 9:46 AM, Michael Starch <starchmd@umich.edu> wrote:
>
>> Sound fine, expect to see this on review board when I get it done and
>> passing my test.
>>
>> Michael
>> On Nov 7, 2014 9:44 AM, "Chris Mattmann" <chris.mattmann@gmail.com>
>>wrote:
>>
>>> Yep I’d like to see the non changing
>>> core structs version, in a patch to
>>> evaluate it. You said you know how, so let’s see it
>>> and then go from there. We can reserve 0.9 to
>>> consider changing structs if the code becomes
>>> too unwieldily to maintain.
>>>
>>> Good?
>>>
>>> Cheers,
>>> Chris
>>>
>>> ------------------------
>>> Chris Mattmann
>>> chris.mattmann@gmail.com
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Michael Starch <starchmd@umich.edu>
>>> Reply-To: <dev@oodt.apache.org>
>>> Date: Friday, November 7, 2014 at 9:21 AM
>>> To: <dev@oodt.apache.org>
>>> Subject: Re: Extra Compiler Tools
>>>
>>> >Chris,
>>> >
>>> >Did you ever come to a conclusion on this?
>>> >
>>> >-Michael
>>> >
>>> >On Wed, Nov 5, 2014 at 7:14 PM, Chris Mattmann
>>><chris.mattmann@gmail.com
>>> >
>>> >wrote:
>>> >
>>> >> OK, let me think about this tonight.
>>> >> Maybe we can figure this out tomorrow,
>>> >> I won’t hold this up longer than that.
>>> >>
>>> >> ------------------------
>>> >> Chris Mattmann
>>> >> chris.mattmann@gmail.com
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: Michael Starch <starchmd@umich.edu>
>>> >> Reply-To: <dev@oodt.apache.org>
>>> >> Date: Wednesday, November 5, 2014 at 8:23 PM
>>> >> To: <dev@oodt.apache.org>
>>> >> Subject: Re: Extra Compiler Tools
>>> >>
>>> >> >According to the specs, any subclass that implements Serializable
>>>must
>>> >> >manually implement the serialization of the parents' members.  I
>>> tested
>>> >> >this and it fails exactly as expected.  The parent's members aren't
>>> >> >serialized.
>>> >> >
>>> >> >Also, JobInput is an interface.... so I would have no way of
>>>catching
>>> >>all
>>> >> >of the possible implementations that could come at me.
>>> >> >
>>> >> >Michael
>>> >> >On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" <
>>> >> >chris.a.mattmann@jpl.nasa.gov> wrote:
>>> >> >
>>> >> >> Got it, Mike.
>>> >> >>
>>> >> >> Hmm, how about simply creating SerializableJobSpec and
>>> >> >> SerializableJob and SerializableJobInput and then making
>>> >> >> them sub-class their parents and implement Serializable.
>>> >> >> Then, use these classes in your Mesos implementation.
>>> >> >> That seems self-contained, doesn’t change core classes,
>>> >> >> and pretty easy, right?
>>> >> >>
>>> >> >> Cheers,
>>> >> >> Chris
>>> >> >>
>>> >> >>
>>> >> >> 
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> >> Chris Mattmann, Ph.D.
>>> >> >> Chief Architect
>>> >> >> Instrument Software and Science Data Systems Section (398)
>>> >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >> >> Office: 168-519, Mailstop: 168-527
>>> >> >> Email: chris.a.mattmann@nasa.gov
>>> >> >> WWW:  http://sunset.usc.edu/~mattmann/
>>> >> >> 
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> >> Adjunct Associate Professor, Computer Science Department
>>> >> >> University of Southern California, Los Angeles, CA 90089 USA
>>> >> >> 
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> -----Original Message-----
>>> >> >> From: Michael Starch <starchmd@umich.edu>
>>> >> >> Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
>>> >> >> Date: Wednesday, November 5, 2014 at 8:00 PM
>>> >> >> To: "dev@oodt.apache.org" <dev@oodt.apache.org>
>>> >> >> Subject: Re: Extra Compiler Tools
>>> >> >>
>>> >> >> >I need to serialize a JobSpec and children (Job and JobInput)
>>>to a
>>> >> >>byte[].
>>> >> >> >Java can do this automatically by marking all three as
>>> Serializable.
>>> >> >> >
>>> >> >> >The work around is to manually serialize to a private inner
>>>struct
>>> >>and
>>> >> >> >back
>>> >> >> >out again.  The inner class will have members for each
member in
>>> the
>>> >> >> >JobSpec and children.  Java can the auto-serialize that
without
>>> >> >>changing
>>> >> >> >the other three.
>>> >> >> >
>>> >> >> >It is ugly, and essentially a reimplementation of those
three
>>> >> >> >classes....but it is entirely self-contained.
>>> >> >> >
>>> >> >> >Michael
>>> >> >> >On Nov 5, 2014 6:45 PM, "Chris Mattmann"
>>><chris.mattmann@gmail.com
>>> >
>>> >> >> wrote:
>>> >> >> >
>>> >> >> >> Hey Mike,
>>> >> >> >>
>>> >> >> >> Hmm, what’s the work around just so I know
>>> >> >> >> what we’re trading against?
>>> >> >> >>
>>> >> >> >> Cheers,
>>> >> >> >> Chris
>>> >> >> >>
>>> >> >> >> ------------------------
>>> >> >> >> Chris Mattmann
>>> >> >> >> chris.mattmann@gmail.com
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> -----Original Message-----
>>> >> >> >> From: Michael Starch <starchmd@umich.edu>
>>> >> >> >> Reply-To: <dev@oodt.apache.org>
>>> >> >> >> Date: Wednesday, November 5, 2014 at 6:31 PM
>>> >> >> >> To: <dev@oodt.apache.org>
>>> >> >> >> Subject: Re: Extra Compiler Tools
>>> >> >> >>
>>> >> >> >> >That is basically what I did. Regardless, protobuff
proves
>>>to be
>>> >> >> >>overkill.
>>> >> >> >> >
>>> >> >> >> >If I mark those classes as serializable, the correct
>>>solution is
>>> >>2
>>> >> >> >>lines
>>> >> >> >> >of
>>> >> >> >> >code.  (protobuff was like 20).  Wrote a test
case, and it
>>>works
>>> >> >> >> >perfectly.
>>> >> >> >> >
>>> >> >> >> >If I cannot make JobSpec Job and JonInput implement
>>>Serializable
>>> >> >>then
>>> >> >> >>the
>>> >> >> >> >work around is simple too.
>>> >> >> >> >
>>> >> >> >> >What do you think?  Should I mark them as Serializable,
or
>>>use a
>>> >> >> >> >work-around.  Either is a better solution than
protobuff.
>>> >> >> >> >
>>> >> >> >> >Michael
>>> >> >> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann"
>>> >><chris.mattmann@gmail.com>
>>> >> >> >> wrote:
>>> >> >> >> >
>>> >> >> >> >> Mike, have you looked at this yet?
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >>
>>> >> >>
>>> >>
>>> >>
>>> 
>>>http://techtraits.com/build%20management/maven/2011/09/09/compiling-prot
>>>o
>>> >> >> >> >>co
>>> >> >> >> >> l-buffers-from-maven/
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> I’m going to play with it tonight and see
if
>>> >> >> >> >> I can help here. Do you have some files I
can test
>>> >> >> >> >> with? Can you attach them to JIRA or dropbox
them to me
>>> >> >> >> >> so I can scope?
>>> >> >> >> >>
>>> >> >> >> >> Cheers,
>>> >> >> >> >> Chris
>>> >> >> >> >>
>>> >> >> >> >> ------------------------
>>> >> >> >> >> Chris Mattmann
>>> >> >> >> >> chris.mattmann@gmail.com
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> -----Original Message-----
>>> >> >> >> >> From: Michael Starch <starchmd@umich.edu>
>>> >> >> >> >> Reply-To: <dev@oodt.apache.org>
>>> >> >> >> >> Date: Wednesday, November 5, 2014 at 5:37
PM
>>> >> >> >> >> To: <dev@oodt.apache.org>
>>> >> >> >> >> Subject: Re: Extra Compiler Tools
>>> >> >> >> >>
>>> >> >> >> >> >Ok....time for an audible.  Protoc needs
to be built from
>>> >> >>source, no
>>> >> >> >> >> >binary
>>> >> >> >> >> >distributions available.  Thus I am going
to purge
>>> >>proto-buffers
>>> >> >> >>from
>>> >> >> >> >>the
>>> >> >> >> >> >new code and be done with it.
>>> >> >> >> >> >
>>> >> >> >> >> >Any problem making the following classes/interfaces
>>>implement
>>> >> >> >> >> >java.io.Serializable:
>>> >> >> >> >> >
>>> >> >> >> >> >JobSpec
>>> >> >> >> >> >Job
>>> >> >> >> >> >JobInput
>>> >> >> >> >> >
>>> >> >> >> >> >Doing so would allow apache and native
java serialization
>>>and
>>> >> >>thus
>>> >> >> >>we
>>> >> >> >> >> >wouldn't need something like proto-buffers.
>>> >> >> >> >> >
>>> >> >> >> >> >-Michael
>>> >> >> >> >> >Thanks Mike +1
>>> >> >> >> >> >
>>> >> >> >> >> >------------------------
>>> >> >> >> >> >Chris Mattmann
>>> >> >> >> >> >chris.mattmann@gmail.com
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >-----Original Message-----
>>> >> >> >> >> >From: Michael Starch <starchmd@umich.edu>
>>> >> >> >> >> >Reply-To: <dev@oodt.apache.org>
>>> >> >> >> >> >Date: Wednesday, November 5, 2014 at
12:31 PM
>>> >> >> >> >> >To: <dev@oodt.apache.org>
>>> >> >> >> >> >Subject: Re: Extra Compiler Tools
>>> >> >> >> >> >
>>> >> >> >> >> >>Looks like you followed the same
reasoning chain that I
>>>did.
>>> >> >>Yes,
>>> >> >> >>I
>>> >> >> >> >>came
>>> >> >> >> >> >>to the same conclusion that ant-build
was best.
>>> >> >> >> >> >>
>>> >> >> >> >> >>I wasn't sure how to download protoc,
but you just
>>>answered
>>> >> >> >> >>that....so I
>>> >> >> >> >> >>think this is a great solution!
>>> >> >> >> >> >>
>>> >> >> >> >> >>Thanks,
>>> >> >> >> >> >>
>>> >> >> >> >> >>Michael
>>> >> >> >> >> >>
>>> >> >> >> >> >>
>>> >> >> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM,
Chris Mattmann
>>> >> >> >> >> >><chris.mattmann@gmail.com>
>>> >> >> >> >> >>wrote:
>>> >> >> >> >> >>
>>> >> >> >> >> >>> Hi Mike,
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Thanks for flushing this out.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> My thoughts on the below:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> -----Original Message-----
>>> >> >> >> >> >>> From: Michael Starch <starchmd@umich.edu>
>>> >> >> >> >> >>> Reply-To: <dev@oodt.apache.org>
>>> >> >> >> >> >>> Date: Wednesday, November 5,
2014 at 12:12 PM
>>> >> >> >> >> >>> To: <dev@oodt.apache.org>
>>> >> >> >> >> >>> Subject: Re: Extra Compiler
Tools
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> >I tried this approach. The
plugin requires a path to
>>>the
>>> >> >> >>"protoc"
>>> >> >> >> >>tool
>>> >> >> >> >> >>>and
>>> >> >> >> >> >>> >thus a working installation.
 This is what prompted
>>>the
>>> >> >> >>discussion.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Ah - no worries, what you could
do is:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> 1. only enable to plugin if
-Pwith-mesos is enabled;
>>>and
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> >
>>> >> >> >> >> >>> >Running the plugin under
a profile works.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Yep.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> > However, not running the
plugin
>>> >> >> >> >> >>> >causes compile errors in
dependant code.  Excluding
>>>this
>>> >>code
>>> >> >> >> >>except
>>> >> >> >> >> >>> >within
>>> >> >> >> >> >>> >the profile doesn't seem
to work, and is considered by
>>> >>some
>>> >> >>to
>>> >> >> >>be
>>> >> >> >> >>bad
>>> >> >> >> >> >>>form
>>> >> >> >> >> >>> >because there is nothing
inside the jar file that
>>>notes
>>> >>which
>>> >> >> >> >>profiles
>>> >> >> >> >> >>> >were
>>> >> >> >> >> >>> >used to compile.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Got it. Suggestion here would
be:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> 2. create a new module, cas-resource-mesos,
and inside
>>>of
>>> >>that
>>> >> >> >> >>module,
>>> >> >> >> >> >>> take one of the following approaches,
assuming the
>>>module
>>> >>is
>>> >> >> >> >>activated
>>> >> >> >> >> >>> when -Pwith-mesos is enabled:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> 2a. Maven Antrun like so (in
this old example):
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >>
>>> >> >> >>
>>> >> >>
>>> >>
>>> 
>>>http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
>>> >> >> >> >> >>>t
>>> >> >> >> >> >>>o-
>>> >> >> >> >> >>> maven2-build
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> (pro: more flexibility in case
protoc isn¹t there; to
>>>fail
>>> >>on
>>> >> >> >> >>error; to
>>> >> >> >> >> >>> only compile if
>>> >> >> >> >> >>> protoc is available
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> 2b. Maven protobuf plugin
>>> >> >> >> >> >>>
>>> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Here¹s how to enable a module
with a profile:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >>
>>> >> >> >>
>>> >> >>
>>> >>
>>> 
>>>http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
>>> >> >> >> >> >>>a
>>> >> >> >> >> >>>ct
>>> >> >> >> >> >>> ive-modules-by-profiles-in-maven/
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> It seems like that is a bad
idea though, based on that
>>> >> >> >>discussion.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> So, here¹s another option:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> 1. Inside of cas-resource (no
special new module or
>>> >>anything
>>> >> >> >>else)
>>> >> >> >> >> >>> 2. include some custom Ant magic
via a build.xml file
>>>and
>>> >>the
>>> >> >> >>Maven
>>> >> >> >> >> >>> AntRun plugin:
>>> >> >> >> >> >>>   2a. test if protoc is on the
system path, and if not,
>>> >> >>download
>>> >> >> >>it,
>>> >> >> >> >> >>>e.g.,
>>> >> >> >> >> >>> into the target directory (gets
deleted on clean)
>>> >> >> >> >> >>>   2b. call protoc and compile
after 2a
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> I would suggest this solution
as I think it¹s the most
>>> >>robust
>>> >> >>and
>>> >> >> >> >> >>>ensures
>>> >> >> >> >> >>> we always have a cas-resource
that includes mesos and
>>> >>compiled
>>> >> >> >> >> >>>correctly.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Cheers,
>>> >> >> >> >> >>> Chris
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> >
>>> >> >> >> >> >>> >Any ideas on how to continue?
>>> >> >> >> >> >>> >
>>> >> >> >> >> >>> >Michael
>>> >> >> >> >> >>> > On Nov 5, 2014 11:04 AM,
"Chris Mattmann"
>>> >> >> >> >><chris.mattmann@gmail.com>
>>> >> >> >> >> >>> >wrote:
>>> >> >> >> >> >>> >
>>> >> >> >> >> >>> >> Hi Mike,
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> Great discussion. It
would be nice if there was
>>> >> >> >> >> >>> >> a protoc Maven plugin:
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> Looks like there is.
My suggestion:
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> 1. use a Profile, something
like -Pwith-mesos and
>>> >> >> >> >> >>> >> then when activated;
>>> >> >> >> >> >>> >> 2. call the above plugin
if -Pwith-mesos is
>>>activated
>>> >> >> >> >> >>> >> in the resource manager
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> Sound good?
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> Cheers,
>>> >> >> >> >> >>> >> Chris
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> ------------------------
>>> >> >> >> >> >>> >> Chris Mattmann
>>> >> >> >> >> >>> >> chris.mattmann@gmail.com
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> -----Original Message-----
>>> >> >> >> >> >>> >> From: Michael Starch
<starchmd@umich.edu>
>>> >> >> >> >> >>> >> Reply-To: <dev@oodt.apache.org>
>>> >> >> >> >> >>> >> Date: Wednesday, November
5, 2014 at 11:46 AM
>>> >> >> >> >> >>> >> To: <dev@oodt.apache.org>
>>> >> >> >> >> >>> >> Subject: Extra Compiler
Tools
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> >All,
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >I am trying to
integrate apache-mesos with our
>>> resource
>>> >> >> >>manager.
>>> >> >> >> >> >>> >>However,
>>> >> >> >> >> >>> >> >mesos uses a technology
called "protobuff" from
>>>Google
>>> >>for
>>> >> >> >> >> >>> >> >marshaling/unmarshaling
data.
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >This requires running
a tool called "protoc" to
>>> >>generate a
>>> >> >> >> >>source
>>> >> >> >> >> >>>file
>>> >> >> >> >> >>> >>in
>>> >> >> >> >> >>> >> >java.  What is
the best way to integrate this step
>>> into
>>> >> >>our
>>> >> >> >> >>build
>>> >> >> >> >> >>> >>process?
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >Options I can conceive
of:
>>> >> >> >> >> >>> >> >   -Check in generated
java file
>>> >> >> >> >> >>> >> >   -Require "protoc"
installation to build resource
>>> >> >>manager
>>> >> >> >> >> >>> >> >   -Separate extra
resource package into new module
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >None of these ideas
are very clean.
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >Any other ideas?
 I tried setting up a profile to
>>>only
>>> >> >> >>compile
>>> >> >> >> >> >>>these
>>> >> >> >> >> >>> >> >sources when selected,
but that turned out not to
>>> work.
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >-Michael Starch
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >>
>>> >> >>
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>

Mime
View raw message