oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mattmann <chris.mattm...@gmail.com>
Subject Re: Extra Compiler Tools
Date Thu, 06 Nov 2014 03:14:28 GMT
OK, let me think about this tonight.
Maybe we can figure this out tomorrow,
I won’t hold this up longer than that.

------------------------
Chris Mattmann
chris.mattmann@gmail.com




-----Original Message-----
From: Michael Starch <starchmd@umich.edu>
Reply-To: <dev@oodt.apache.org>
Date: Wednesday, November 5, 2014 at 8:23 PM
To: <dev@oodt.apache.org>
Subject: Re: Extra Compiler Tools

>According to the specs, any subclass that implements Serializable must
>manually implement the serialization of the parents' members.  I tested
>this and it fails exactly as expected.  The parent's members aren't
>serialized.
>
>Also, JobInput is an interface.... so I would have no way of catching all
>of the possible implementations that could come at me.
>
>Michael
>On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Got it, Mike.
>>
>> Hmm, how about simply creating SerializableJobSpec and
>> SerializableJob and SerializableJobInput and then making
>> them sub-class their parents and implement Serializable.
>> Then, use these classes in your Mesos implementation.
>> That seems self-contained, doesn’t change core classes,
>> and pretty easy, right?
>>
>> Cheers,
>> Chris
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Chief Architect
>> Instrument Software and Science Data Systems Section (398)
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 168-519, Mailstop: 168-527
>> Email: chris.a.mattmann@nasa.gov
>> WWW:  http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Associate Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Michael Starch <starchmd@umich.edu>
>> Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
>> Date: Wednesday, November 5, 2014 at 8:00 PM
>> To: "dev@oodt.apache.org" <dev@oodt.apache.org>
>> Subject: Re: Extra Compiler Tools
>>
>> >I need to serialize a JobSpec and children (Job and JobInput) to a
>>byte[].
>> >Java can do this automatically by marking all three as Serializable.
>> >
>> >The work around is to manually serialize to a private inner struct and
>> >back
>> >out again.  The inner class will have members for each member in the
>> >JobSpec and children.  Java can the auto-serialize that without
>>changing
>> >the other three.
>> >
>> >It is ugly, and essentially a reimplementation of those three
>> >classes....but it is entirely self-contained.
>> >
>> >Michael
>> >On Nov 5, 2014 6:45 PM, "Chris Mattmann" <chris.mattmann@gmail.com>
>> wrote:
>> >
>> >> Hey Mike,
>> >>
>> >> Hmm, what’s the work around just so I know
>> >> what we’re trading against?
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> ------------------------
>> >> Chris Mattmann
>> >> chris.mattmann@gmail.com
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Michael Starch <starchmd@umich.edu>
>> >> Reply-To: <dev@oodt.apache.org>
>> >> Date: Wednesday, November 5, 2014 at 6:31 PM
>> >> To: <dev@oodt.apache.org>
>> >> Subject: Re: Extra Compiler Tools
>> >>
>> >> >That is basically what I did. Regardless, protobuff proves to be
>> >>overkill.
>> >> >
>> >> >If I mark those classes as serializable, the correct solution is 2
>> >>lines
>> >> >of
>> >> >code.  (protobuff was like 20).  Wrote a test case, and it works
>> >> >perfectly.
>> >> >
>> >> >If I cannot make JobSpec Job and JonInput implement Serializable
>>then
>> >>the
>> >> >work around is simple too.
>> >> >
>> >> >What do you think?  Should I mark them as Serializable, or use a
>> >> >work-around.  Either is a better solution than protobuff.
>> >> >
>> >> >Michael
>> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann" <chris.mattmann@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Mike, have you looked at this yet?
>> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>> 
>>http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
>> >> >>co
>> >> >> l-buffers-from-maven/
>> >> >>
>> >> >>
>> >> >> I’m going to play with it tonight and see if
>> >> >> I can help here. Do you have some files I can test
>> >> >> with? Can you attach them to JIRA or dropbox them to me
>> >> >> so I can scope?
>> >> >>
>> >> >> Cheers,
>> >> >> Chris
>> >> >>
>> >> >> ------------------------
>> >> >> Chris Mattmann
>> >> >> chris.mattmann@gmail.com
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: Michael Starch <starchmd@umich.edu>
>> >> >> Reply-To: <dev@oodt.apache.org>
>> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM
>> >> >> To: <dev@oodt.apache.org>
>> >> >> Subject: Re: Extra Compiler Tools
>> >> >>
>> >> >> >Ok....time for an audible.  Protoc needs to be built from
>>source, no
>> >> >> >binary
>> >> >> >distributions available.  Thus I am going to purge proto-buffers
>> >>from
>> >> >>the
>> >> >> >new code and be done with it.
>> >> >> >
>> >> >> >Any problem making the following classes/interfaces implement
>> >> >> >java.io.Serializable:
>> >> >> >
>> >> >> >JobSpec
>> >> >> >Job
>> >> >> >JobInput
>> >> >> >
>> >> >> >Doing so would allow apache and native java serialization and
>>thus
>> >>we
>> >> >> >wouldn't need something like proto-buffers.
>> >> >> >
>> >> >> >-Michael
>> >> >> >Thanks Mike +1
>> >> >> >
>> >> >> >------------------------
>> >> >> >Chris Mattmann
>> >> >> >chris.mattmann@gmail.com
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >-----Original Message-----
>> >> >> >From: Michael Starch <starchmd@umich.edu>
>> >> >> >Reply-To: <dev@oodt.apache.org>
>> >> >> >Date: Wednesday, November 5, 2014 at 12:31 PM
>> >> >> >To: <dev@oodt.apache.org>
>> >> >> >Subject: Re: Extra Compiler Tools
>> >> >> >
>> >> >> >>Looks like you followed the same reasoning chain that I
did.
>>Yes,
>> >>I
>> >> >>came
>> >> >> >>to the same conclusion that ant-build was best.
>> >> >> >>
>> >> >> >>I wasn't sure how to download protoc, but you just answered
>> >> >>that....so I
>> >> >> >>think this is a great solution!
>> >> >> >>
>> >> >> >>Thanks,
>> >> >> >>
>> >> >> >>Michael
>> >> >> >>
>> >> >> >>
>> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
>> >> >> >><chris.mattmann@gmail.com>
>> >> >> >>wrote:
>> >> >> >>
>> >> >> >>> Hi Mike,
>> >> >> >>>
>> >> >> >>> Thanks for flushing this out.
>> >> >> >>>
>> >> >> >>> My thoughts on the below:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> -----Original Message-----
>> >> >> >>> From: Michael Starch <starchmd@umich.edu>
>> >> >> >>> Reply-To: <dev@oodt.apache.org>
>> >> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
>> >> >> >>> To: <dev@oodt.apache.org>
>> >> >> >>> Subject: Re: Extra Compiler Tools
>> >> >> >>>
>> >> >> >>> >I tried this approach. The plugin requires a path
to the
>> >>"protoc"
>> >> >>tool
>> >> >> >>>and
>> >> >> >>> >thus a working installation.  This is what prompted
the
>> >>discussion.
>> >> >> >>>
>> >> >> >>> Ah - no worries, what you could do is:
>> >> >> >>>
>> >> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled;
and
>> >> >> >>>
>> >> >> >>> >
>> >> >> >>> >Running the plugin under a profile works.
>> >> >> >>>
>> >> >> >>> Yep.
>> >> >> >>>
>> >> >> >>> > However, not running the plugin
>> >> >> >>> >causes compile errors in dependant code.  Excluding
this code
>> >> >>except
>> >> >> >>> >within
>> >> >> >>> >the profile doesn't seem to work, and is considered
by some
>>to
>> >>be
>> >> >>bad
>> >> >> >>>form
>> >> >> >>> >because there is nothing inside the jar file that
notes which
>> >> >>profiles
>> >> >> >>> >were
>> >> >> >>> >used to compile.
>> >> >> >>>
>> >> >> >>> Got it. Suggestion here would be:
>> >> >> >>>
>> >> >> >>> 2. create a new module, cas-resource-mesos, and inside
of that
>> >> >>module,
>> >> >> >>> take one of the following approaches, assuming the
module is
>> >> >>activated
>> >> >> >>> when -Pwith-mesos is enabled:
>> >> >> >>>
>> >> >> >>> 2a. Maven Antrun like so (in this old example):
>> >> >> >>>
>> >> >> >>>
>> >> >>
>> >>
>> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
>> >> >> >>>t
>> >> >> >>>o-
>> >> >> >>> maven2-build
>> >> >> >>>
>> >> >> >>> (pro: more flexibility in case protoc isn¹t there;
to fail on
>> >> >>error; to
>> >> >> >>> only compile if
>> >> >> >>> protoc is available
>> >> >> >>>
>> >> >> >>> 2b. Maven protobuf plugin
>> >> >> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >> >>>
>> >> >> >>> Here¹s how to enable a module with a profile:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >>
>> >>
>> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
>> >> >> >>>a
>> >> >> >>>ct
>> >> >> >>> ive-modules-by-profiles-in-maven/
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> It seems like that is a bad idea though, based on
that
>> >>discussion.
>> >> >> >>>
>> >> >> >>> So, here¹s another option:
>> >> >> >>>
>> >> >> >>> 1. Inside of cas-resource (no special new module or
anything
>> >>else)
>> >> >> >>> 2. include some custom Ant magic via a build.xml file
and the
>> >>Maven
>> >> >> >>> AntRun plugin:
>> >> >> >>>   2a. test if protoc is on the system path, and if
not,
>>download
>> >>it,
>> >> >> >>>e.g.,
>> >> >> >>> into the target directory (gets deleted on clean)
>> >> >> >>>   2b. call protoc and compile after 2a
>> >> >> >>>
>> >> >> >>> I would suggest this solution as I think it¹s the
most robust
>>and
>> >> >> >>>ensures
>> >> >> >>> we always have a cas-resource that includes mesos
and compiled
>> >> >> >>>correctly.
>> >> >> >>>
>> >> >> >>> Cheers,
>> >> >> >>> Chris
>> >> >> >>>
>> >> >> >>> >
>> >> >> >>> >Any ideas on how to continue?
>> >> >> >>> >
>> >> >> >>> >Michael
>> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
>> >> >><chris.mattmann@gmail.com>
>> >> >> >>> >wrote:
>> >> >> >>> >
>> >> >> >>> >> Hi Mike,
>> >> >> >>> >>
>> >> >> >>> >> Great discussion. It would be nice if there
was
>> >> >> >>> >> a protoc Maven plugin:
>> >> >> >>> >>
>> >> >> >>> >> 
>>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> Looks like there is. My suggestion:
>> >> >> >>> >>
>> >> >> >>> >> 1. use a Profile, something like -Pwith-mesos
and
>> >> >> >>> >> then when activated;
>> >> >> >>> >> 2. call the above plugin if -Pwith-mesos
is activated
>> >> >> >>> >> in the resource manager
>> >> >> >>> >>
>> >> >> >>> >> Sound good?
>> >> >> >>> >>
>> >> >> >>> >> Cheers,
>> >> >> >>> >> Chris
>> >> >> >>> >>
>> >> >> >>> >> ------------------------
>> >> >> >>> >> Chris Mattmann
>> >> >> >>> >> chris.mattmann@gmail.com
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> -----Original Message-----
>> >> >> >>> >> From: Michael Starch <starchmd@umich.edu>
>> >> >> >>> >> Reply-To: <dev@oodt.apache.org>
>> >> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46
AM
>> >> >> >>> >> To: <dev@oodt.apache.org>
>> >> >> >>> >> Subject: Extra Compiler Tools
>> >> >> >>> >>
>> >> >> >>> >> >All,
>> >> >> >>> >> >
>> >> >> >>> >> >I am trying to integrate apache-mesos
with our resource
>> >>manager.
>> >> >> >>> >>However,
>> >> >> >>> >> >mesos uses a technology called "protobuff"
from Google for
>> >> >> >>> >> >marshaling/unmarshaling data.
>> >> >> >>> >> >
>> >> >> >>> >> >This requires running a tool called "protoc"
to generate a
>> >> >>source
>> >> >> >>>file
>> >> >> >>> >>in
>> >> >> >>> >> >java.  What is the best way to integrate
this step into
>>our
>> >> >>build
>> >> >> >>> >>process?
>> >> >> >>> >> >
>> >> >> >>> >> >Options I can conceive of:
>> >> >> >>> >> >   -Check in generated java file
>> >> >> >>> >> >   -Require "protoc" installation to
build resource
>>manager
>> >> >> >>> >> >   -Separate extra resource package into
new module
>> >> >> >>> >> >
>> >> >> >>> >> >None of these ideas are very clean.
>> >> >> >>> >> >
>> >> >> >>> >> >Any other ideas?  I tried setting up
a profile to only
>> >>compile
>> >> >> >>>these
>> >> >> >>> >> >sources when selected, but that turned
out not to work.
>> >> >> >>> >> >
>> >> >> >>> >> >-Michael Starch
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>> >>
>>
>>



Mime
View raw message