oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Starch <starc...@umich.edu>
Subject Re: Extra Compiler Tools
Date Thu, 06 Nov 2014 03:23:07 GMT
According to the specs, any subclass that implements Serializable must
manually implement the serialization of the parents' members.  I tested
this and it fails exactly as expected.  The parent's members aren't
serialized.

Also, JobInput is an interface.... so I would have no way of catching all
of the possible implementations that could come at me.

Michael
On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" <
chris.a.mattmann@jpl.nasa.gov> wrote:

> Got it, Mike.
>
> Hmm, how about simply creating SerializableJobSpec and
> SerializableJob and SerializableJobInput and then making
> them sub-class their parents and implement Serializable.
> Then, use these classes in your Mesos implementation.
> That seems self-contained, doesn’t change core classes,
> and pretty easy, right?
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Michael Starch <starchmd@umich.edu>
> Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Date: Wednesday, November 5, 2014 at 8:00 PM
> To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Subject: Re: Extra Compiler Tools
>
> >I need to serialize a JobSpec and children (Job and JobInput) to a byte[].
> >Java can do this automatically by marking all three as Serializable.
> >
> >The work around is to manually serialize to a private inner struct and
> >back
> >out again.  The inner class will have members for each member in the
> >JobSpec and children.  Java can the auto-serialize that without changing
> >the other three.
> >
> >It is ugly, and essentially a reimplementation of those three
> >classes....but it is entirely self-contained.
> >
> >Michael
> >On Nov 5, 2014 6:45 PM, "Chris Mattmann" <chris.mattmann@gmail.com>
> wrote:
> >
> >> Hey Mike,
> >>
> >> Hmm, what’s the work around just so I know
> >> what we’re trading against?
> >>
> >> Cheers,
> >> Chris
> >>
> >> ------------------------
> >> Chris Mattmann
> >> chris.mattmann@gmail.com
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Michael Starch <starchmd@umich.edu>
> >> Reply-To: <dev@oodt.apache.org>
> >> Date: Wednesday, November 5, 2014 at 6:31 PM
> >> To: <dev@oodt.apache.org>
> >> Subject: Re: Extra Compiler Tools
> >>
> >> >That is basically what I did. Regardless, protobuff proves to be
> >>overkill.
> >> >
> >> >If I mark those classes as serializable, the correct solution is 2
> >>lines
> >> >of
> >> >code.  (protobuff was like 20).  Wrote a test case, and it works
> >> >perfectly.
> >> >
> >> >If I cannot make JobSpec Job and JonInput implement Serializable then
> >>the
> >> >work around is simple too.
> >> >
> >> >What do you think?  Should I mark them as Serializable, or use a
> >> >work-around.  Either is a better solution than protobuff.
> >> >
> >> >Michael
> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann" <chris.mattmann@gmail.com>
> >> wrote:
> >> >
> >> >> Mike, have you looked at this yet?
> >> >>
> >> >>
> >> >>
> >>
> >>
> http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
> >> >>co
> >> >> l-buffers-from-maven/
> >> >>
> >> >>
> >> >> I’m going to play with it tonight and see if
> >> >> I can help here. Do you have some files I can test
> >> >> with? Can you attach them to JIRA or dropbox them to me
> >> >> so I can scope?
> >> >>
> >> >> Cheers,
> >> >> Chris
> >> >>
> >> >> ------------------------
> >> >> Chris Mattmann
> >> >> chris.mattmann@gmail.com
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> -----Original Message-----
> >> >> From: Michael Starch <starchmd@umich.edu>
> >> >> Reply-To: <dev@oodt.apache.org>
> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM
> >> >> To: <dev@oodt.apache.org>
> >> >> Subject: Re: Extra Compiler Tools
> >> >>
> >> >> >Ok....time for an audible.  Protoc needs to be built from source,
no
> >> >> >binary
> >> >> >distributions available.  Thus I am going to purge proto-buffers
> >>from
> >> >>the
> >> >> >new code and be done with it.
> >> >> >
> >> >> >Any problem making the following classes/interfaces implement
> >> >> >java.io.Serializable:
> >> >> >
> >> >> >JobSpec
> >> >> >Job
> >> >> >JobInput
> >> >> >
> >> >> >Doing so would allow apache and native java serialization and thus
> >>we
> >> >> >wouldn't need something like proto-buffers.
> >> >> >
> >> >> >-Michael
> >> >> >Thanks Mike +1
> >> >> >
> >> >> >------------------------
> >> >> >Chris Mattmann
> >> >> >chris.mattmann@gmail.com
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >-----Original Message-----
> >> >> >From: Michael Starch <starchmd@umich.edu>
> >> >> >Reply-To: <dev@oodt.apache.org>
> >> >> >Date: Wednesday, November 5, 2014 at 12:31 PM
> >> >> >To: <dev@oodt.apache.org>
> >> >> >Subject: Re: Extra Compiler Tools
> >> >> >
> >> >> >>Looks like you followed the same reasoning chain that I did.
 Yes,
> >>I
> >> >>came
> >> >> >>to the same conclusion that ant-build was best.
> >> >> >>
> >> >> >>I wasn't sure how to download protoc, but you just answered
> >> >>that....so I
> >> >> >>think this is a great solution!
> >> >> >>
> >> >> >>Thanks,
> >> >> >>
> >> >> >>Michael
> >> >> >>
> >> >> >>
> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
> >> >> >><chris.mattmann@gmail.com>
> >> >> >>wrote:
> >> >> >>
> >> >> >>> Hi Mike,
> >> >> >>>
> >> >> >>> Thanks for flushing this out.
> >> >> >>>
> >> >> >>> My thoughts on the below:
> >> >> >>>
> >> >> >>>
> >> >> >>> -----Original Message-----
> >> >> >>> From: Michael Starch <starchmd@umich.edu>
> >> >> >>> Reply-To: <dev@oodt.apache.org>
> >> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
> >> >> >>> To: <dev@oodt.apache.org>
> >> >> >>> Subject: Re: Extra Compiler Tools
> >> >> >>>
> >> >> >>> >I tried this approach. The plugin requires a path
to the
> >>"protoc"
> >> >>tool
> >> >> >>>and
> >> >> >>> >thus a working installation.  This is what prompted
the
> >>discussion.
> >> >> >>>
> >> >> >>> Ah - no worries, what you could do is:
> >> >> >>>
> >> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
> >> >> >>>
> >> >> >>> >
> >> >> >>> >Running the plugin under a profile works.
> >> >> >>>
> >> >> >>> Yep.
> >> >> >>>
> >> >> >>> > However, not running the plugin
> >> >> >>> >causes compile errors in dependant code.  Excluding
this code
> >> >>except
> >> >> >>> >within
> >> >> >>> >the profile doesn't seem to work, and is considered
by some to
> >>be
> >> >>bad
> >> >> >>>form
> >> >> >>> >because there is nothing inside the jar file that
notes which
> >> >>profiles
> >> >> >>> >were
> >> >> >>> >used to compile.
> >> >> >>>
> >> >> >>> Got it. Suggestion here would be:
> >> >> >>>
> >> >> >>> 2. create a new module, cas-resource-mesos, and inside
of that
> >> >>module,
> >> >> >>> take one of the following approaches, assuming the module
is
> >> >>activated
> >> >> >>> when -Pwith-mesos is enabled:
> >> >> >>>
> >> >> >>> 2a. Maven Antrun like so (in this old example):
> >> >> >>>
> >> >> >>>
> >> >>
> >>
> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
> >> >> >>>t
> >> >> >>>o-
> >> >> >>> maven2-build
> >> >> >>>
> >> >> >>> (pro: more flexibility in case protoc isn¹t there; to
fail on
> >> >>error; to
> >> >> >>> only compile if
> >> >> >>> protoc is available
> >> >> >>>
> >> >> >>> 2b. Maven protobuf plugin
> >> >> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >> >> >>>
> >> >> >>> Here¹s how to enable a module with a profile:
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >>
> >>
> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
> >> >> >>>a
> >> >> >>>ct
> >> >> >>> ive-modules-by-profiles-in-maven/
> >> >> >>>
> >> >> >>>
> >> >> >>> It seems like that is a bad idea though, based on that
> >>discussion.
> >> >> >>>
> >> >> >>> So, here¹s another option:
> >> >> >>>
> >> >> >>> 1. Inside of cas-resource (no special new module or anything
> >>else)
> >> >> >>> 2. include some custom Ant magic via a build.xml file
and the
> >>Maven
> >> >> >>> AntRun plugin:
> >> >> >>>   2a. test if protoc is on the system path, and if not,
download
> >>it,
> >> >> >>>e.g.,
> >> >> >>> into the target directory (gets deleted on clean)
> >> >> >>>   2b. call protoc and compile after 2a
> >> >> >>>
> >> >> >>> I would suggest this solution as I think it¹s the most
robust and
> >> >> >>>ensures
> >> >> >>> we always have a cas-resource that includes mesos and
compiled
> >> >> >>>correctly.
> >> >> >>>
> >> >> >>> Cheers,
> >> >> >>> Chris
> >> >> >>>
> >> >> >>> >
> >> >> >>> >Any ideas on how to continue?
> >> >> >>> >
> >> >> >>> >Michael
> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
> >> >><chris.mattmann@gmail.com>
> >> >> >>> >wrote:
> >> >> >>> >
> >> >> >>> >> Hi Mike,
> >> >> >>> >>
> >> >> >>> >> Great discussion. It would be nice if there was
> >> >> >>> >> a protoc Maven plugin:
> >> >> >>> >>
> >> >> >>> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> Looks like there is. My suggestion:
> >> >> >>> >>
> >> >> >>> >> 1. use a Profile, something like -Pwith-mesos
and
> >> >> >>> >> then when activated;
> >> >> >>> >> 2. call the above plugin if -Pwith-mesos is activated
> >> >> >>> >> in the resource manager
> >> >> >>> >>
> >> >> >>> >> Sound good?
> >> >> >>> >>
> >> >> >>> >> Cheers,
> >> >> >>> >> Chris
> >> >> >>> >>
> >> >> >>> >> ------------------------
> >> >> >>> >> Chris Mattmann
> >> >> >>> >> chris.mattmann@gmail.com
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> -----Original Message-----
> >> >> >>> >> From: Michael Starch <starchmd@umich.edu>
> >> >> >>> >> Reply-To: <dev@oodt.apache.org>
> >> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
> >> >> >>> >> To: <dev@oodt.apache.org>
> >> >> >>> >> Subject: Extra Compiler Tools
> >> >> >>> >>
> >> >> >>> >> >All,
> >> >> >>> >> >
> >> >> >>> >> >I am trying to integrate apache-mesos with
our resource
> >>manager.
> >> >> >>> >>However,
> >> >> >>> >> >mesos uses a technology called "protobuff"
from Google for
> >> >> >>> >> >marshaling/unmarshaling data.
> >> >> >>> >> >
> >> >> >>> >> >This requires running a tool called "protoc"
to generate a
> >> >>source
> >> >> >>>file
> >> >> >>> >>in
> >> >> >>> >> >java.  What is the best way to integrate
this step into our
> >> >>build
> >> >> >>> >>process?
> >> >> >>> >> >
> >> >> >>> >> >Options I can conceive of:
> >> >> >>> >> >   -Check in generated java file
> >> >> >>> >> >   -Require "protoc" installation to build
resource manager
> >> >> >>> >> >   -Separate extra resource package into
new module
> >> >> >>> >> >
> >> >> >>> >> >None of these ideas are very clean.
> >> >> >>> >> >
> >> >> >>> >> >Any other ideas?  I tried setting up a profile
to only
> >>compile
> >> >> >>>these
> >> >> >>> >> >sources when selected, but that turned out
not to work.
> >> >> >>> >> >
> >> >> >>> >> >-Michael Starch
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message