oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyler Palsulich <tpalsul...@gmail.com>
Subject Re: Extra Compiler Tools
Date Thu, 06 Nov 2014 03:03:14 GMT
What is the disadvantage of using Java's built in Serialization?

Tyler

On Wed, Nov 5, 2014 at 10:00 PM, Michael Starch <starchmd@umich.edu> wrote:

> I need to serialize a JobSpec and children (Job and JobInput) to a byte[].
> Java can do this automatically by marking all three as Serializable.
>
> The work around is to manually serialize to a private inner struct and back
> out again.  The inner class will have members for each member in the
> JobSpec and children.  Java can the auto-serialize that without changing
> the other three.
>
> It is ugly, and essentially a reimplementation of those three
> classes....but it is entirely self-contained.
>
> Michael
> On Nov 5, 2014 6:45 PM, "Chris Mattmann" <chris.mattmann@gmail.com> wrote:
>
> > Hey Mike,
> >
> > Hmm, what’s the work around just so I know
> > what we’re trading against?
> >
> > Cheers,
> > Chris
> >
> > ------------------------
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> > -----Original Message-----
> > From: Michael Starch <starchmd@umich.edu>
> > Reply-To: <dev@oodt.apache.org>
> > Date: Wednesday, November 5, 2014 at 6:31 PM
> > To: <dev@oodt.apache.org>
> > Subject: Re: Extra Compiler Tools
> >
> > >That is basically what I did. Regardless, protobuff proves to be
> overkill.
> > >
> > >If I mark those classes as serializable, the correct solution is 2 lines
> > >of
> > >code.  (protobuff was like 20).  Wrote a test case, and it works
> > >perfectly.
> > >
> > >If I cannot make JobSpec Job and JonInput implement Serializable then
> the
> > >work around is simple too.
> > >
> > >What do you think?  Should I mark them as Serializable, or use a
> > >work-around.  Either is a better solution than protobuff.
> > >
> > >Michael
> > >On Nov 5, 2014 4:44 PM, "Chris Mattmann" <chris.mattmann@gmail.com>
> > wrote:
> > >
> > >> Mike, have you looked at this yet?
> > >>
> > >>
> > >>
> >
> http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
> > >>co
> > >> l-buffers-from-maven/
> > >>
> > >>
> > >> I’m going to play with it tonight and see if
> > >> I can help here. Do you have some files I can test
> > >> with? Can you attach them to JIRA or dropbox them to me
> > >> so I can scope?
> > >>
> > >> Cheers,
> > >> Chris
> > >>
> > >> ------------------------
> > >> Chris Mattmann
> > >> chris.mattmann@gmail.com
> > >>
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Michael Starch <starchmd@umich.edu>
> > >> Reply-To: <dev@oodt.apache.org>
> > >> Date: Wednesday, November 5, 2014 at 5:37 PM
> > >> To: <dev@oodt.apache.org>
> > >> Subject: Re: Extra Compiler Tools
> > >>
> > >> >Ok....time for an audible.  Protoc needs to be built from source, no
> > >> >binary
> > >> >distributions available.  Thus I am going to purge proto-buffers from
> > >>the
> > >> >new code and be done with it.
> > >> >
> > >> >Any problem making the following classes/interfaces implement
> > >> >java.io.Serializable:
> > >> >
> > >> >JobSpec
> > >> >Job
> > >> >JobInput
> > >> >
> > >> >Doing so would allow apache and native java serialization and thus
we
> > >> >wouldn't need something like proto-buffers.
> > >> >
> > >> >-Michael
> > >> >Thanks Mike +1
> > >> >
> > >> >------------------------
> > >> >Chris Mattmann
> > >> >chris.mattmann@gmail.com
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >-----Original Message-----
> > >> >From: Michael Starch <starchmd@umich.edu>
> > >> >Reply-To: <dev@oodt.apache.org>
> > >> >Date: Wednesday, November 5, 2014 at 12:31 PM
> > >> >To: <dev@oodt.apache.org>
> > >> >Subject: Re: Extra Compiler Tools
> > >> >
> > >> >>Looks like you followed the same reasoning chain that I did.  Yes,
I
> > >>came
> > >> >>to the same conclusion that ant-build was best.
> > >> >>
> > >> >>I wasn't sure how to download protoc, but you just answered
> > >>that....so I
> > >> >>think this is a great solution!
> > >> >>
> > >> >>Thanks,
> > >> >>
> > >> >>Michael
> > >> >>
> > >> >>
> > >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
> > >> >><chris.mattmann@gmail.com>
> > >> >>wrote:
> > >> >>
> > >> >>> Hi Mike,
> > >> >>>
> > >> >>> Thanks for flushing this out.
> > >> >>>
> > >> >>> My thoughts on the below:
> > >> >>>
> > >> >>>
> > >> >>> -----Original Message-----
> > >> >>> From: Michael Starch <starchmd@umich.edu>
> > >> >>> Reply-To: <dev@oodt.apache.org>
> > >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
> > >> >>> To: <dev@oodt.apache.org>
> > >> >>> Subject: Re: Extra Compiler Tools
> > >> >>>
> > >> >>> >I tried this approach. The plugin requires a path to the
"protoc"
> > >>tool
> > >> >>>and
> > >> >>> >thus a working installation.  This is what prompted the
> discussion.
> > >> >>>
> > >> >>> Ah - no worries, what you could do is:
> > >> >>>
> > >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
> > >> >>>
> > >> >>> >
> > >> >>> >Running the plugin under a profile works.
> > >> >>>
> > >> >>> Yep.
> > >> >>>
> > >> >>> > However, not running the plugin
> > >> >>> >causes compile errors in dependant code.  Excluding this
code
> > >>except
> > >> >>> >within
> > >> >>> >the profile doesn't seem to work, and is considered by
some to be
> > >>bad
> > >> >>>form
> > >> >>> >because there is nothing inside the jar file that notes
which
> > >>profiles
> > >> >>> >were
> > >> >>> >used to compile.
> > >> >>>
> > >> >>> Got it. Suggestion here would be:
> > >> >>>
> > >> >>> 2. create a new module, cas-resource-mesos, and inside of
that
> > >>module,
> > >> >>> take one of the following approaches, assuming the module
is
> > >>activated
> > >> >>> when -Pwith-mesos is enabled:
> > >> >>>
> > >> >>> 2a. Maven Antrun like so (in this old example):
> > >> >>>
> > >> >>>
> > >>
> > http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
> > >> >>>t
> > >> >>>o-
> > >> >>> maven2-build
> > >> >>>
> > >> >>> (pro: more flexibility in case protoc isn¹t there; to fail
on
> > >>error; to
> > >> >>> only compile if
> > >> >>> protoc is available
> > >> >>>
> > >> >>> 2b. Maven protobuf plugin
> > >> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> > >> >>>
> > >> >>> Here¹s how to enable a module with a profile:
> > >> >>>
> > >> >>>
> > >> >>>
> > >>
> > http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
> > >> >>>a
> > >> >>>ct
> > >> >>> ive-modules-by-profiles-in-maven/
> > >> >>>
> > >> >>>
> > >> >>> It seems like that is a bad idea though, based on that discussion.
> > >> >>>
> > >> >>> So, here¹s another option:
> > >> >>>
> > >> >>> 1. Inside of cas-resource (no special new module or anything
else)
> > >> >>> 2. include some custom Ant magic via a build.xml file and
the
> Maven
> > >> >>> AntRun plugin:
> > >> >>>   2a. test if protoc is on the system path, and if not, download
> it,
> > >> >>>e.g.,
> > >> >>> into the target directory (gets deleted on clean)
> > >> >>>   2b. call protoc and compile after 2a
> > >> >>>
> > >> >>> I would suggest this solution as I think it¹s the most robust
and
> > >> >>>ensures
> > >> >>> we always have a cas-resource that includes mesos and compiled
> > >> >>>correctly.
> > >> >>>
> > >> >>> Cheers,
> > >> >>> Chris
> > >> >>>
> > >> >>> >
> > >> >>> >Any ideas on how to continue?
> > >> >>> >
> > >> >>> >Michael
> > >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
> > >><chris.mattmann@gmail.com>
> > >> >>> >wrote:
> > >> >>> >
> > >> >>> >> Hi Mike,
> > >> >>> >>
> > >> >>> >> Great discussion. It would be nice if there was
> > >> >>> >> a protoc Maven plugin:
> > >> >>> >>
> > >> >>> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> > >> >>> >>
> > >> >>> >>
> > >> >>> >> Looks like there is. My suggestion:
> > >> >>> >>
> > >> >>> >> 1. use a Profile, something like -Pwith-mesos and
> > >> >>> >> then when activated;
> > >> >>> >> 2. call the above plugin if -Pwith-mesos is activated
> > >> >>> >> in the resource manager
> > >> >>> >>
> > >> >>> >> Sound good?
> > >> >>> >>
> > >> >>> >> Cheers,
> > >> >>> >> Chris
> > >> >>> >>
> > >> >>> >> ------------------------
> > >> >>> >> Chris Mattmann
> > >> >>> >> chris.mattmann@gmail.com
> > >> >>> >>
> > >> >>> >>
> > >> >>> >>
> > >> >>> >>
> > >> >>> >> -----Original Message-----
> > >> >>> >> From: Michael Starch <starchmd@umich.edu>
> > >> >>> >> Reply-To: <dev@oodt.apache.org>
> > >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
> > >> >>> >> To: <dev@oodt.apache.org>
> > >> >>> >> Subject: Extra Compiler Tools
> > >> >>> >>
> > >> >>> >> >All,
> > >> >>> >> >
> > >> >>> >> >I am trying to integrate apache-mesos with our
resource
> manager.
> > >> >>> >>However,
> > >> >>> >> >mesos uses a technology called "protobuff" from
Google for
> > >> >>> >> >marshaling/unmarshaling data.
> > >> >>> >> >
> > >> >>> >> >This requires running a tool called "protoc"
to generate a
> > >>source
> > >> >>>file
> > >> >>> >>in
> > >> >>> >> >java.  What is the best way to integrate this
step into our
> > >>build
> > >> >>> >>process?
> > >> >>> >> >
> > >> >>> >> >Options I can conceive of:
> > >> >>> >> >   -Check in generated java file
> > >> >>> >> >   -Require "protoc" installation to build resource
manager
> > >> >>> >> >   -Separate extra resource package into new
module
> > >> >>> >> >
> > >> >>> >> >None of these ideas are very clean.
> > >> >>> >> >
> > >> >>> >> >Any other ideas?  I tried setting up a profile
to only compile
> > >> >>>these
> > >> >>> >> >sources when selected, but that turned out not
to work.
> > >> >>> >> >
> > >> >>> >> >-Michael Starch
> > >> >>> >>
> > >> >>> >>
> > >> >>> >>
> > >> >>>
> > >> >>>
> > >> >>>
> > >>
> > >>
> > >>
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message