Return-Path: X-Original-To: apmail-oodt-dev-archive@www.apache.org Delivered-To: apmail-oodt-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D188C17836 for ; Thu, 6 Nov 2014 04:15:12 +0000 (UTC) Received: (qmail 83374 invoked by uid 500); 6 Nov 2014 04:15:12 -0000 Delivered-To: apmail-oodt-dev-archive@oodt.apache.org Received: (qmail 83331 invoked by uid 500); 6 Nov 2014 04:15:12 -0000 Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@oodt.apache.org Delivered-To: mailing list dev@oodt.apache.org Received: (qmail 83317 invoked by uid 99); 6 Nov 2014 04:15:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Nov 2014 04:15:12 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of chris.mattmann@gmail.com designates 209.85.192.176 as permitted sender) Received: from [209.85.192.176] (HELO mail-pd0-f176.google.com) (209.85.192.176) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Nov 2014 04:14:46 +0000 Received: by mail-pd0-f176.google.com with SMTP id ft15so336423pdb.7 for ; Wed, 05 Nov 2014 20:14:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=user-agent:date:subject:from:to:message-id:thread-topic:references :in-reply-to:mime-version:content-type:content-transfer-encoding; bh=sOp8/12YPm051gCVQ285ZXwMTFi/AH0SvvbtgkzIUaY=; b=pohxOZ2z3F3oOxIfDxuv0pOxCRp8orLy7Dnj4AiVuP8i8B8vsEKxMfjRn8GHwXdqcS vxrjjGRDGDMp2QOs8/ds71FQSVs88QsxQtlse5Pgw7nA5AakcBuXkfWuM7iHNB7znH9E cNekTO6EG1mBNrXtpYq+OoT2obea0+nBOwyjiitc/4ldagQzTHfutIcSYreFipUPMkgV GttHJrXvu+V72FYLM3+g9WPzqflo7xgphgax33kneUFH55iL1jaBL8Xi32qRLTVqsyIh /rKEJWNdqVNR3beerRQoVCZZGyzucjiq1AFPcnK/MwTEfLx1XGxhH06bu25ZarhZb3Xx C/Ug== X-Received: by 10.66.119.70 with SMTP id ks6mr1856548pab.74.1415247284652; Wed, 05 Nov 2014 20:14:44 -0800 (PST) Received: from [192.168.1.131] (107-128-41-172.lightspeed.irvnca.sbcglobal.net. [107.128.41.172]) by mx.google.com with ESMTPSA id pl3sm4535224pbb.50.2014.11.05.20.14.42 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 05 Nov 2014 20:14:43 -0800 (PST) User-Agent: Microsoft-MacOutlook/14.4.5.141003 Date: Wed, 05 Nov 2014 20:14:28 -0700 Subject: Re: Extra Compiler Tools From: Chris Mattmann To: Message-ID: Thread-Topic: Extra Compiler Tools References: In-Reply-To: Mime-version: 1.0 Content-type: text/plain; charset="EUC-KR" Content-transfer-encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org OK, let me think about this tonight. Maybe we can figure this out tomorrow, I won=A1=AFt hold this up longer than that. ------------------------ Chris Mattmann chris.mattmann@gmail.com -----Original Message----- From: Michael Starch Reply-To: Date: Wednesday, November 5, 2014 at 8:23 PM To: Subject: Re: Extra Compiler Tools >According to the specs, any subclass that implements Serializable must >manually implement the serialization of the parents' members. I tested >this and it fails exactly as expected. The parent's members aren't >serialized. > >Also, JobInput is an interface.... so I would have no way of catching all >of the possible implementations that could come at me. > >Michael >On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" < >chris.a.mattmann@jpl.nasa.gov> wrote: > >> Got it, Mike. >> >> Hmm, how about simply creating SerializableJobSpec and >> SerializableJob and SerializableJobInput and then making >> them sub-class their parents and implement Serializable. >> Then, use these classes in your Mesos implementation. >> That seems self-contained, doesn=A1=AFt change core classes, >> and pretty easy, right? >> >> Cheers, >> Chris >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Chris Mattmann, Ph.D. >> Chief Architect >> Instrument Software and Science Data Systems Section (398) >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 168-519, Mailstop: 168-527 >> Email: chris.a.mattmann@nasa.gov >> WWW: http://sunset.usc.edu/~mattmann/ >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Adjunct Associate Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >> >> >> >> -----Original Message----- >> From: Michael Starch >> Reply-To: "dev@oodt.apache.org" >> Date: Wednesday, November 5, 2014 at 8:00 PM >> To: "dev@oodt.apache.org" >> Subject: Re: Extra Compiler Tools >> >> >I need to serialize a JobSpec and children (Job and JobInput) to a >>byte[]. >> >Java can do this automatically by marking all three as Serializable. >> > >> >The work around is to manually serialize to a private inner struct and >> >back >> >out again. The inner class will have members for each member in the >> >JobSpec and children. Java can the auto-serialize that without >>changing >> >the other three. >> > >> >It is ugly, and essentially a reimplementation of those three >> >classes....but it is entirely self-contained. >> > >> >Michael >> >On Nov 5, 2014 6:45 PM, "Chris Mattmann" >> wrote: >> > >> >> Hey Mike, >> >> >> >> Hmm, what=A1=AFs the work around just so I know >> >> what we=A1=AFre trading against? >> >> >> >> Cheers, >> >> Chris >> >> >> >> ------------------------ >> >> Chris Mattmann >> >> chris.mattmann@gmail.com >> >> >> >> >> >> >> >> >> >> -----Original Message----- >> >> From: Michael Starch >> >> Reply-To: >> >> Date: Wednesday, November 5, 2014 at 6:31 PM >> >> To: >> >> Subject: Re: Extra Compiler Tools >> >> >> >> >That is basically what I did. Regardless, protobuff proves to be >> >>overkill. >> >> > >> >> >If I mark those classes as serializable, the correct solution is 2 >> >>lines >> >> >of >> >> >code. (protobuff was like 20). Wrote a test case, and it works >> >> >perfectly. >> >> > >> >> >If I cannot make JobSpec Job and JonInput implement Serializable >>then >> >>the >> >> >work around is simple too. >> >> > >> >> >What do you think? Should I mark them as Serializable, or use a >> >> >work-around. Either is a better solution than protobuff. >> >> > >> >> >Michael >> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann" >> >> wrote: >> >> > >> >> >> Mike, have you looked at this yet? >> >> >> >> >> >> >> >> >> >> >> >> >> >>=20 >>http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto >> >> >>co >> >> >> l-buffers-from-maven/ >> >> >> >> >> >> >> >> >> I=A1=AFm going to play with it tonight and see if >> >> >> I can help here. Do you have some files I can test >> >> >> with? Can you attach them to JIRA or dropbox them to me >> >> >> so I can scope? >> >> >> >> >> >> Cheers, >> >> >> Chris >> >> >> >> >> >> ------------------------ >> >> >> Chris Mattmann >> >> >> chris.mattmann@gmail.com >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -----Original Message----- >> >> >> From: Michael Starch >> >> >> Reply-To: >> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM >> >> >> To: >> >> >> Subject: Re: Extra Compiler Tools >> >> >> >> >> >> >Ok....time for an audible. Protoc needs to be built from >>source, no >> >> >> >binary >> >> >> >distributions available. Thus I am going to purge proto-buffers >> >>from >> >> >>the >> >> >> >new code and be done with it. >> >> >> > >> >> >> >Any problem making the following classes/interfaces implement >> >> >> >java.io.Serializable: >> >> >> > >> >> >> >JobSpec >> >> >> >Job >> >> >> >JobInput >> >> >> > >> >> >> >Doing so would allow apache and native java serialization and >>thus >> >>we >> >> >> >wouldn't need something like proto-buffers. >> >> >> > >> >> >> >-Michael >> >> >> >Thanks Mike +1 >> >> >> > >> >> >> >------------------------ >> >> >> >Chris Mattmann >> >> >> >chris.mattmann@gmail.com >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> >-----Original Message----- >> >> >> >From: Michael Starch >> >> >> >Reply-To: >> >> >> >Date: Wednesday, November 5, 2014 at 12:31 PM >> >> >> >To: >> >> >> >Subject: Re: Extra Compiler Tools >> >> >> > >> >> >> >>Looks like you followed the same reasoning chain that I did. >>Yes, >> >>I >> >> >>came >> >> >> >>to the same conclusion that ant-build was best. >> >> >> >> >> >> >> >>I wasn't sure how to download protoc, but you just answered >> >> >>that....so I >> >> >> >>think this is a great solution! >> >> >> >> >> >> >> >>Thanks, >> >> >> >> >> >> >> >>Michael >> >> >> >> >> >> >> >> >> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann >> >> >> >> >> >> >> >>wrote: >> >> >> >> >> >> >> >>> Hi Mike, >> >> >> >>> >> >> >> >>> Thanks for flushing this out. >> >> >> >>> >> >> >> >>> My thoughts on the below: >> >> >> >>> >> >> >> >>> >> >> >> >>> -----Original Message----- >> >> >> >>> From: Michael Starch >> >> >> >>> Reply-To: >> >> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM >> >> >> >>> To: >> >> >> >>> Subject: Re: Extra Compiler Tools >> >> >> >>> >> >> >> >>> >I tried this approach. The plugin requires a path to the >> >>"protoc" >> >> >>tool >> >> >> >>>and >> >> >> >>> >thus a working installation. This is what prompted the >> >>discussion. >> >> >> >>> >> >> >> >>> Ah - no worries, what you could do is: >> >> >> >>> >> >> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and >> >> >> >>> >> >> >> >>> > >> >> >> >>> >Running the plugin under a profile works. >> >> >> >>> >> >> >> >>> Yep. >> >> >> >>> >> >> >> >>> > However, not running the plugin >> >> >> >>> >causes compile errors in dependant code. Excluding this code >> >> >>except >> >> >> >>> >within >> >> >> >>> >the profile doesn't seem to work, and is considered by some >>to >> >>be >> >> >>bad >> >> >> >>>form >> >> >> >>> >because there is nothing inside the jar file that notes which >> >> >>profiles >> >> >> >>> >were >> >> >> >>> >used to compile. >> >> >> >>> >> >> >> >>> Got it. Suggestion here would be: >> >> >> >>> >> >> >> >>> 2. create a new module, cas-resource-mesos, and inside of that >> >> >>module, >> >> >> >>> take one of the following approaches, assuming the module is >> >> >>activated >> >> >> >>> when -Pwith-mesos is enabled: >> >> >> >>> >> >> >> >>> 2a. Maven Antrun like so (in this old example): >> >> >> >>> >> >> >> >>> >> >> >> >> >> >> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in >> >> >> >>>t >> >> >> >>>o- >> >> >> >>> maven2-build >> >> >> >>> >> >> >> >>> (pro: more flexibility in case protoc isn=A9=F6t there; to fail on >> >> >>error; to >> >> >> >>> only compile if >> >> >> >>> protoc is available >> >> >> >>> >> >> >> >>> 2b. Maven protobuf plugin >> >> >> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html >> >> >> >>> >> >> >> >>> Here=A9=F6s how to enable a module with a profile: >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >> >> >> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de >> >> >> >>>a >> >> >> >>>ct >> >> >> >>> ive-modules-by-profiles-in-maven/ >> >> >> >>> >> >> >> >>> >> >> >> >>> It seems like that is a bad idea though, based on that >> >>discussion. >> >> >> >>> >> >> >> >>> So, here=A9=F6s another option: >> >> >> >>> >> >> >> >>> 1. Inside of cas-resource (no special new module or anything >> >>else) >> >> >> >>> 2. include some custom Ant magic via a build.xml file and the >> >>Maven >> >> >> >>> AntRun plugin: >> >> >> >>> 2a. test if protoc is on the system path, and if not, >>download >> >>it, >> >> >> >>>e.g., >> >> >> >>> into the target directory (gets deleted on clean) >> >> >> >>> 2b. call protoc and compile after 2a >> >> >> >>> >> >> >> >>> I would suggest this solution as I think it=A9=F6s the most robust >>and >> >> >> >>>ensures >> >> >> >>> we always have a cas-resource that includes mesos and compiled >> >> >> >>>correctly. >> >> >> >>> >> >> >> >>> Cheers, >> >> >> >>> Chris >> >> >> >>> >> >> >> >>> > >> >> >> >>> >Any ideas on how to continue? >> >> >> >>> > >> >> >> >>> >Michael >> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann" >> >> >> >> >> >> >>> >wrote: >> >> >> >>> > >> >> >> >>> >> Hi Mike, >> >> >> >>> >> >> >> >> >>> >> Great discussion. It would be nice if there was >> >> >> >>> >> a protoc Maven plugin: >> >> >> >>> >> >> >> >> >>> >>=20 >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> Looks like there is. My suggestion: >> >> >> >>> >> >> >> >> >>> >> 1. use a Profile, something like -Pwith-mesos and >> >> >> >>> >> then when activated; >> >> >> >>> >> 2. call the above plugin if -Pwith-mesos is activated >> >> >> >>> >> in the resource manager >> >> >> >>> >> >> >> >> >>> >> Sound good? >> >> >> >>> >> >> >> >> >>> >> Cheers, >> >> >> >>> >> Chris >> >> >> >>> >> >> >> >> >>> >> ------------------------ >> >> >> >>> >> Chris Mattmann >> >> >> >>> >> chris.mattmann@gmail.com >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> -----Original Message----- >> >> >> >>> >> From: Michael Starch >> >> >> >>> >> Reply-To: >> >> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM >> >> >> >>> >> To: >> >> >> >>> >> Subject: Extra Compiler Tools >> >> >> >>> >> >> >> >> >>> >> >All, >> >> >> >>> >> > >> >> >> >>> >> >I am trying to integrate apache-mesos with our resource >> >>manager. >> >> >> >>> >>However, >> >> >> >>> >> >mesos uses a technology called "protobuff" from Google for >> >> >> >>> >> >marshaling/unmarshaling data. >> >> >> >>> >> > >> >> >> >>> >> >This requires running a tool called "protoc" to generate a >> >> >>source >> >> >> >>>file >> >> >> >>> >>in >> >> >> >>> >> >java. What is the best way to integrate this step into >>our >> >> >>build >> >> >> >>> >>process? >> >> >> >>> >> > >> >> >> >>> >> >Options I can conceive of: >> >> >> >>> >> > -Check in generated java file >> >> >> >>> >> > -Require "protoc" installation to build resource >>manager >> >> >> >>> >> > -Separate extra resource package into new module >> >> >> >>> >> > >> >> >> >>> >> >None of these ideas are very clean. >> >> >> >>> >> > >> >> >> >>> >> >Any other ideas? I tried setting up a profile to only >> >>compile >> >> >> >>>these >> >> >> >>> >> >sources when selected, but that turned out not to work. >> >> >> >>> >> > >> >> >> >>> >> >-Michael Starch >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>