hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <els...@apache.org>
Subject Re: [DISCUSS] More Shading
Date Thu, 13 Apr 2017 23:46:27 GMT


Stack wrote:
> On Wed, Apr 12, 2017 at 8:22 AM, Josh Elser<elserj@apache.org>  wrote:
> ....
>
>> This makes me wonder if we could construct source jars just the same as
>> we're creating shaded jars. Google has lead me to [2][3], but I've never
>> tried either. The latter option seems to be acknowledging that the source
>> might not actually compile, but the package names would at least be correct.
>>
>> I think this would be a good early experiment which, if it does work out,
>> removes the only acknowledged "hole" in the current plan.
>>
>>
> Mighty Josh.
>
> As it happens, the 'createSources' you cite is currently in use by our
> hbase-protocol-shaded module for the specific purpose of keeping our IDE'rs
> happy; we relocate protobuf itself so we can have pb3.2 and pb2.5 on our
> CLASSPATH but IDEs don't complain because the src for the relocated pb is
> checked-in to our codebase.

Aha! Wizards are ahead of me on the path already :)

> Here's more if interested:
>
> All modules that make use of protos by convention require a 'pre-build'
> step. Generally, the pre-build is required if you add or mod .proto files.
> At pre-build we generate class files from the amended .protos and then
> check-in the product. See the README in each of our proto-carrying modules.
>
> In the hbase-protocol-shaded case, we abuse the pre-build notion so we can
> house an internal pb version, one that does not agree w/ hadoops nor w/
> that used by our CPs describing Endpoint services.  We do as follows (the
> order may not be exact in below):
>
>   * Generate class files from protos
>   * Shade the built artifact (with the createSource flag set).
>   * Unjar the artifact which has generated proto and protobuf src classes in
> it
>   * Apply a few patches to protobuf to support offheap work (stuff we need
> to push back up to pb)
>   * Overlay our current protobuf src w/ the new version in the src tree -- a
> big no-no (smile).
>
> You then check it all in....
>
> It has been suggested we undo these hokey pre-build steps and just generate
> classes from protobuf inline w/ the main build. A plugin we just figured
> makes this possible since it provides the platform-appropriate protoc
> (org.xolstice.maven.plugins). We could do the simplification currently for
> all but the hbase-protocol-shaded module because of the above jujitsu. One
> thought is that if we had a pre-build artifact as is being suggested in
> this thread, I could move the patched pb there and purge the pre-build step
> everywhere.

The xolstice plugin has been working well for me elsewhere so far. +1

I think we're speaking the same language. My realization was that, 
hopefully, we could automatically create that pre-built artifact via 
Maven magic instead of having a human copy source files around.

I think pushing this part forward with some code is the next logical 
step. Seems to be consensus about taking our known internal dependencies 
and performing this shade magic.

Don't want to stomp on your worries, Nick. I think your worries are more 
about the presentation to downstream and we're in agreement about 
isolating our internal deps with the described approach?

> Thanks,
> St.Ack
>
>
>
>> We (The HBase PMC) will have to make releases of this new artifact and vote
>>> on them. I think it will be a relatively rare event.
>>>
>>> I'd be up for doing the first cut if folks are game.
>>>
>>> St.Ack
>>>
>>>
>>> 1. URL via Sean but for committers to view only:
>>> https://reporeq.apache.org/
>>>
>>>
>> [2] https://maven.apache.org/plugins/maven-shade-plugin/shade-
>> mojo.html#createSourcesJar
>> [3] https://maven.apache.org/plugins/maven-shade-plugin/shade-
>> mojo.html#shadeSourcesContent
>>
>

Mime
View raw message