lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Weiss <dawid.we...@gmail.com>
Subject Re: The Lucene Solr Gradle Build Game plan
Date Fri, 11 Oct 2019 18:28:15 GMT
Sure. Try and see if you can make it work. It is just about the only thing
that still needs to be done, the rest works like a charm.

On Fri, Oct 11, 2019, 20:20 Cassandra Targett <casstargett@gmail.com> wrote:

> Sorry for the delay getting back to you Dawid. That problem is actually
> because of the Asciidoctor version. The jekyll-asciidoc plugin will install
> Asciidoctor if it is not already installed, and it installs a version where
> the way the links are constructed is different and breaks our validation.
>
> More details about why this happens (if you’re curious) is in
> https://issues.apache.org/jira/browse/SOLR-12786?focusedCommentId=16622115&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16622115
>
> Both the Jenkins script used for Jenkins Ref Guide jobs
> (./dev-tools/scripts/jenkins.build.ref.guide.sh) and the Ref Guide README
> (./solr/solr-ref-guide/README.adoc) show examples of how to make sure the
> right Asciidoctor version is installed - the easiest is to install the
> Asciidoctor gem version we want first. Let me see if I can insert a line to
> install before the jekyll-asciidoc gem and see if that fixes it.
>
> Cassandra
> On Oct 10, 2019, 2:22 AM -0500, Dawid Weiss <dawid.weiss@gmail.com>,
> wrote:
>
> Ping, ping. I wanted to finalize the build of solr ref guide since I
> started it. Almost everything is working on the branch but I can't get
> minor differences to work and I believe they're due to a different
> jekyll version (than that mentioned in the docs).
>
> Specifically, the invalid links are because of asciidoc sections like
> this (in the processed resource-and-plugin-loading.adoc):
>
> === solr_home/lib
>
> In bare bones html (pure asciidoctor) this gets emitted as:
>
> <h3 id="solr_home-lib">solr_home/lib</h3>
>
> but when compiled via jekyll this becomes:
>
> <h3 id="solr_homelib">solr_home/lib</h3>
>
> which I can't really explain.
>
> Cassandra what's the exact version of jekyll that runs the compilation
> that is working for you?
>
> D.
>
> On Fri, Oct 4, 2019 at 11:42 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
> Seems like pygments is to blame for the python requirement... I didn't
> check but there seem to be ruby-only
> highlighters for jekyll as well:
>
> https://jekyll-windows.juthilo.com/3-syntax-highlighting/
>
> On Fri, Oct 4, 2019 at 11:39 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
> Hi Cassandra,
>
> Apologies this took so long -- I wasn't familiar with these
> site-generation tools and the whole ecosystem is rather... fragile :)
> After a few attempts at using gradle plugins I eventually leaned
> towards using asciidoctor and jekyll explicitly (so that we know which
> versions are being used and don't have to rely on dependencies).
>
> I got bare bone html checking working, PDF generation working and site
> generation working although the final link check currently fail for me
> with a bunch of errors. This works for me on Windows... on Linux I get
> site-generation generate a strange error from within jekyll:
>
> Conversion error: Jekyll::AsciiDoc::Converter encountered an error
> while converting 'about-filters.adoc':
> Bad file descriptor - /usr/bin/python2
>
> I could install python but I don't see why it'd need it. Perhaps there
> is something in the docs that would avoid using python altogether but
> I haven't had the time to look into it.
>
> Please feel free to check out the jira/SOLR-13452_gradle_7_refguide
> branch and try to run:
>
> ./gradlew -p solr/solr-ref-guide buildPdf buildSite
>
> There is a lot of room for improvement -- from property substitution,
> through how the "tools" are handled at the moment to task naming but I
> left this for the future. The initial step would be probably to get
> the site generation running on Linux/ Macs but I'd gladly hand it over
> back to you -- I can help with Gradle but a the rest of those tools
> are a mistery to me.
>
> Dawid
>
> On Fri, Sep 27, 2019 at 7:53 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
>
> No problem. I will get it to work entirely, but not before next week - I
> am away for the weekend.
>
> Dawid
>
> On Fri, Sep 27, 2019, 16:17 Cassandra Targett <casstargett@gmail.com>
> wrote:
>
>
> Thanks Dawid for working on this! I’ve been a bit swamped the last couple
> of days but will take a look today at what you’ve been able to do so far
> and see where we might need to go from here.
>
> Cassandra
> On Sep 26, 2019, 7:25 AM -0500, Dawid Weiss <dawid.weiss@gmail.com>,
> wrote:
>
> I agree. Although I also understand the concern of trying to merge the
> changes while we're in the transition period... it'd be hell. I'd say
> move as much stuff as possible with the current folder structure (and
> ignore what cannot be ported easily) then switch as soon as possible
> to gradle and hack the old cruft with a chainsaw...
>
> D.
>
> On Thu, Sep 26, 2019 at 2:13 PM Erick Erickson <erickerickson@gmail.com>
> wrote:
>
>
> Of course I’ll completely defer to Dawid and Mark (well and anybody else
> actually, you know, doing _work_), but just can’t resist chiming in ;).
>
> My vote would be to “do it the Gradle way”. Yes, it’s a PITA to learn new
> stuff and I won’t like it. Tough. I see no reason to carry a bunch of cruft
> around because “that the way we always did it”.
>
> If we lose functionality, that’s a different discussion, starting with “do
> we need that functionality". But jumping through hoops and having to
> maintain that awkwardness forever going forward just because we forced the
> Ant structure on Gradle strikes me as a poor trade off.
>
> That said, I’m not doing the work so I really have no vote. But don’t
> strain to do it the old way on my account ;)
>
> Erick
>
> P.S. Thanks Dawid for jumping in!
>
> On Sep 26, 2019, at 3:57 AM, Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
> I pushed it in to Lucene repo (it's on Cassandra's refguide branch
> anyway, so shouldn't interfere with anything else); seems like it's in
> better shape than previous code anyway (those questions I asked about
> the nature of the gradle port still hold though).
>
> I got as far as building initial bare-bones HTML.
>
> .\gradlew -p solr\solr-ref-guide clean bareBonesHtmlValidation
>
> I don't know anything about the pipeline involved (asciidoctor, etc.)
> so it's very likely some attributes will have to be corrected later
> on.
>
> Dawid
>
> On Wed, Sep 25, 2019 at 9:14 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
> I looked at the solr ref guide build and started converting it to
> Gradle but have a question to Mark (because he coordinates the
> effort).
>
> What immediately jumps into face is the decision problem -- do we want
> to emulate what ant does at the moment or do we want to clean it up
> (breaking file/ folder structure and causing incompatibility with ant
> build).
>
> I went the "compatible" way and started porting ant tasks but it's
> quite awkward. For example -- there are template properties that refer
> to ivy version properties... we could emulate/ compute these but it's
> a pain. The way the module is currently structured is also awkward -
> it'd be more natural to have a separate java project with the "tools"
> required to compile extra stuff and just reference it from the manual
> build (and this would be a plain module, not a java module). This
> would limit the need for customizing source sets, classpaths, etc.
>
> My few initial tasks syncing sources, setting up infrastructure to
> filter templates and compiling the required tools are here:
>
> https://github.com/apache/lucene-solr/compare/jira/SOLR-13452_gradle_7_refguide...dweiss:jira/SOLR-13452_gradle_7_refguide?expand=1
>
> I'll stop and wait for feedback (especially on the ivy versions issue)
> before I resume.
>
> Dawid
>
> On Wed, Sep 25, 2019 at 6:20 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
> Never mind, I've got it.
>
> D.
>
> On Wed, Sep 25, 2019 at 7:59 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
> Hi Cassandra,
>
> I’m more than happy to share more details our current build so we can
> replicate some of the above steps, but I’m stuck without a lot more basic
> Gradle skills that I don’t have time to acquire with day-job/personal life
> commitments. I put it into a separate branch so we could iterate a little
> easier, can anyone help?
>
>
> Where is this branch you made changes on? If you can point me at the
> corresponding ant code I'll try to help you out.
>
> Dawid
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Mime
View raw message