From dev-return-322241-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Tue May 15 17:16:20 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 649DC180634 for ; Tue, 15 May 2018 17:16:19 +0200 (CEST) Received: (qmail 11546 invoked by uid 500); 15 May 2018 15:16:13 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 11526 invoked by uid 99); 15 May 2018 15:16:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 May 2018 15:16:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 10619C00A9 for ; Tue, 15 May 2018 15:16:12 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.879 X-Spam-Level: ** X-Spam-Status: No, score=2.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_LIVE=1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id Zyflfkf5pJ73 for ; Tue, 15 May 2018 15:16:09 +0000 (UTC) Received: from mail-ot0-f169.google.com (mail-ot0-f169.google.com [74.125.82.169]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 2D8315FB29 for ; Tue, 15 May 2018 15:16:08 +0000 (UTC) Received: by mail-ot0-f169.google.com with SMTP id 15-v6so586321otn.12 for ; Tue, 15 May 2018 08:16:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=lB+KxvUvWEnNcp0/DO6ZR9UVrvCnhR+a9ETV/zP+lwQ=; b=QRgi/7jryUENbOCZ5DvTYRR8nIRwMVMuLBvNdDIfRq+gQlNRy2aqPj5KxWBf/O/T3N 7PRw/sX2DshnUJUFn5jxyRDc9ct1x8Ta2C/+X2GFEsT4gwa/WG1RqO2p7U1cdTAH81ar Wy84PUylBxXjBIpijH3eesgkZR/PuTxlsIVwlvNiiRAYD9MbAaPKbGPrVoGvsqW+fLoc mgX9rpLVKbIWaI4vXUlIaoljG717aeZv4U9Vk9vNySllzIR404oy05GkwtcEZK0HRzpt Ut5TlgMy2+tbXUFBJR5i6HyTNyhhSjA4wY2xcrNJMRr7GWYMqFEhs3qZ19LSpIB+5ODs 3moA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=lB+KxvUvWEnNcp0/DO6ZR9UVrvCnhR+a9ETV/zP+lwQ=; b=OSiTJ6Hy2tbeYem5tdUr2HkO1+EZgO+Xc0rYr8xQGx7M7xpuKie17+D/vB2bsgu7g7 VPFRymlkvnk3hmltyDs7jq51qqZ/pPOz0R5UuPDqV1mJmL33OsoBH39ZgzL1ya2TUFK/ 4jQ+J7KEEoRPMBnF6licoWZTGJEtqqTJTF58Q2ik4yxYS6U/i/5MAnC9Hq4EwVLt2Nwd +JeBzRhuATzRckGT5UVjI3Xiz/BCSYFOkYl+PG7pW1AhdZnLXC2qfIk2RaSs135i0ZNf BNaymT00jRPLZPD0oOPH1YSzMPEldr/O3HvykyvsZrc8qdjXq5Dzx+6TDuhSw4gPsrB8 k7KA== X-Gm-Message-State: ALKqPweBKL2mAqgHzB04XOWfud6m+rDKqPqJdYUKowKlvDP/KjZpPuRu YvlL/HlUhb/EkCC36JQzTRjSEfaCR/Tt80SLaU91pA== X-Google-Smtp-Source: AB8JxZrl5wt9rBKTydfVa20GD8wEpuasjF8CrEfBdbOaIGCkFKVZlJChuE5O5vgaoop4iEkUF1/bufQtUez8NStcARY= X-Received: by 2002:a9d:56f3:: with SMTP id b48-v6mr10376125otj.175.1526397366562; Tue, 15 May 2018 08:16:06 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Cassandra Targett Date: Tue, 15 May 2018 10:15:55 -0500 Message-ID: Subject: Re: Solr Ref Guide builds, precommit, and a plea To: dev@lucene.apache.org Content-Type: multipart/alternative; boundary="000000000000d1f3e3056c40144b" --000000000000d1f3e3056c40144b Content-Type: text/plain; charset="UTF-8" Just to close the loop (I was out for a couple days)....I hear you that linking is a bit confusing, let me try to explain a bit and hopefully it will help. Nearly all the rules are there because of the PDF version which is the only official release of the Ref Guide. To generate the PDF, every page of the Guide is assembled into one massive page and then converted to PDF. If you think about an online corollary to that - if we put the whole Guide into one single HTML page - then it should become clear why every single link between pages needs to be unique. If you want to link to another page, the pattern is <>. Someday we hopefully won't have to do the repetition there (the "page-name.adoc#page-name" part), but since it's unintuitive, one of the validation rules checks to catch cases where people may have missed doing it. Here again if we think about one single massive page of the whole Ref Guide as the paradigm, "page-name.adoc" doesn't exist in that scenario - only "SolrRefGuide-all.adoc" does - so the only thing left to reference a link is the anchor. So, you must have an anchor in the link definition or the one single page won't know where within itself to link. It would be nice if the PDF could be smarter about it without the unintuitive pattern, but it's not yet. The other issue you ran into was that one page was an orphan (had no parent). We don't allow that because pages shouldn't be added until they have a place in the overall page hierarchy. The assumption is that it only happens accidentally and an orphan should be corrected as soon as possible, so build/precommit fails on that case. All these validation rules are in place to try to find common mistakes with the Ref Guide pages and ensure what comes out of the builds is as close to release-able as possible without human page-by-page review (which we used to need to do). I have no doubt there is room for improvement to all these processes, so contributions are welcome if there's a way to make it easier/clearer. On Thu, May 10, 2018 at 1:46 PM Joel Bernstein wrote: > I see that the ref-guide build was also failing. When I made the first fix > to ref-guide build I didn't see any errors for a while figured that build > was now working. But then a new set of errors were generated. I didn't > realize that the ref-guide fails in stages. > > I think in general I'm finding all the strictness in the refguide hard > figure out. Things like internal linking have taken a long time to get > right even though they work fine when viewed on github, the build often > still fails due to the links > > > > Joel Bernstein > http://joelsolr.blogspot.com/ > > On Thu, May 10, 2018 at 2:33 PM, Joel Bernstein > wrote: > >> Ah, just saw all the errors caused by the refguide change that I made. I >> hadn't realized that precommit had been setup for refguide changes. I >> apologize for the noise this caused. I originally got one error reported >> for the ref guide build and fixed that one, and figured I was done. Next >> time I'll run precommit before pushing out ref guide changes. >> >> Joel Bernstein >> http://joelsolr.blogspot.com/ >> >> On Thu, May 10, 2018 at 10:30 AM, Cassandra Targett >> wrote: >> >>> There were some recent changes to the way the Solr Ref Guide gets built >>> made about a month ago, but stuff moves fast throughout the project so I >>> suspect it's likely a few have missed some details. >>> >>> The first, and most important, change is now when you run "ant >>> precommit", internal (page to page) & javadoc links in the Ref Guide are >>> checked and validated. This used to only run when you specifically built >>> the Ref Guide - now it happens with precommit. This was all done with >>> SOLR-12134. >>> >>> This means that if you didn't set up your env to build the HTML and you >>> don't want to build the PDF before editing docs, you don't have to. Run >>> precommit like you're already supposed to for every commit. It will fail if >>> your doc edits are messed up. >>> >>> This was one step in the process of merging the Ref Guide build with the >>> main build. There's more to do, but I don't have a concrete plan in mind. >>> >>> The second new thing that's worth mentioning is that the Ref Guide now >>> uses variables in place of "hard coded" version numbers for 3rd party >>> components. These variables pull their values from ivy-versions.properties >>> so they are accurate for each release without human intervention. This >>> includes the versions of Tika, ZooKeeper, Log4J, OpenNLP, Velocity, Commons >>> Codec, and Dropwizard today - more can be added if they are needed. >>> >>> Finally - and this likely applies to more than just the Ref Guide - >>> let's try to use branches for big stuff [1]. Branches are cheap and if >>> things go awry you don't stall everyone trying to use precommit for 12-18 >>> hours or more. We can set up Jenkins jobs for a branch if you want it, but >>> master really isn't the place for stuff that's not done yet, hasn't been >>> tested locally, and in a state where it's kinda broken, even if it's "just >>> docs". It also makes it easier for all of us to collaborate with each >>> other, which is never a bad thing. >>> >>> Hope this info helps - >>> Cassandra >>> >>> [1] And by big, I mean things on the order of adding an entire new >>> section of multiple new pages or reorganizing content in multiple pages, >>> particularly when you know it's going to take several weeks or months for >>> you to finally achieve your vision. >>> >> >> > --000000000000d1f3e3056c40144b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Just to close the loop (I was out for a couple days)....I = hear you that linking is a bit confusing, let me try to explain a bit and h= opefully it will help.=C2=A0

Nearly all the rules are th= ere because of the PDF version which is the only official release of the Re= f Guide. To generate the PDF, every page of the Guide is assembled into one= massive page and then converted to PDF. If you think about an online corol= lary to that - if we put the whole Guide into one single HTML page - then i= t should become clear why every single link between pages needs to be uniqu= e.

If you want to link to another page, the pattern is &= lt;<page-name.adoc#page-name,Page Name>>. Someday we hopefully won= 't have to do the repetition there (the "page-name.adoc#page-name&= quot; part), but since it's unintuitive, one of the validation rules ch= ecks to catch cases where people may have missed doing it.

Here again if we think about one single massive page of the whole = Ref Guide as the paradigm, "page-name.adoc" doesn't exist in = that scenario - only "SolrRefGuide-all.adoc" does - so the only t= hing left to reference a link is the anchor. So, you must have an anchor in= the link definition or the one single page won't know where within its= elf to link. It would be nice if the PDF could be smarter about it without = the unintuitive pattern, but it's not yet.

The= other issue you ran into was that one page was an orphan (had no parent). = We don't allow that because pages shouldn't be added until they hav= e a place in the overall page hierarchy. The assumption is that it only hap= pens accidentally and an orphan should be corrected as soon as possible, so= build/precommit fails on that case.

All the= se validation rules are in place to try to find common mistakes with the Re= f Guide pages and ensure what comes out of the builds is as close to releas= e-able as possible without human page-by-page review (which we used to need= to do).

I have no doubt there is room for improve= ment to all these processes, so contributions are welcome if there's a = way to make it easier/clearer.

On Thu, May 10, 2018 at 1:46 PM Joel Bernstein <joelsolr@gmail.com>= wrote:
I see that= the ref-guide build was also failing. When I made the first fix to ref-gui= de build I didn't see any errors for a while figured that build was now= working. But then a new set of errors were generated. I didn't realize= that the ref-guide fails in stages.

I think in general = I'm finding all the strictness in the refguide hard figure out. Things = like internal linking have taken a long time to get right even though they = work fine when viewed on github, the build often still fails due to the lin= ks




On Thu, May 10, 2018 at 2:33 PM, Joel Bernst= ein <joelsolr@gmail.com> wrote:
Ah, just saw all the errors caused by the refguide = change that I made. I hadn't realized that precommit had been setup for= refguide changes. I apologize for the noise this caused. I originally got = one error reported for the ref guide build and fixed that one, and figured = I was done. Next time I'll run precommit before pushing out ref guide c= hanges.


On Thu, May 10, 2018 at 10:30 AM, Cassandra = Targett <ctargett@apache.org> wrote:
There were some recent changes to the way th= e Solr Ref Guide gets built made about a month ago, but stuff moves fast th= roughout the project so I suspect it's likely a few have missed some de= tails.

The first, and most important, change is now when= you run "ant precommit", internal (page to page) & javadoc l= inks in the Ref Guide are checked and validated. This used to only run when= you specifically built the Ref Guide - now it happens with precommit. This= was all done with SOLR-12134.

This means that if = you didn't set up your env to build the HTML and you don't want to = build the PDF before editing docs, you don't have to. Run precommit lik= e you're already supposed to for every commit. It will fail if your doc= edits are messed up.

This was one step in the pro= cess of merging the Ref Guide build with the main build. There's more t= o do, but I don't have a concrete plan in mind.

The second new thing that's worth mentioning is that the Ref Guide no= w uses variables in place of "hard coded" version numbers for 3rd= party components. These variables pull their values from ivy-versions.prop= erties so they are accurate for each release without human intervention. Th= is includes the versions of Tika, ZooKeeper, Log4J, OpenNLP, Velocity, Comm= ons Codec, and Dropwizard today - more can be added if they are needed.

Finally - and this likely applies to more than just t= he Ref Guide - let's try to use branches for big stuff [1]. Branches ar= e cheap and if things go awry you don't stall everyone trying to use pr= ecommit for 12-18 hours or more. We can set up Jenkins jobs for a branch if= you want it, but master really isn't the place for stuff that's no= t done yet, hasn't been tested locally, and in a state where it's k= inda broken, even if it's "just docs". It also makes it easie= r for all of us to collaborate with each other, which is never a bad thing.=

Hope this info helps -
Cassandra
<= div>
[1]=C2=A0And by big, I mean things on= the order of adding an entire new section of multiple new pages or reorgan= izing content in multiple pages, particularly when you know it's going = to take several weeks or months for you to finally achieve your vision.=C2=A0


--000000000000d1f3e3056c40144b--