From user-return-33853-archive-asf-public=cust-asf.ponee.io@nutch.apache.org  Fri Feb  9 21:45:20 2018
Return-Path: <user-return-33853-archive-asf-public=cust-asf.ponee.io@nutch.apache.org>
X-Original-To: archive-asf-public@eu.ponee.io
Delivered-To: archive-asf-public@eu.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by mx-eu-01.ponee.io (Postfix) with ESMTP id 3019D180654
	for <archive-asf-public@eu.ponee.io>; Fri,  9 Feb 2018 21:45:20 +0100 (CET)
Received: by cust-asf.ponee.io (Postfix)
	id 1F9A4160C5E; Fri,  9 Feb 2018 20:45:20 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 3CC47160C4C
	for <archive-asf-public@cust-asf.ponee.io>; Fri,  9 Feb 2018 21:45:19 +0100 (CET)
Received: (qmail 93283 invoked by uid 500); 9 Feb 2018 20:45:16 -0000
Mailing-List: contact user-help@nutch.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@nutch.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@nutch.apache.org>
List-Post: <mailto:user@nutch.apache.org>
List-Id: <user.nutch.apache.org>
Reply-To: user@nutch.apache.org
Delivered-To: mailing list user@nutch.apache.org
Received: (qmail 92987 invoked by uid 99); 9 Feb 2018 20:45:16 -0000
Received: from ui-eu-01.ponee.io (HELO localhost) (176.9.59.70)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Feb 2018 20:45:16 +0000
x-ponymail-agent: PonyMail Composer/0.2
References: <77BEE181-F2B8-42FE-888D-46520B1E85AE@zion.com> <CAGaRif17HGkPfMmMGfwA1c5yeC+YxNf4q2O+3ti+N6-OBRzqpw@mail.gmail.com>
In-Reply-To: <77BEE181-F2B8-42FE-888D-46520B1E85AE@zion.com>
Message-ID: <pony-b7497055821405926d63668ab1112e0f108e2346-748aeb15238347934f3a9c031610761486fc3c0e@user.nutch.apache.org>
Subject: Re: NUTCH-1129, Any23, microdata parsing, indexing, and extraction?
x-ponymail-sender: b7497055821405926d63668ab1112e0f108e2346
Date: Fri, 09 Feb 2018 20:45:12 -0000
From: Lewis John McGibbney <lewismc@apache.org>
Content-Type: text/plain; charset="iso-8859-1"
X-Mailer: LuaSocket 3.0-rc1
MIME-Version: 1.0
To: <user@nutch.apache.org>

Hi David,
We are in the process of releasing Any23 2.2, this will include the fix.
We can then come back to Nutch and make the upgrade and you should be all set.
Hopefully this will be achieved within around 72hrs. In the meantime, you can clone, build and deploy Any23 master. This will do the trick.
Lewis

On 2018/02/09 07:31:10, David Ferrero <david.ferrero@zion.com> wrote: 
> Thank you for this information. Since this is very much related to Any23 and microdata parsing, I’m going to ask what I believe is a related question but keep this same thread so it will be organized in one place:
> 
> I noticed a lot of job boards such as dice.com <http://dice.com/>, monster.com <http://monster.com/>, etc use http://schema.org/JobPosting <http://schema.org/JobPosting> information, however many seem to use <script type="application/ld+json”>…</script> rather than RDF.
> Summer 2017, Google announced structured data guidance for Jobs:
> https://developers.google.com/search/docs/data-types/job-posting <https://developers.google.com/search/docs/data-types/job-posting>
> and a testing tool to validate your HTML: https://search.google.com/structured-data/testing-tool
> I verified a few sample listings on the above mentioned job boards on google’s testing-tool and they validate OK.
> 
> So after looking at http://any23.apache.org/getting-started.html <http://any23.apache.org/getting-started.html> for the supported extractors, I see Any23 mentions it supports JSON+LD input, so I added this to nutch-site.xml to override the same property in nutch-default.xml:
> 
> <property>
>     <name>any23.extractors</name>
>     <value>html-microdata,html-embedded-jsonld,rdf-jsonld</value>
>     <description>Comma-separated list of Any23 extractors (a list of extractors is available here: http://any23.apache.org/getting-started.html)</description>
> </property>
> 
> I expected to see additional information from nutch parsechecker after adding the jsonld extractors, however I see NO changes to Any23-Triples microdata parsed. 
> 
> What might I be doing wrong?
> 
> > On Feb 8, 2018, at 11:17 AM, lewis john mcgibbney <lewismc@apache.org> wrote:
> > 
> > Hi David,
> > Answers inline
> > 
> > On Thu, Feb 8, 2018 at 9:19 AM, <user-digest-help@nutch.apache.org> wrote:
> > 
> >> 
> >> From: David Ferrero <david.ferrero@zion.com>
> >> To: user@nutch.apache.org
> >> Cc:
> >> Bcc:
> >> Date: Thu, 8 Feb 2018 10:19:52 -0700
> >> Subject: NUTCH-1129, Any23, microdata parsing, indexing, and extraction?
> >> Pull request #205 was recently merged into master branch for Nutch 1.x in
> >> fulfillment of NUTCH-1129 "microdata for Nutch 1.x"
> >> 
> >> I am new to nutch and solr and have just started crawling and indexing a
> >> few select websites. Using the built in html parsing/indexing, I am getting
> >> searchable fields like url, content, host, sometimes a title, and a few
> >> other indexing related fields like digest, boost, segment, and tstamp. That
> >> said, I realized very quickly that I need better results. While exploring
> >> the source of the website, I noticed references to schema.org and get
> >> excited by what I see. That’s how I stumbled upon NUTCH-1129.
> >> 
> >> I’ve built apache-nutch-1.15-SNAPSHOT which includes Any23 parser/indexer.
> >> 
> > 
> > Excellent.
> > 
> > 
> >> 
> >> Q: Now what?  How do I gain Any23 microdata parsing / indexing
> >> capabilities introduced by NUTCH-1129?
> >> Q: Do I replace parse-(html | tika)|index-(basic | anchor) in
> >> plugin.includes with something like parse-(html | tika |
> >> any23)|index-(basic | anchor | any23)
> >> 
> > 
> > No, you just add 'any23' to the list of plugins within the plugin.includes
> > property of nutch-site.xml
> > 
> > 
> >> Q: How do I expose the discovered microdata structure / items to end-user
> >> such as Solr? For example, what are the microdata items and do I need to
> >> map them to Solr in solrindex-mapping.xml?
> >> 
> > 
> > OK, so current configuration for the Any23 plugin, is to store extracted
> > structured data markup in the Nutch Metadata object with a key "
> > Any23-Triples". You can locate it using something like the ParserChekcer
> > tool provided via the 'nutch' script. Liekwise you can also locate it, as a
> > representation of what would be indexed, by using the IndexerChecker
> > tooling also provided within the 'nutch' script.
> > 
> > An example would be as follows, data is now indexed as follows (example
> > after crawling https://smartive.ch/jobs):
> > 
> > 
> >          "structured_data": [
> >            {
> >              "node": "<https://smartive.ch/jobs>",
> >              "value": "\"IE-edge,chrome=1\"@de",
> >              "key": "<http://vocab.sindice.net/any23#X-UA-Compatible>",
> >              "short_key": "X-UA-Compatible"
> >            },
> >            {
> >              "node": "<https://smartive.ch/jobs>",
> >              "value": "\"Wir sind smartive \\u2014 eine dynamische,
> > innovative Schweizer Webentwicklungsagentur. Die Realisierung
> > zeitgem\\u00E4sser Webl\\u00F6sungen geh\\u00F6rt genauso zu unserer
> > Passion, wie die konstruktive Zusammenarbeit mit unseren Kundinnen und
> > Kunden.\"@de",
> >              "key": "<http://vocab.sindice.net/any23#description>",
> >              "short_key": "description"
> >            },
> >            {
> >              "node": "<https://smartive.ch/jobs>",
> >              "value": "\"width=device-width, initial-scale=1,
> > shrink-to-fit=no\"@de",
> >              "key": "<http://vocab.sindice.net/any23#viewport>",
> >              "short_key": "viewport"
> >            },
> >            {
> >              "node": "<https://smartive.ch/jobs>",
> >              "value": "\"width=device-width,initial-scale=1\"@de",
> >              "key": "<http://vocab.sindice.net/any23#viewport>",
> >              "short_key": "viewport"
> >            },
> >            {
> >              "node": "<https://smartive.ch/jobs>",
> >              "value": "\"ie=edge\"@de",
> >              "key": "<http://vocab.sindice.net/any23#x-ua-compatible>",
> >              "short_key": "x-ua-compatible"
> >            }
> >          ],
> > 
> > 
> > Note from above, that the 'predicate' key field is very useful for quickly
> > filtering through, for example, Hotel Ratings, or something similar.
> > 
> > 
> >> 
> >> I’d also be interested to learn how to point at a specific URL and see how
> >> nutch sees the microdata (best case), then learn how to leverage this into
> >> nutch and finally into solr.
> >> 
> >> 
> > See the tooling for ParserChecker and IndexerChecker as explained above.
> > Any further question, please let me know.
> > Lewis
> 
>