manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Unexpected job status encountered
Date Thu, 27 Dec 2018 12:30:41 GMT
Thanks for looking harder into this!

The credential encoding in httpcomponents/httpclient has been problem free
as far as I have seen, so if you determine that that's the issue I am sure
it will be news to a lot of people.  But by using the wire logging you
should be able to see the headers, including the encoded credentials, and
compare/contrast what's working and what's not pretty easily.

Karl


On Thu, Dec 27, 2018 at 5:42 AM Erlend Garåsen <e.f.garasen@usit.uio.no>
wrote:

>
> It wasn't necessary to deal with tools like tcpdump etc. Adding the
> following to the logging.xml did the trick:
> <Logger name="org.apache.http.wire" level="debug" additivity="false">
>   <AppenderRef ref="MyFile" />
> </Logger>
>
> So now I know what's going on. Bad credentials:
>
> DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 <<
> "HTTP/1.1 401 Unauthorized[\r][\n]"
>
> Strange, because connection is working according to the Solr Output
> Connector. I'll double-check whether the Solr server has another
> password for index writing (path "/solr/uio/update/extract"). Or maybe
> we have an encoding issue with the password since it's long and contains
> special characters.
>
> --8<--
>
> DEBUG 2018-12-27T11:18:41,591 (Thread-1508) - http-outgoing-2 >>
> "</div><!-- container --> [\n]"
> DEBUG 2018-12-27T11:18:41,591 (Thread-1508) - http-outgoing-2 >>
> "</body> [\n]"
> DEBUG 2018-12-27T11:18:41,591 (Thread-1508) - http-outgoing-2 >>
> "</html>[\n]"
> DEBUG 2018-12-27T11:18:41,591 (Thread-1508) - http-outgoing-2 >> "[\n]"
> DEBUG 2018-12-27T11:18:41,591 (Thread-1508) - http-outgoing-2 >> "[\r][\n]"
> DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >>
> "2f[\r][\n]"
> DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >> "[\r][\n]"
> DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >>
> "******************[\r][\n]"
> DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >> "[\r][\n]"
> DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >>
> "0[\r][\n]"
> DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >> "[\r][\n]"
> DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 <<
> "HTTP/1.1 401 Unauthorized[\r][\n]"
> DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << "Date:
> Thu, 27 Dec 2018 10:18:41 GMT[\r][\n]"
> DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 <<
> "Server: Apache/2.4.6 (Red Hat Enterprise Linux)
> OpenSSL/1.0.2k-fips[\r][\n]"
> DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 <<
> "WWW-Authenticate: Basic realm="Solr"[\r][\n]"
> DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 <<
> "Content-Length: 381[\r][\n]"
> DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 <<
> "Keep-Alive: timeout=10, max=100[\r][\n]"
> DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 <<
> "Connection: Keep-Alive[\r][\n]"
> DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 <<
> "Content-Type: text/html; charset=iso-8859-1[\r][\n]"
> DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << "[\r][\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 <<
> "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">[\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 <<
> "<html><head>[\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 <<
> "<title>401 Unauthorized</title>[\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 <<
> "</head><body>[\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 <<
> "<h1>Unauthorized</h1>[\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 <<
> "<p>This server could not verify that you[\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << "are
> authorized to access the document[\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 <<
> "requested.  Either you supplied the wrong[\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 <<
> "credentials (e.g., bad password), or your[\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 <<
> "browser doesn't understand how to supply[\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << "the
> credentials required.</p>[\n]"
> DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 <<
> "</body></html>[\n]"
>  WARN 2018-12-27T11:18:41,599 (Worker thread '48') - IO exception during
> indexing https://www.journals.uio.no/index.php/Dialogia: null
> org.apache.http.client.ClientProtocolException
>
> Erlend
>
> On 21/12/2018 16:39, Karl Wright wrote:
> > I'll have a look as time permits, but it won't be for a couple of weeks.
> > Usually it's best to debug with http wire debugging, not packet
> > captures.  I'm not an HTTP expert so it may be better to bring this up
> > to the HttpComponents/HttpClient list, not ManifoldCF.
> >
> > Karl
> >
> >
> > On Fri, Dec 21, 2018 at 8:54 AM Erlend Garåsen <e.f.garasen@usit.uio.no
> > <mailto:e.f.garasen@usit.uio.no>> wrote:
> >
> >
> >     I tried to configure preemptive authentication by modifying the
> >     HttpPoster class, but I still get the same errors. Then I ran the
> >     following command in order to analyze the traffic:
> >     tcpdump -w /tmp/chatter.dmp -s 0 -i ens192 -X host
> >     solr-test01.uio.no <http://solr-test01.uio.no>
> >
> >     I'm not an expert in reading such outputs, but I can send you the
> dump
> >     file, Karl. I don't want to let the file be available for others,
> even
> >     though it's from our test environment.
> >
> >     Erlend
> >
> >     On 13/12/2018 15:09, Karl Wright wrote:
> >     > Ok, thanks, I misunderstood where the SSL error was coming from.
> The
> >     > Solr connection is what is complaining.  Do you see "Connection
> >     working"
> >     > for your output connection?  Please forgive me if you already
> answered
> >     > this; I didn't note it in your response.  If you see that, then the
> >     > connector was able to talk to your Solr "ping" handler, which
> >     would mean
> >     > that the SSL configuration is right but there's something else
> >     about the
> >     > connection that we would have to figure out and deal with.
> >     >
> >     > Karl
> >     >
> >     >
> >     > On Thu, Dec 13, 2018 at 8:58 AM Erlend Garåsen
> >     <e.f.garasen@usit.uio.no <mailto:e.f.garasen@usit.uio.no>
> >     > <mailto:e.f.garasen@usit.uio.no <mailto:e.f.garasen@usit.uio.no>>>
> >     wrote:
> >     >
> >     >     On 13/12/2018 14:26, Karl Wright wrote:
> >     >     > This is SSL.  Did you add the server's cert to the web
> >     connector's
> >     >     > keystore?  Or, if not, add a "trust all" rule?
> >     >
> >     >     Thanks for the reply, Karl.
> >     >
> >     >     Yes, the root certificate was added both in the repository
> >     connection
> >     >     and for the Solr connector.
> >     >
> >     >     I checked the "trust all" option and restarted the job. Same
> >     problem.
> >     >
> >     >     BTW, I get a lot of 200s, so MCF is able to fetch the URLs.
> >     >
> >     >     I have attached a screenshot of what I can see in the simple
> >     history
> >     >     report.
> >     >
> >     >     Erlend
> >     >
> >
>
>

Mime
View raw message