manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CONNECTORS-1533) Solr Connector is unable to ingest documents
Date Fri, 21 Sep 2018 12:46:00 GMT

    [ https://issues.apache.org/jira/browse/CONNECTORS-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623544#comment-16623544
] 

Karl Wright commented on CONNECTORS-1533:
-----------------------------------------

For your reference, here is the Lucene master version of the HttpSolrClient method we needed
to update in ModifiedHttpSolrClient to make compatible with SolrJ 7.4:

{code}
  protected HttpRequestBase createMethod(SolrRequest request, String collection) throws IOException,
SolrServerException {
    if (request instanceof V2RequestSupport) {
      request = ((V2RequestSupport) request).getV2Request();
    }
    SolrParams params = request.getParams();
    RequestWriter.ContentWriter contentWriter = requestWriter.getContentWriter(request);
    Collection<ContentStream> streams = contentWriter == null ? requestWriter.getContentStreams(request)
: null;
    String path = requestWriter.getPath(request);
    if (path == null || !path.startsWith("/")) {
      path = DEFAULT_PATH;
    }
    
    ResponseParser parser = request.getResponseParser();
    if (parser == null) {
      parser = this.parser;
    }
    
    // The parser 'wt=' and 'version=' params are used instead of the original
    // params
    ModifiableSolrParams wparams = new ModifiableSolrParams(params);
    if (parser != null) {
      wparams.set(CommonParams.WT, parser.getWriterType());
      wparams.set(CommonParams.VERSION, parser.getVersion());
    }
    if (invariantParams != null) {
      wparams.add(invariantParams);
    }

    String basePath = baseUrl;
    if (collection != null)
      basePath += "/" + collection;

    if (request instanceof V2Request) {
      if (System.getProperty("solr.v2RealPath") == null) {
        basePath = baseUrl.replace("/solr", "/api");
      } else {
        basePath = baseUrl + "/____v2";
      }
    }

    if (SolrRequest.METHOD.GET == request.getMethod()) {
      if (streams != null || contentWriter != null) {
        throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "GET can't send streams!");
      }

      return new HttpGet(basePath + path + wparams.toQueryString());
    }

    if (SolrRequest.METHOD.DELETE == request.getMethod()) {
      return new HttpDelete(basePath + path + wparams.toQueryString());
    }

    if (SolrRequest.METHOD.POST == request.getMethod() || SolrRequest.METHOD.PUT == request.getMethod())
{

      String url = basePath + path;
      boolean hasNullStreamName = false;
      if (streams != null) {
        for (ContentStream cs : streams) {
          if (cs.getName() == null) {
            hasNullStreamName = true;
            break;
          }
        }
      }
      boolean isMultipart = ((this.useMultiPartPost && SolrRequest.METHOD.POST ==
request.getMethod())
          || (streams != null && streams.size() > 1)) && !hasNullStreamName;

      LinkedList<NameValuePair> postOrPutParams = new LinkedList<>();

      if(contentWriter != null) {
        String fullQueryUrl = url + wparams.toQueryString();
        HttpEntityEnclosingRequestBase postOrPut = SolrRequest.METHOD.POST == request.getMethod()
?
            new HttpPost(fullQueryUrl) : new HttpPut(fullQueryUrl);
        postOrPut.addHeader("Content-Type",
            contentWriter.getContentType());
        postOrPut.setEntity(new BasicHttpEntity(){
          @Override
          public boolean isStreaming() {
            return true;
          }

          @Override
          public void writeTo(OutputStream outstream) throws IOException {
            contentWriter.write(outstream);
          }
        });
        return postOrPut;

      } else if (streams == null || isMultipart) {
        // send server list and request list as query string params
        ModifiableSolrParams queryParams = calculateQueryParams(this.queryParams, wparams);
        queryParams.add(calculateQueryParams(request.getQueryParams(), wparams));
        String fullQueryUrl = url + queryParams.toQueryString();
        HttpEntityEnclosingRequestBase postOrPut = fillContentStream(request, streams, wparams,
isMultipart, postOrPutParams, fullQueryUrl);
        return postOrPut;
      }
      // It is has one stream, it is the post body, put the params in the URL
      else {
        String fullQueryUrl = url + wparams.toQueryString();
        HttpEntityEnclosingRequestBase postOrPut = SolrRequest.METHOD.POST == request.getMethod()
?
            new HttpPost(fullQueryUrl) : new HttpPut(fullQueryUrl);
        fillSingleContentStream(streams, postOrPut);

        return postOrPut;
      }
    }

    throw new SolrServerException("Unsupported method: " + request.getMethod());

  }
{code}

Please compare and contrast with what is committed to the ManifoldCF code base.


> Solr Connector is unable to ingest documents
> --------------------------------------------
>
>                 Key: CONNECTORS-1533
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1533
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Lucene/SOLR connector
>    Affects Versions: ManifoldCF 2.11
>            Reporter: Julien Massiera
>            Assignee: Karl Wright
>            Priority: Major
>
> The "r69acbd9 - Fix solr connector content deletion bug" has introduced another bug :

> It is now impossible to ingest documents into Solr 7.4.0, we obtain the following error
: Error from server at http://localhost:8983/solr/FileShare: missing content stream
> The fact is, the requestWriter.getContentWriter(request) object is equal to null only
on commit requests. So the new lines of code introduced by the fix, which are based on the
test of this object, result in a null Collection<ContentStream> streams object and
so the update request is failing.
> Concerned class : org.apache.manifoldcf.agents.output.solr.ModifiedHttpSolrClient



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message