From java-user-return-63523-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Thu Jan 18 17:04:43 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 156F9180654 for ; Thu, 18 Jan 2018 17:04:43 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 05885160C36; Thu, 18 Jan 2018 16:04:43 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 4B717160C2B for ; Thu, 18 Jan 2018 17:04:42 +0100 (CET) Received: (qmail 29307 invoked by uid 500); 18 Jan 2018 16:04:40 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 29295 invoked by uid 99); 18 Jan 2018 16:04:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jan 2018 16:04:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 13FF61A732C for ; Thu, 18 Jan 2018 16:04:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.379 X-Spam-Level: X-Spam-Status: No, score=0.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id mXZf7bFEeTSa for ; Thu, 18 Jan 2018 16:04:35 +0000 (UTC) Received: from mail-lf0-f52.google.com (mail-lf0-f52.google.com [209.85.215.52]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 01C855F3F0 for ; Thu, 18 Jan 2018 16:04:34 +0000 (UTC) Received: by mail-lf0-f52.google.com with SMTP id x196so7306017lfd.12 for ; Thu, 18 Jan 2018 08:04:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=zkAAt1WruElhWKiQG7q/TO2Fl7FPyPHaVxVDhbY0BS0=; b=RPtzFakM32HZKMLo5X8/sOQ7RDv0GftZCKyPxg23bZxEYPXaCLdCAerWIGwo17JMdD jXF8R5ovue+OZYeZMNZ+uhdf9WTKv8Q2cLtXkHR+2yvStlVyEo5NPAmQrQq+1HHB07He 3jyHlqGhad02C0DOWE3gLrmNww9AgdeIGPjpYb+c+RHGaxr3O7Oxv1rkNrX03/0dloe+ HkTvn7uwnmfTzrguq+zKsE3oo6K/qUrhDHDruJTExtLPlgoUJ+cJ3IvwVbNeF7pJ4wmQ RJuNK8DxreJQM8AC1APpn5DnkB0QYcOPuYR1Rz34/NBYFGCx7h6Bkfsd9DeskrXVahYV SZZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=zkAAt1WruElhWKiQG7q/TO2Fl7FPyPHaVxVDhbY0BS0=; b=LL1FK1lq6AJ9jUqR55abBglH2yFaF2cCQGSC2NB7B4l7VxHQw2hgsXYKemCqHBJ2Z6 40kTtcTqAM88GwppV/zoO2DNAiFnshRguRwPdq64DQwmZ/C/x3xGNfaROX79yhDJec4F n4spBik6v0xjNJsk5xKkh6eTGHlYJQTTLasJlfrynTT8pAJhfzUJKhN6+X84R5OcIxCf t7KXZ9odgBVu/OudLQhHZtUwh4N2NPrTOtO6xF1VcxlRp9TzZz/Obfc4e1B06rIWf0PP lU/a7rJC2aWbisqLx113fYXjIqzHDtGUwfRdO90i1TZ1Vpp44sj87BriueAq0gLR1SQL s23g== X-Gm-Message-State: AKGB3mJ4aYq5vE6j2SvncQUna13OTizbgzxqdMhlQ8JMqBmYRaEql1tF yGNV+cP7s6jf4kt4BpWD5x5ezRMIkmAA+4KyW1XlOw== X-Google-Smtp-Source: ACJfBosHKg4/VQM7YPY7B031vc8PLVEn8TVwEVWggkhdDKauxkJBv9vhO2y5rj5mYyq1szL2wLpws/Uvt8EKKbHeDU4= X-Received: by 10.46.22.71 with SMTP id 7mr27289694ljw.21.1516291473484; Thu, 18 Jan 2018 08:04:33 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.84.65 with HTTP; Thu, 18 Jan 2018 08:03:52 -0800 (PST) In-Reply-To: References: From: Erick Erickson Date: Thu, 18 Jan 2018 08:03:52 -0800 Message-ID: Subject: Re: indexing performance 6.6 vs 7.1 To: java-user Content-Type: text/plain; charset="UTF-8" My first question is always "are you running the Solr CPUs flat out?". My guess in this case is that the indexing client is the same and the problem is in Solr, but it's worth checking whether the clients are just somehow not delivering docs as fast as they were before. My suspicion is that the indexing client hasn't changed, but it's worth checking. Best, Erick On Thu, Jan 18, 2018 at 2:23 AM, Rob Audenaerde wrote: > Hi all, > > We recently upgraded from Lucene 6.6 to 7.1. We see a significant drop in > indexing performace. > > We have a-typical use of Lucene, as we (also) index some database tables > and add all the values as AssociatedFacetFields as well. This allows us to > create pivot tables on search results really fast. > > These tables have some overlapping columns, but also disjoint ones. > > We anticipated a decrease in index size because of the sparse docvalues. We > see this happening, with decreases to ~50%-80% of the original index size. > But we did not expect an drop in indexing performance (client systems > indexing time increased with +50% to +250%). > > (Our indexing-speed used to be mainly bound by the speed the Taxonomy could > deliver new ordinals for new values, currently we are investigating if this > is still the case, will report later when a profiler run has been done) > > Does anyone know if this increase in indexing time is to be expected as > result of the sparse docvalues change? > > Kind regards, > > Rob Audenaerde --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org