From user-return-5431-archive-asf-public=cust-asf.ponee.io@manifoldcf.apache.org Tue Sep 4 18:04:34 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id AB73B180668 for ; Tue, 4 Sep 2018 18:04:32 +0200 (CEST) Received: (qmail 3930 invoked by uid 500); 4 Sep 2018 16:04:31 -0000 Mailing-List: contact user-help@manifoldcf.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@manifoldcf.apache.org Delivered-To: mailing list user@manifoldcf.apache.org Received: (qmail 3904 invoked by uid 99); 4 Sep 2018 16:04:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Sep 2018 16:04:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 3ABF4C550F for ; Tue, 4 Sep 2018 16:04:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.501 X-Spam-Level: **** X-Spam-Status: No, score=4.501 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, KAM_BADIPHTTP=2, KAM_NUMSUBJECT=0.5, KAM_SHORT=0.001, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, T_DKIMWL_WL_MED=-0.01, T_REMOTE_IMAGE=0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=remcam-net.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Klc2WosuiTfJ for ; Tue, 4 Sep 2018 16:04:29 +0000 (UTC) Received: from mail-ua1-f65.google.com (mail-ua1-f65.google.com [209.85.222.65]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 7AE495F3AA for ; Tue, 4 Sep 2018 16:04:28 +0000 (UTC) Received: by mail-ua1-f65.google.com with SMTP id m26-v6so3318402uap.2 for ; Tue, 04 Sep 2018 09:04:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=remcam-net.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Oyhim6u/E4ERDXqp8VH4matUl6g6ea37YINIKQh+h8Q=; b=TI/ykCNoJzvkoDj6rH+2JablVzqpt5fsr0Gbh0Hx2SmZn12qvsWGdef9Gf65MSX2Ov 9gQezh8vwwQW/bXg5JMtf080SfG3nPhHAoPpJ3ZPje2CP8KVOeYKVPx51GXlNFr9cQxM vvQXdjYCHeBVKTrrFZmxuI6UKd9J6e6LYDDPLmv1E/t3wfjupvYmCUvg1toK9PTLYB7R 7dCnMkOORsDTAY7qlEhz/PR5qQUCqLfceSyUUDRgz1mtWf75GL0fAUCy9HKMj9vYQbiX +tHLWGjokuYn+PC5u5q7UA3LDRNndj5tM67nhkkl4NFaDIsnEcBXX/a/nyxEV03ctlpE dUtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Oyhim6u/E4ERDXqp8VH4matUl6g6ea37YINIKQh+h8Q=; b=UsNyLJoOhW6j/FP2jystlNTiXjO/l0ArYkGgYi0rxb2d3bGV/OCBBa4czvOdpl53/V x7kmOn5vSxvAgAuast+VYvDOHtC3bLemMUrKKRE9Cz3MdvXk87KVDQh2jWwK4rdRXDzM CYivosCWuY+bCJ72LIgVx6UJTOLhgE4R+68iOJSk62PFbhEZI4sBahbnAAq4Q11DvF8c oFfSlPPYDq+XI5YHmlj/fs2Hlu5X6RoqPsOqfGnfkovzQVe0s6Bv1uuPCUEw0iBaiz2U Lrncnj9vSTDBcf6yUaMqxVXB1zgDenuvog239pwHjvIOYaiHjHBnImC1DlgqoyF9241n MyLA== X-Gm-Message-State: APzg51AzLAY9EeIVbJ43CBjdErf/gNYxCrxWGRKpQ5GZ644NTGRmC9y7 9gfOlbgnv78vj6O37PfSbaBLp3tz3urfE+pGRx0AbXhy X-Google-Smtp-Source: ANB0VdYUU/kxgWxPr3QJYJmcYRN1Mfuy+OeW8BsIgCRDDCG36gErtIOY7Qe71l2L8N4rylBWavrwgRlDTq4Xo7psUcs= X-Received: by 2002:ab0:52a4:: with SMTP id v33-v6mr18368511uav.32.1536077067005; Tue, 04 Sep 2018 09:04:27 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:ab0:5945:0:0:0:0:0 with HTTP; Tue, 4 Sep 2018 09:04:26 -0700 (PDT) In-Reply-To: References: <21802caf4cb34bd6afa11556101498d8@FJDUKEX03.fjdefence.fs.fujitsu.com> <517470e84acc434eafc77a37038aab30@FJDUKEX03.fjdefence.fs.fujitsu.com> <6C6FECF8-51AA-4846-9C88-3F39140C24C4@francelabs.com> <94191950-8CF9-4CA6-B76B-D3C64F9C4BE7@francelabs.com> From: Steph van Schalkwyk Date: Tue, 4 Sep 2018 11:04:26 -0500 Message-ID: Subject: Re: PostgreSQL version to support MCF v2.10 To: user@manifoldcf.apache.org Content-Type: multipart/alternative; boundary="000000000000ed444b05750dcfdb" --000000000000ed444b05750dcfdb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks Karl. I'll look into that. Another note: Regarding the ES connector - I have made two additions to it and should probably diff them for inclusion after approval: 1. lowercased _id (the doc URI). 2. Removed dual "/" , e.g. "//" in the _id (I have sloppy sources, particularly IIS...) 3. Added a "url" metadata field to the ES connector (as ES 6.x does not allow accedd to _id in the schema anymore, so no copy_field etc. from _id). Hence "url". Regards, Steph *Steph van Schalkwyk* Principal, Remcam Search Engines +1.314.452. <+1+314+452+2896>2896 steph@remcam.net http://remcam.net Skype: svanschalkwyk On Tue, Sep 4, 2018 at 10:50 AM, Karl Wright wrote: > Hi Steph, I suspect that Jetty is leaking some resource, and we may need > to upgrade it. > > Karl > > > On Tue, Sep 4, 2018 at 11:26 AM Steph van Schalkwyk > wrote: > >> Olivier >> By all means. >> The only issue I have seen (totally unrelated) is with Jetty, which has >> to be restarted about once a week. Still trying to find the issue. >> I may be overly sensitive, but I suspect MCF 2.10 with Postgres10 may be >> a bit slower. I have no empiric evidence at the moment as I'm still >> delivering the project to UAT. Will keep you posted. >> Regards, >> Steph >> >> >> >> *Steph van Schalkwyk* >> Principal, Remcam Search Engines >> +1.314.452. <+1+314+452+2896>2896 steph@remcam.net http://remcam.ne= t >> Skype: svanschalkwyk >> >> >> >> On Tue, Sep 4, 2018 at 9:59 AM, Olivier Tavard < >> olivier.tavard@francelabs.com> wrote: >> >>> Hello, >>> >>> Thanks a lot for sharing your PostgreSQL configuration (sorry for the >>> late answer). I will test it soon. >>> >>> Best regards, >>> >>> >>> Olivier TAVARD >>> >>> >>> Le 23 ao=C3=BBt 2018 =C3=A0 19:20, Steph van Schalkwyk a =C3=A9crit >>> : >>> >>> >>> >>> These are the rpm installs: >>> - file:///tmp/postgres10/postgresql10-libs-10.4-1PGDG.rhel7.x86_64.rpm >>> - file:///tmp/postgres10/postgresql10-10.4-1PGDG.rhel7.x86_64.rpm >>> - file:///tmp/postgres10/postgresql10-contrib-10.4- >>> 1PGDG.rhel7.x86_64.rpm >>> - file:///tmp/postgres10/postgresql10-devel-10.4-1PGDG.rhel7.x86_64.rpm >>> - file:///tmp/postgres10/postgresql10-server-10.4-1PGDG.rhel7.x86_64.rp= m >>> >>> postgresql_version: 10 >>> postgresql_data_dir: /var/lib/pgsql/10/data >>> postgresql_bin_path: /usr/pgsql-10/bin >>> postgresql_config_path: /var/lib/pgsql/10/data >>> postgresql_daemon: postgresql-10.service >>> postgresql_packages: >>> - postgresql10-libs >>> - postgresql10 >>> - postgresql10-server >>> - postgresql10-contrib >>> # - postgresql10-devel >>> >>> postgresql_hba_entries: >>> - { type: local, database: all, user: postgres, auth_method: peer } >>> - { type: local, database: all, user: all, auth_method: peer } >>> - { type: host, database: all, user: all, address: '127.0.0.1/32', >>> auth_method: md5 } >>> - { type: host, database: all, user: all, address: '::1/128', >>> auth_method: md5 } >>> - { type: host, database: all, user: all, address: '0.0.0.0/0', >>> auth_method: md5 } >>> - { type: host, database: all, user: all, address: '::0/0', auth_method= : >>> md5 } >>> >>> postgresql_global_config_options: >>> - option: unix_socket_directories >>> value: '{{ postgresql_unix_socket_directories | join(",") }}' >>> >>> - option: standard_conforming_strings >>> value: 'on' >>> >>> - option: shared_buffers >>> value: '1024MB' >>> >>> # max_wal_size =3D (3 * checkpoint_segments) * 16MB >>> # checkpoint_segments=3D300 >>> - option: max_wal_size >>> value: '14400MB' >>> >>> - option: min_wal_size >>> value: '80MB' >>> >>> - option: maintenance_work_mem >>> value: '2MB' >>> >>> - option: listen_addresses >>> value: '*' >>> >>> - option: max_connections >>> value: '400' >>> >>> - option: checkpoint_timeout >>> value: '900' >>> >>> - option: datestyle >>> value: "iso, mdy" >>> >>> - option: autovacuum >>> value: 'off' >>> >>> # vacuum all databases every night (full vacuum on Sunday night, lazy >>> vacuum every night) >>> - name: add postgresql cron lazy vacuum >>> cron: >>> name: lazy_vacuum >>> hour: 8 >>> minute: 0 >>> job: "su - postgres -c 'vacuumdb --all --analyze --quiet'" >>> - name: add postgresql cron full vacuum >>> cron: >>> name: full_vacuum >>> weekday: 0 >>> hour: 10 >>> minute: 0 >>> job: "su - postgres -c 'vacuumdb --all --full --analyze --quiet'" >>> # re-index all databases once a week >>> - name: add postgresql cron reindex >>> cron: >>> name: reindex >>> weekday: 0 >>> hour: 12 >>> minute: 0 >>> job: "su - postgres -c 'psql -t -c \"select datname from pg_database >>> order by datname;\" | xargs -n 1 -I\"{}\" -- psql -U postgres {} -c >>> \"reindex database {};\"' " >>> >>> >>> This is how I run 2.10. >>> Been running fine for some weeks without user intervention. >>> @Karl: Any comments please? >>> Steph >>> >>> >>> >>> >> --000000000000ed444b05750dcfdb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks Karl.=C2=A0

I'll look into t= hat.

Another note:
Regarding the ES conn= ector - I have made two additions to it and should probably diff them for i= nclusion after approval:
1. lowercased _id (the doc URI).
2. Removed dual "/" , e.g. "//" in the _id (I have s= loppy sources, particularly IIS...)
3. Added a "url" me= tadata field to the ES connector (as ES 6.x does not allow accedd to _id in= the schema anymore, so no copy_field etc. from _id). Hence "url"= .

Regards,
Steph




= Steph van Schalkwyk
Principal,=C2=A0Remcam Search Engines<= /div>
+1.314.452.2896=C2=A0=C2=A0=C2=A0=C2=A0steph@remcam.n= et=C2=A0 =C2=A0htt= p://remcam.net=C2=A0Skype:=C2=A0svanschalkwyk=C2=A0

On Tue, Sep 4, 2018 at 10:50 AM, Karl Wright= <daddywri@gmail.com> wrote:
Hi Steph, I suspect that Jetty is leaking some resourc= e, and we may need to upgrade it.

Karl


On Tue, Sep 4, 2018 at 11:26 AM Steph van Schalkwyk <steph@remcam.net> wrote:=
Olivier
By all means.
The only issue I have seen (totally unrelated) is with= Jetty, which has to be restarted about once a week. Still trying to find t= he issue.
I may be overly sensitive, but I = suspect MCF 2.10 with Postgres10 may be a bit slower. I have no empiric evi= dence at the moment as I'm still delivering the project to UAT. Will ke= ep you posted.
Regards,
Steph

<= div class=3D"m_-5751334797126891156m_7150606971559475428gmail_signature" da= ta-smartmail=3D"gmail_signature">


Steph van Sch= alkwyk
Principal,=C2=A0Remcam Search Engines
+1.314.452.2896=C2=A0=C2=A0= =C2=A0=C2=A0steph@remcam.net=C2=A0 =C2=A0http://remca= m.net=C2=A0Skype:=C2=A0sv= anschalkwyk=C2=A0

On Tue, Sep 4, 2018 at 9:59 AM, Olivier Tava= rd <olivier.tavard@francelabs.com> wrote:
Hello,

Thanks a lot fo= r sharing your PostgreSQL configuration (sorry for the late answer). I will= test it soon.

Best regards,

<= br>
Olivier TAVARD


Le 23 ao=C3=BBt 2018 =C3=A0 19:20, = Steph van Schalkwyk <steph@remcam.net> a =C3=A9crit :


=
postgresql_version:= 10
postgresql_data_dir: /var/lib/pgsql/10/data
postgresql_bin_path: /usr/pgsql-10/b= in
postgresql_config_path: /var/lib/pgsql/10/data
postgresql_daemon: postgresql-10.service
post= gresql_packages:
=
- postgresql10-libs
- postgresql10
= - postgresql10-server<= /span>
- postgresql10-contrib
= # - postgresql10-devel

<= span style=3D"color:rgb(212,212,212)"> postgresql_hba_entries:
- { type: local, database: all, user: <= span style=3D"color:rgb(206,145,120)">postgres, auth_method: peer }
= - { type: local, database: all, use= r: all, auth_method: peer= }
- { type: host, database: all, user: all, = address: '127.0.0.1/32'= , auth_method: md5 }
- { type: host, database: all, user: all, address: '::1/128', auth_metho= d: md5 } =
- { <= /span>type: host, database: all, user: all, address: '0.0.0.0/0', auth_= method: md5 = }
- { type: host<= span style=3D"color:rgb(212,212,212)">, database: all, user: all, address: '::0/0'= , auth_method: md5 }

postgresql_global_config_options:
- option: unix_socket_directories
value: = '{{ postgresql_unix_socke= t_directories | join(",") }}'

- option: = standard_conforming_strings
value: 'on'
- option: shared_buffers
value: '1024MB'

# max_wal_size =3D (3 * checkpoint_segmen= ts) * 16MB
= # checkpoint_segments=3D300
- option: max_wal_size
value: '14400MB'

- option: min_wal_siz= e
= value: '80MB'= ;

- option: maintena= nce_work_mem
= value: &#= 39;2MB'

- option: listen_addresse= s
value: '*= 9;

- option: max_con= nections
= value: '4= 00'

= - option: ch= eckpoint_timeout
= value: '900'

- option: datestyle
= value: "iso, mdy"

- option= : autovacuum
value: 'off'

# vacuum all databases every night (full vacuum on Sunday night, l= azy vacuum every night)
- name: add postgresql cron lazy vacuum
cro= n:
name: lazy_vacuum
hour: 8
minute<= span style=3D"color:rgb(212,212,212)">: 0
= job: "= su - postgres -c 'vacuumdb --all --analyze --quiet'"
- name: = add postgresql cron full vacu= um
cron:
= name: full_v= acuum
weekday: 0<= /div>
hour: 10
minute: 0
jo= b: "su - postgres -c 'vacuumdb --all --full --ana= lyze --quiet'"
# re-index all databas= es once a week
= - name: add p= ostgresql cron reindex
cron:
name: reindex
weekday: 0
hour: 12=
minute: 0
job: "su - postgres -c 'psql -t -c = \"select datname from pg_database order by datname;\" | xargs -n = 1 -I\"{}\" -- psql -U postgres {} -c \"reindex database {};\= "' "


This is how I run 2= .10.
Been running fine for some weeks without user intervention.
<= div>@Karl: Any comments please?
Steph





--000000000000ed444b05750dcfdb--