manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Tavard <olivier.tav...@francelabs.com>
Subject Re: PostgreSQL version to support MCF v2.10
Date Thu, 23 Aug 2018 07:12:41 GMT
Hi,

I am also interested by the migration to PostgreSQL 10.
@Steph it would be nice if you could precise the changes you did in the configuration file
when you wrote :
"I'm using 10.4 with no issues. 

One or two of the recommended settings for MCF have changed between 9.6 and 10. 

Simple to resolve though.
"

Thanks !
Best regards,

Olivier


> Le 6 août 2018 à 15:52, Karl Wright <daddywri@gmail.com> a écrit :
> 
> It is what is expected with multiple threads active at the same time.
> Karl
> 
> 
> On Mon, Aug 6, 2018 at 7:26 AM Standen Guy <Guy.Standen@uk.fujitsu.com <mailto:Guy.Standen@uk.fujitsu.com>>
wrote:
> Hi Karl,
> 
> I haven’t experienced any job aborts, so all seems OK in that respect.
> 
> Is there anything I can do to reduce these errors in the first place, or it is just to
be expected with the nature of the multiple worker threads and the query types issued by ManifoldCF?
> 
> Best Regards,
> 
>  
> 
> Guy
> 
>  
> 
> From: Karl Wright [mailto:daddywri@gmail.com <mailto:daddywri@gmail.com>] 
> Sent: 06 August 2018 12:16
> To: user@manifoldcf.apache.org <mailto:user@manifoldcf.apache.org>
> Subject: Re: PostgreSQL version to support MCF v2.10
> 
>  
> 
> These are exactly the same kind of issue as the first "error" reported.  They will be
retried.  If they did not get retried, they would abort the job immediately.
> 
>  
> 
> Karl
> 
>  
> 
>  
> 
> On Mon, Aug 6, 2018 at 6:57 AM Standen Guy <Guy.Standen@uk.fujitsu.com <mailto:Guy.Standen@uk.fujitsu.com>>
wrote:
> 
> Hi Karl,
> 
>                Thanks for the prompt response regarding the first  error example.   Do
you have a view as to second error  i.e.
> 
> “2018-08-03 15:52:42.855 BST [5272] ERROR:  could not serialize access due to concurrent
update
> 
> 2018-08-03 15:52:42.855 BST [5272] STATEMENT:  SELECT id,status,checktime FROM jobqueue
WHERE dochash=$1 AND jobid=$2 FOR UPDATE
> 
> 2018-08-03 15:52:42.855 BST [7424] ERROR:  could not serialize access due to concurrent
update
> 
> 2018-08-03 15:52:42.855 BST [7424] STATEMENT:  SELECT id,status,checktime FROM jobqueue
WHERE dochash=$1 AND jobid=$2 FOR UPDATE
> 
> 2018-08-03 15:52:42.855 BST [5716] ERROR:  could not serialize access due to concurrent
update
> 
> “
> 
>  
> 
> These errors don’t suggest a retry may sort them out  - is this an issue?
> 
>  
> 
> Many Thanks,
> 
>  
> 
> Guy
> 
>  
> 
> From: Karl Wright [mailto:daddywri@gmail.com <mailto:daddywri@gmail.com>] 
> Sent: 06 August 2018 10:52
> To: user@manifoldcf.apache.org <mailto:user@manifoldcf.apache.org>
> Subject: Re: PostgreSQL version to support MCF v2.10
> 
>  
> 
> Ah, the following errors:
> 
> >>>>>>
> 
> 2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due to read/write
dependencies among transactions
> 
> 2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on identification as
a pivot, during conflict in checking.
> 
> 2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed if retried.
> 
> <<<<<< 
> 
>  
> 
> ... occur because of concurrent transactions.  The transaction is indeed retried when
this occurs, so unless your job aborts, you are fine.
> 
>  
> 
> Karl
> 
>  
> 
>  
> 
> On Mon, Aug 6, 2018 at 5:49 AM Karl Wright <daddywri@gmail.com <mailto:daddywri@gmail.com>>
wrote:
> 
> What errors are these?  Please include them and I can let you know.
> 
>  
> 
> Karl
> 
>  
> 
>  
> 
> On Mon, Aug 6, 2018 at 4:50 AM Standen Guy <Guy.Standen@uk.fujitsu.com <mailto:Guy.Standen@uk.fujitsu.com>>
wrote:
> 
> Thank you Karl and Steph,
> 
>  
> 
> Steph, yes I don’t seem to have any issues with running the MCF jobs, but am concerned
about the PostgreSQL errors. Do you ( or anyone else)  have a view on the errors I have seen
in the PostgreSQL logs  - is this something you have seen with 10.4  and if so was it corrected
by changing some settings? 
> 
>  
> 
> Best Regards
> 
>  
> 
> Guy
> 
>  
> 
> From: Steph van Schalkwyk [mailto:steph@remcam.net <mailto:steph@remcam.net>] 
> Sent: 03 August 2018 23:21
> To: user@manifoldcf.apache.org <mailto:user@manifoldcf.apache.org>
> Subject: Re: PostgreSQL version to support MCF v2.10
> 
>  
> 
> I'm using 10.4 with no issues. 
> 
> One or two of the recommended settings for MCF have changed between 9.6 and 10. 
> 
> Simple to resolve though.
> 
> Steph
> 
>  
> 
> 
> 
>  
> 
> On Fri, Aug 3, 2018 at 1:29 PM, Karl Wright <daddywri@gmail.com <mailto:daddywri@gmail.com>>
wrote:
> 
> Hi Guy,
> 
>  
> 
> I use Postgresql 9.6 myself and have found no issues with it.  I don't know about v 10
however.
> 
>  
> 
> Karl
> 
>  
> 
>  
> 
> On Fri, Aug 3, 2018 at 11:32 AM Standen Guy <Guy.Standen@uk.fujitsu.com <mailto:Guy.Standen@uk.fujitsu.com>>
wrote:
> 
> Hi Karl/All,
> 
>                I am upgrading from MCF v2.6  supported by PostgreSQL v 9.3.16   to  MCF
v2.10.  I wonder if there is any official advice as to which version of PostgreSQL  will support
 MCF v2.10? The  MCF v2.10 build and deployment instructions still suggest that PostgreSQL
9.3 is the latest tested version of PostgreSQL.  Given that PostgreSQL 9.3.x  is going end
of life next month ( Sept 2018), is there a preferred newer version that should be used?
> 
>  
> 
> As an experiment I have installed MCF 2.10  supported by PostgreSQL 10.4.  From the outside
all seems to work OK, but investigation of the PostgreSQL  logs shows a lot of errors:
> 
>  
> 
> e.g.
> 
> “2018-08-03 15:50:00.629 BST [7920] LOG:  database system was shut down at 2018-08-03
15:47:30 BST
> 
> 2018-08-03 15:50:00.734 BST [6344] LOG:  database system is ready to accept connections
> 
> 2018-08-03 15:52:11.140 BST [6460] WARNING:  there is already a transaction in progress
> 
> 2018-08-03 15:52:11.219 BST [6460] WARNING:  there is no transaction in progress
> 
> 2018-08-03 15:52:13.844 BST [5716] WARNING:  there is already a transaction in progress
> 
> 2018-08-03 15:52:13.879 BST [5716] WARNING:  there is no transaction in progress
> 
> 2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due to read/write
dependencies among transactions
> 
> 2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on identification as
a pivot, during conflict in checking.
> 
> 2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed if retried.
> 
> 2018-08-03 15:52:25.218 BST [4140] STATEMENT:  INSERT INTO jobqueue (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
> 
> 2018-08-03 15:52:25.219 BST [5800] ERROR:  could not serialize access due to read/write
dependencies among transactions
> 
> 2018-08-03 15:52:25.219 BST [5800] DETAIL:  Reason code: Canceled on identification as
a pivot, during conflict in checking.
> 
> 2018-08-03 15:52:25.219 BST [5800] HINT:  The transaction might succeed if retried.
> 
> 2018-08-03 15:52:25.219 BST [5800] STATEMENT:  INSERT INTO jobqueue (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
> 
> 2018-08-03 15:52:25.222 BST [5692] ERROR:  could not serialize access due to read/write
dependencies among transactions
> 
> 2018-08-03 15:52:25.222 BST [5692] DETAIL:  Reason code: Canceled on identification as
a pivot, during conflict in checking.
> 
> 2018-08-03 15:52:25.222 BST [5692] HINT:  The transaction might succeed if retried.
> 
> 2018-08-03 15:52:25.222 BST [5692] STATEMENT:  INSERT INTO jobqueue (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
> 
> 2018-08-03 15:52:28.149 BST [4140] ERROR:  could not serialize access due to read/write
dependencies among transactions
> 
> 2018-08-03 15:52:28.149 BST [4140] DETAIL:  Reason code: Canceled on identification as
a pivot, during write.
> 
> 2018-08-03 15:52:28.149 BST [4140] HINT:  The transaction might succeed if retried.
> 
> 2018-08-03 15:52:28.149 BST [4140] STATEMENT:  UPDATE intrinsiclink SET processid=$1,isnew=$2
WHERE jobid=$3 AND parentidhash=$4 AND linktype=$5 AND childidhash=$6
> 
> 2018-08-03 15:52:28.261 BST [5156] ERROR:  could not serialize access due to read/write
dependencies among transactions
> 
> 2018-08-03 15:52:28.261 BST [5156] DETAIL:  Reason code: Canceled on identification as
a pivot, during write.
> 
> 2018-08-03 15:52:28.261 BST [5156] HINT:  The transaction might succeed if retried.”
> 
>  
> 
> And
> 
>  
> 
> “2018-08-03 15:52:42.855 BST [5272] ERROR:  could not serialize access due to concurrent
update
> 
> 2018-08-03 15:52:42.855 BST [5272] STATEMENT:  SELECT id,status,checktime FROM jobqueue
WHERE dochash=$1 AND jobid=$2 FOR UPDATE
> 
> 2018-08-03 15:52:42.855 BST [7424] ERROR:  could not serialize access due to concurrent
update
> 
> 2018-08-03 15:52:42.855 BST [7424] STATEMENT:  SELECT id,status,checktime FROM jobqueue
WHERE dochash=$1 AND jobid=$2 FOR UPDATE
> 
> 2018-08-03 15:52:42.855 BST [5716] ERROR:  could not serialize access due to concurrent
update
> 
> 2018-08-03 15:52:42.855 BST [5716] STATEMENT:  SELECT id,status,checktime FROM jobqueue
WHERE dochash=$1 AND jobid=$2 FOR UPDATE
> 
> 2018-08-03 15:52:42.856 BST [1328] ERROR:  could not serialize access due to concurrent
update
> 
> 2018-08-03 15:52:42.856 BST [1328] STATEMENT:  SELECT id,status,checktime FROM jobqueue
WHERE dochash=$1 AND jobid=$2 FOR UPDATE
> 
> 2018-08-03 15:52:42.856 BST [5800] ERROR:  could not serialize access due to concurrent
update
> 
> 2018-08-03 15:52:42.856 BST [5800] STATEMENT:  SELECT id,status,checktime FROM jobqueue
WHERE dochash=$1 AND jobid=$2 FOR UPDATE”
> 
>  
> 
> Do you have any advice as to whether it is sensible to use PostgreSQL v10.x   and if
so can these errors be overcome?
> 
>  
> 
> Best Regards,
> 
>  
> 
> Guy
> 
> 
> Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered
in England No 96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered
offices at: 22 Baker Street, London W1U 3BW <https://maps.google.com/?q=22+Baker+Street,+London+W1U+3BW&entry=gmail&source=g>;
PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu Laboratories of Europe
Limited (registered in England No. 4153469) both with registered offices at: Hayes Park Central,
Hayes End Road, Hayes, Middlesex, UB4 8FE. 
> This email is only for the use of its intended recipient. Its contents are subject to
a duty of confidence and may be privileged. Fujitsu does not guarantee that this email has
not been intercepted and amended or that it is virus-free.
> 
>  
> 
> 
> Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered
in England No 96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered
offices at: 22 Baker Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No
1578652) and Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) both
with registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.

> This email is only for the use of its intended recipient. Its contents are subject to
a duty of confidence and may be privileged. Fujitsu does not guarantee that this email has
not been intercepted and amended or that it is virus-free.
> 
> 
> Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered
in England No 96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered
offices at: 22 Baker Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No
1578652) and Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) both
with registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.

> This email is only for the use of its intended recipient. Its contents are subject to
a duty of confidence and may be privileged. Fujitsu does not guarantee that this email has
not been intercepted and amended or that it is virus-free.
> 
> 
> Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered
in England No 96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered
offices at: 22 Baker Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No
1578652) and Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) both
with registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.

> This email is only for the use of its intended recipient. Its contents are subject to
a duty of confidence and may be privileged. Fujitsu does not guarantee that this email has
not been intercepted and amended or that it is virus-free.


Mime
View raw message