hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tao tony <tonytao0...@outlook.com>
Subject ISSUE:process not exit immediately while connection interrupted or query finished
Date Thu, 05 Jul 2018 08:57:28 GMT
hi,

When client (such as pgbouncer,jdbc,zeppelin) connected to hawq and 
execute a long query,if the client connection interrupted before query 
finished,the server process will not exit until  an hour later.

We had replay this issue on HAWQ 2.3.0.0-incubating and HDB-2.3.0.0.And 
set parameter gp_interconnect_transmit_timeout to 600(default 3600) will 
reduce the time to 10 minutes.

When the query wa running,we could see its status in 
pg_stat_activty,but  after it finished  we could only saw the process id 
in pg_locks and OS process.

We could saw some  error  log as below:

$ tailf hawq-2018-07-04_063514.csv|grep p294
2018-07-04 08:13:29.595365 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 UTC,58896,con19,cmd32,seg-1,,,x58896,sx1,"LOG","00000","ConnID 
5. Returned resource to resource manager.",,,,,,,0,,"rmcomm_QD2RM.c",951,
2018-07-04 08:13:29.595555 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 UTC,58896,con19,cmd32,seg-1,,,x58896,sx1,"LOG","00000","ConnID 
5. Unregistered from HAWQ resource manager.",,,,,,,0,,"rmcomm_QD2RM.c",661,
2018-07-04 08:15:58.706458 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 UTC,58903,con19,cmd34,seg-1,,,x58903,sx1,"LOG","00000","ConnID 
6. Registered in HAWQ resource manager (By OID)",,,,,,"select * from 
cppayorderproduct",0,,"rmcomm_QD2RM.c",609,
2018-07-04 08:15:58.706640 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 UTC,58903,con19,cmd34,seg-1,,,x58903,sx1,"LOG","00000","ConnID 
6. Acquired resource from resource manager, (256 MB, 0.062500 CORE) x 
18.",,,,,,"select * from cppayorderproduct",0,,"rmcomm_QD2RM.c",868,
2018-07-04 09:04:56.190873 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","08006","could 
not send data to client: Connection reset by peer",,,,,,"select * from 
cppayorderproduct",0,,"pqcomm.c",1413,
2018-07-04 09:04:56.192347 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"FATAL","08006","connection to 
client lost",,,,,,"select * from cppayorderproduct",0,,"postgres.c",3606,
2018-07-04 10:04:56.306412 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_consume meets error, connection is bad.",,,,,,,0,,,,
2018-07-04 10:04:56.306535 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","dispmgt_thread_func_run(): 
fail to consume data. Will exit and clean up.",,,,,,,0,,,,
2018-07-04 10:04:56.309663 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.312741 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.315364 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.317885 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.320411 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.322998 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.327342 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.330034 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.332656 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.335257 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.337972 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.340634 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.343785 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.346309 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.350458 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.352760 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.354846 
UTC,"dev","hdb",p294,th1627412224,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","function 
executormgr_cancel calling executormgr_catch_error",,,,,,,0,,,,
2018-07-04 10:04:56.354996 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","dispatcher 
thinks centos7-datanode3.centos7_hawq_network is 
down.",,,,,,,0,,"rmcomm_QD2RM.c",1209,
2018-07-04 10:04:56.355020 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","dispatcher sends 
1 failed host(s) to resource manager.",,,,,,,0,,"rmcomm_QD2RM.c",1213,
2018-07-04 10:04:56.355398 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","succeed 
in sending failed host to resource manager.",,,,,,,0,,"rmcomm_QD2RM.c",1232,
2018-07-04 10:04:56.356559 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","ConnID 
6. Returned resource to resource manager.",,,,,,,0,,"rmcomm_QD2RM.c",951,
2018-07-04 10:04:56.356869 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 UTC,58903,con19,cmd35,seg-1,,,x58903,sx1,"LOG","00000","ConnID 
6. Unregistered from HAWQ resource manager.",,,,,,,0,,"rmcomm_QD2RM.c",661,
2018-07-04 10:04:56.356892 
UTC,"dev","hdb",p294,th1628359104,"172.17.10.148","63974",2018-07-04 
06:37:28 UTC,0,con19,cmd35,seg-1,,,,,"LOG","00000","clean up 
communication to resource manager now.",,,,,,,0,,"rmcomm_QD2RM.c",460,
2018-07-04 10:04:56.379198 
UTC,"dev","hdb",p294,th998237952,"172.17.10.148","63974",2018-07-04 
06:37:28 
UTC,0,con19,cmd35,seg-1,,,,,"LOG","00000","generateResourceRefreshHeartBeat 
exits.",,,,,,,0,,,,


The master process backtrace is in attched.

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message