predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Donald Szeto <don...@apache.org>
Subject Re: PIO deploy: old processes still live after redeployment
Date Wed, 11 Apr 2018 18:48:25 GMT
Hi George,

Mind checking the content of pio.log at where you run `pio deploy`? It
should contain some messages regarding undeploying existing engine servers
on the same port.

Regards,
Donald

On Thu, Apr 5, 2018 at 6:43 PM, George Yarish <gyarish@griddynamics.com>
wrote:

> Hi everyone,
>
> I faced a strange situation recently.
> One of our engines has frequently old processes after daily redeployment.
>
> We used to redeploy by crontab rule that looks like:
> nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 &
>
>
> This is output of ps aux command:
>
> [da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004
>
> da_mcom+  7854  0.0  0.0 112660   996 pts/1    S+   21:34   0:00 grep
> --color=auto *8004*
>
> 1. da_mcom+ 30932  0.6  0.9 26473632 595932 ?     Sl   01:57   7:51 java
> -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/
> hadoop/conf:/data/PredictionIO/vendors/hbase/conf
> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>
> 2. da_mcom+ 31097  1.2  1.2 17065124 822596 ?     Sl   01:57  14:48
> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.
> 0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-ap
> i-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanuc
> leus-core-3.2.10.jar:/data/PredictionIO/vendors/spark/
> lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop
> /conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.
> 10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-
> sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
> org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
> ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/
> hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf
> spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
> org.apache.predictionio.workflow.CreateServer --jars
> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
> /target/scala-2.10/universal-recommender-assembly
> -0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating
> /lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubatin
> g.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/
> pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/
> PredictionIO-0.11.0-incubating/lib/spark/pio-data-
> hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-
> 0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-incubating
> /lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar --files
> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port *8004*
> --event-server-ip 0.0.0.0 --event-server-port 7070 --json-extractor Both
> --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_ST
> ORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/ho
> me/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARC
> H_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/data/
> PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.
> 11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_
> milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/
> home/da_mcom_milan/.pio_store/models,PIO_STORAGE_SOURCES_
> ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=
> ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=
> LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=
> pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=
> pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,
> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_
> STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_
> CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_
> STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_
> SOURCES_LOCALFS_TYPE=localfs
>
> 3. da_mcom+ 32505  0.6  0.8 26473632 581164 ?     Sl   02:00   7:54 java
> -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/
> hadoop/conf:/data/PredictionIO/vendors/hbase/conf
> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>
> 4. da_mcom+ 32671  1.4  1.3 19757488 881448 ?     Sl   02:00  16:27
> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.
> 0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-ap
> i-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanuc
> leus-core-3.2.10.jar:/data/PredictionIO/vendors/spark/
> lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop
> /conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.
> 10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-
> sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
> org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
> ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/
> hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf
> spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
> org.apache.predictionio.workflow.CreateServer --jars
> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
> /target/scala-2.10/universal-recommender-assembly
> -0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating
> /lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubatin
> g.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/
> pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/
> PredictionIO-0.11.0-incubating/lib/spark/pio-data-
> hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-
> 0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-incubating
> /lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar --files
> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port *8004*
> --event-server-ip 0.0.0.0 --event-server-port 7070 --json-extractor Both
> --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_ST
> ORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/ho
> me/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARC
> H_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/data/
> PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.
> 11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_
> milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/
> home/da_mcom_milan/.pio_store/models,PIO_STORAGE_SOURCES_
> ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=
> ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=
> LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=
> pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=
> pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,
> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_
> STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_
> CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_
> STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_
> SOURCES_LOCALFS_TYPE=localfs
>
> For some reasons, processes 30932 and 31097 still alive and process
> listened by 32671
> Any ideas?
>
> Thank you,
> George
>

Mime
View raw message