From user-return-1794-archive-asf-public=cust-asf.ponee.io@predictionio.apache.org Mon Feb 12 23:02:55 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 7795A180652 for ; Mon, 12 Feb 2018 23:02:55 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 67881160C3F; Mon, 12 Feb 2018 22:02:55 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 09543160C30 for ; Mon, 12 Feb 2018 23:02:53 +0100 (CET) Received: (qmail 33104 invoked by uid 500); 12 Feb 2018 22:02:53 -0000 Mailing-List: contact user-help@predictionio.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@predictionio.apache.org Delivered-To: mailing list user@predictionio.apache.org Received: (qmail 33094 invoked by uid 99); 12 Feb 2018 22:02:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Feb 2018 22:02:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 960B618008E for ; Mon, 12 Feb 2018 22:02:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id NCNgWQVmYnha for ; Mon, 12 Feb 2018 22:02:51 +0000 (UTC) Received: from mail-lf0-f51.google.com (mail-lf0-f51.google.com [209.85.215.51]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 5956F5F3CD for ; Mon, 12 Feb 2018 22:02:50 +0000 (UTC) Received: by mail-lf0-f51.google.com with SMTP id w10so9663755lfc.9 for ; Mon, 12 Feb 2018 14:02:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=H1e41ClJZQklUOem52yn10ulYTAHrjbUmVQkhnloU/s=; b=Pp+xsKSVeP49aQePFL3V0NDR9dUI76RQ9WPOElrj4B+OJV/r0KmqKDxWD6RJbqql3p Cj7aAVBD31fqRZMNVXS+eHnxNelvdxjumyQMUJbI34/dyUk7fRXi6KOkulu1II+f08Dk w8hNCY2Mby7TCwZxCIfwYdxv/5F38f69kIKIEZ03W0xj/aJgZQBquRfdfr6BAoT4Kbpe wa3KZRv49ZOVohhHiUixenxI5sIzWczDph34PlnpGAv/iN66g/Mgjs7ywg+YXugrPLlA kC/+EMTgEd0sUeuCtX0MiKjwBXSyVbKSsX+DxHcVdaM8D+W2aIv8MiDB1G+BhMwU5KTy BBvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=H1e41ClJZQklUOem52yn10ulYTAHrjbUmVQkhnloU/s=; b=ro/cJRo2DuokHMvSw8+MO11NtnrGHw0XEsqRExtY+xaZvDvayLXZGbcGG7xhqH9fPi zyXrzg0pJo1Tb//54LbhIkFDcDeLu5Pl2ypl0ruxoF5CZp3hqIazUBumzDYVFNQkVLT5 IcRSFjLlxDrAZJwSvxJhpnaEvxvjkjrZ1/atHQEfoMdvZ4ErW2nvPLUfbJt6Mx40J0to BSn0XJLir88zkactlXw6Ph8zYobwyMvHy0Uee1pgsOKmwv8haLTQ2VdVCAGVMJ7tQ0mh 9qOx5v2363OE4vCMPazTZDTHf6NrxKdRo/Ore4mIGsR0lnOBcERRYnvDkP0JpBdXXs+B yYwQ== X-Gm-Message-State: APf1xPA3i4ZnR450udwZPjfEt1lzET1LpiDbCxBHv0gM2UvhSmqpioVU UGHuAJcJpGGNXss2sgOfmyWu2DFPxIhVGZVCaGzYQA== X-Google-Smtp-Source: AH8x224OMxbgRyp5FTA4igUmvvMEDZZJyd0/6zNfpCqR/f7pP16uUsaFAq7r38wX1DD9VYkOFJ2C4vv0UoXoZGyZSko= X-Received: by 10.25.24.203 with SMTP id 72mr5878494lfy.132.1518472969323; Mon, 12 Feb 2018 14:02:49 -0800 (PST) MIME-Version: 1.0 Received: by 10.46.57.22 with HTTP; Mon, 12 Feb 2018 14:02:48 -0800 (PST) From: =?UTF-8?Q?Tihomir_Loli=C4=87?= Date: Mon, 12 Feb 2018 23:02:48 +0100 Message-ID: Subject: Dynamically change parameter list To: user@predictionio.apache.org Content-Type: multipart/alternative; boundary="001a11406ce2f00a2205650b09b7" --001a11406ce2f00a2205650b09b7 Content-Type: text/plain; charset="UTF-8" Hi, I am trying to figure out how to dynamically update algorithm parameter list. After the train is finished only model is updated. The reason why I need this data to be updated is that I am creating data mapping based on the training data. Is there a way to update this data after the train is done? Here is the code that I am using. The variable that and should be updated after the train is marked *bold red.* import io.prediction.controller.{EmptyParams, EngineParams} import io.prediction.data.storage.EngineInstance import io.prediction.workflow.CreateWorkflow.WorkflowConfig import io.prediction.workflow._ import org.apache.spark.ml.linalg.SparseVector import org.joda.time.DateTime import org.json4s.JsonAST._ import scala.collection.mutable object TrainApp extends App { val envs = Map("FOO" -> "BAR") val sparkEnv = Map("spark.master" -> "local") val sparkConf = Map("spark.executor.extraClassPath" -> ".") val engineFactoryName = "LogisticRegressionEngine" val workflowConfig = WorkflowConfig( engineId = EngineConfig.engineId, engineVersion = EngineConfig.engineVersion, engineVariant = EngineConfig.engineVariantId, engineFactory = engineFactoryName ) val workflowParams = WorkflowParams( verbose = workflowConfig.verbosity, skipSanityCheck = workflowConfig.skipSanityCheck, stopAfterRead = workflowConfig.stopAfterRead, stopAfterPrepare = workflowConfig.stopAfterPrepare, sparkEnv = WorkflowParams().sparkEnv ++ sparkEnv ) WorkflowUtils.modifyLogging(workflowConfig.verbose) val dataSourceParams = DataSourceParams(sys.env.get("APP_NAME").get) val preparatorParams = EmptyParams() * val algorithmParamsList = Seq("Logistic" -> LogisticParams(columns = Array[String](),* * dataMapping = Map[String, Map[String, SparseVector]]()))* val servingParams = EmptyParams() val engineInstance = EngineInstance( id = "", status = "INIT", startTime = DateTime.now, endTime = DateTime.now, engineId = workflowConfig.engineId, engineVersion = workflowConfig.engineVersion, engineVariant = workflowConfig.engineVariant, engineFactory = workflowConfig.engineFactory, batch = workflowConfig.batch, env = envs, sparkConf = sparkConf, dataSourceParams = JsonExtractor.paramToJson(workflowConfig.jsonExtractor, workflowConfig.engineParamsKey -> dataSourceParams), preparatorParams = JsonExtractor.paramToJson(workflowConfig.jsonExtractor, workflowConfig.engineParamsKey -> preparatorParams), algorithmsParams = JsonExtractor.paramsToJson(workflowConfig.jsonExtractor, algorithmParamsList), servingParams = JsonExtractor.paramToJson(workflowConfig.jsonExtractor, workflowConfig.engineParamsKey -> servingParams) ) val (engineLanguage, engineFactory) = WorkflowUtils.getEngine(engineInstance.engineFactory, getClass.getClassLoader) val engine = engineFactory() val engineParams = EngineParams( dataSourceParams = dataSourceParams, preparatorParams = preparatorParams, algorithmParamsList = algorithmParamsList, servingParams = servingParams ) val engineInstanceId = CreateServer.engineInstances.insert(engineInstance) CoreWorkflow.runTrain( env = envs, params = workflowParams, engine = engine, engineParams = engineParams, engineInstance = engineInstance.copy(id = engineInstanceId) ) CreateServer.actorSystem.shutdown() } Thank you, Tihomir --001a11406ce2f00a2205650b09b7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I am trying to figure out how to dy= namically=C2=A0update algorithm parameter list. After the train is finished= only model is updated. The reason why I need this data to be updated is th= at I am creating=C2=A0data mapping based on the training data. Is there a w= ay to update this data after the train is done?

He= re is the code that I am using. The variable that and should be updated aft= er the train is marked bold red.

import io.prediction.controller.{EmptyParams, Eng= ineParams}
import io.prediction.data.storage.EngineInstance
=
import io.prediction.workflow.CreateWorkflow.WorkflowConfig
= import io.prediction.workflow._
import org.apache.spark.ml.linalg= .SparseVector
import org.joda.time.DateTime
import org.= json4s.JsonAST._

import scala.collection.mutable

object TrainApp extends App {

<= div>=C2=A0 val envs =3D Map("FOO" -> "BAR")

=C2=A0 val sparkEnv =3D Map("spark.master" ->= "local")

=C2=A0 val sparkConf =3D Map(&= quot;spark.executor.extraClassPath" -> ".")
=C2=A0 val engineFactoryName =3D "LogisticRegressionEngine= "

=C2=A0 val workflowConfig =3D WorkflowConfi= g(
=C2=A0 =C2=A0 engineId =3D EngineConfig.engineId,
= =C2=A0 =C2=A0 engineVersion =3D EngineConfig.engineVersion,
=C2= =A0 =C2=A0 engineVariant =3D EngineConfig.engineVariantId,
=C2=A0= =C2=A0 engineFactory =3D engineFactoryName
=C2=A0 )
=C2=A0 val workflowParams =3D WorkflowParams(
=C2=A0= =C2=A0 verbose =3D workflowConfig.verbosity,
=C2=A0 =C2=A0 skipS= anityCheck =3D workflowConfig.skipSanityCheck,
=C2=A0 =C2=A0 stop= AfterRead =3D workflowConfig.stopAfterRead,
=C2=A0 =C2=A0 stopAft= erPrepare =3D workflowConfig.stopAfterPrepare,
=C2=A0 =C2=A0 spar= kEnv =3D WorkflowParams().sparkEnv ++ sparkEnv
=C2=A0 )

=C2=A0 WorkflowUtils.modifyLogging(workflowConfig.verbose)<= /div>

=C2=A0 val dataSourceParams =3D DataSourceParams(s= ys.env.get("APP_NAME").get)
=C2=A0 val preparatorParams= =3D EmptyParams()

= =C2=A0 val algorithmParamsList =3D Seq("Log= istic" -> LogisticParams(columns =3D Array[String](),
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 dataMapping =3D Map[String, Map[String, Spa= rseVector]]()))
=C2=A0 val servingParams =3D EmptyPara= ms()

=C2=A0 val engineInstance =3D EngineInstance(=
=C2=A0 =C2=A0 id =3D "",
=C2=A0 =C2=A0 statu= s =3D "INIT",
=C2=A0 =C2=A0 startTime =3D DateTime.now,=
=C2=A0 =C2=A0 endTime =3D DateTime.now,
=C2=A0 =C2=A0 = engineId =3D workflowConfig.engineId,
=C2=A0 =C2=A0 engineVersion= =3D workflowConfig.engineVersion,
=C2=A0 =C2=A0 engineVariant = =3D workflowConfig.engineVariant,
=C2=A0 =C2=A0 engineFactory =3D= workflowConfig.engineFactory,
=C2=A0 =C2=A0 batch =3D workflowCo= nfig.batch,
=C2=A0 =C2=A0 env =3D envs,
=C2=A0 =C2=A0 s= parkConf =3D sparkConf,
=C2=A0 =C2=A0 dataSourceParams =3D JsonEx= tractor.paramToJson(workflowConfig.jsonExtractor, workflowConfig.enginePara= msKey -> dataSourceParams),
=C2=A0 =C2=A0 preparatorParams =3D= JsonExtractor.paramToJson(workflowConfig.jsonExtractor, workflowConfig.eng= ineParamsKey -> preparatorParams),
=C2=A0 =C2=A0 algorithmsPar= ams =3D JsonExtractor.paramsToJson(workflowConfig.jsonExtractor, algorithmP= aramsList),
=C2=A0 =C2=A0 servingParams =3D JsonExtractor.paramTo= Json(workflowConfig.jsonExtractor, workflowConfig.engineParamsKey -> ser= vingParams)
=C2=A0 )

=C2=A0 val (engineL= anguage, engineFactory) =3D WorkflowUtils.getEngine(engineInstance.engineFa= ctory, getClass.getClassLoader)

=C2=A0 val engine = =3D engineFactory()

=C2=A0 val engineParams =3D En= gineParams(
=C2=A0 =C2=A0 dataSourceParams =3D dataSourceParams,<= /div>
=C2=A0 =C2=A0 preparatorParams =3D preparatorParams,
= =C2=A0 =C2=A0 algorithmParamsList =3D algorithmParamsList,
=C2=A0= =C2=A0 servingParams =3D servingParams
=C2=A0 )

=C2=A0 val engineInstanceId =3D CreateServer.engineInstances.inser= t(engineInstance)

=C2=A0 CoreWorkflow.runTrain(
=C2=A0 =C2=A0 env =3D envs,
=C2=A0 =C2=A0 params =3D work= flowParams,
=C2=A0 =C2=A0 engine =3D engine,
=C2=A0 =C2= =A0 engineParams =3D engineParams,
=C2=A0 =C2=A0 engineInstance = =3D engineInstance.copy(id =3D engineInstanceId)
=C2=A0 )

=C2=A0 CreateServer.actorSystem.shutdown()
}


Thank you,
Tihomir

--001a11406ce2f00a2205650b09b7--