Mailing-List: contact dev-help@openwhisk.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@openwhisk.apache.org
Mime-Version: 1.0
Date: Thu, 27 Apr 2017 11:30:44 +0000
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Message-ID: <99ca90387b96b85f511963d4a7576b96@rainloop.cbickel.de>
From: apache@cbickel.de
Subject: Re: Save activations in a new activations DB
To: dev@openwhisk.apache.org
In-Reply-To: <65A699ED-11F7-4CA7-9A19-AFC414589B34@gmail.com>
References: <65A699ED-11F7-4CA7-9A19-AFC414589B34@gmail.com>
 <c90a4c1ad5d1cf57d186855a41b14fca@rainloop.cbickel.de>
 <df42459fcb772aa1e28ffd5f973f534c@rainloop.cbickel.de>
 <BL2PR02MB5161A4C42AA6E5A13C08924D7050@BL2PR02MB516.namprd02.prod.outlook.com>
archived-at: Thu, 27 Apr 2017 11:31:06 -0000

Hi OpenWhisk developers,=0A=0Athere are again some news on separating the=
 activations into a new database.=0AThe next Pull request (https://github=
.com/openwhisk/openwhisk/pull/2134) will be merged soon. After=0Athis Pul=
l request is merged, the flag (db_split_actions_and_activations) to enabl=
e or disable the=0Aseparation of the activations is required to deploy Op=
enWhisk.=0AIt has to be set in your environemnts group_vars/all file.=0AA=
dditionally, this PR will change the default of the 3 environments (local=
, mac, distributed) to=0Ause the additional activations database for all =
activations.=0AIf you want to have new activations still in your whisks d=
atabase, just set this flag to "false" in=0Ayour environment. But keep in=
 mind, that the ability to use the whisks database for activations=0Awill=
 go away soon.=0A=0AIf you want to keep all activations and want to keep =
them accessible for the users of your=0Adeployment, you have to migrate t=
he activations into the new activations database. The steps to do=0Athis =
are described in: https://gist.github.com/cbickel/37e651965781b27de245eac=
0ce831a53=0AIf it is okay for your environment to loose all data stored i=
n the database, the databases will be=0Aregenerated with the right struct=
ure with the "wipe.yml" playbook.=0A=0AMy next PR will remove the ability=
 to use the whisks database for activations.=0A=0AGreetings=0AChristian=
=0A=0AApril 16, 2017 1:17 AM, "Rodric Rabbah" <rodric@gmail.com> wrote:=
=0A=0A> I've seen a proposal and initial prototype from Jeremias which te=
es logs to an external drain. The=0A> idea is to make this plugable in th=
e way you might be thinking but I'll let him share the details=0A> and th=
oughts and you can evaluate and we can iterate.=0A> =0A> I don't think we=
 want to give up the concurrent draining of logs to Couch where they are =
available=0A> nearly instantaneously as part of the core system. This is =
superior to other systems including=0A> Lambda for example (a common comp=
laint is that logs take a long time to be available). I view this=0A> as =
essential to a fast debug cycle while developing.=0A> =0A> That said, it =
is clear that we need to drain logs to a system that will allow more gene=
ral=0A> indexing, searching, measuring, and analyzing logs - and in past =
two weeks this has come up=0A> repeatedly in our public slack channel. Th=
is in my view should be done as a plugable external layer=0A> rather than=
 part of the core API. This isn't to say the core API can't provide some =
more views that=0A> can be generally useful.=0A> =0A> -r=0A> =0A>> On Apr=
 14, 2017, at 7:55 PM, Michael Marth <mmarth@adobe.com> wrote:=0A>> =0A>>=
 Hi Christian,=0A>> =0A>> Sorry to chime in late - I was out.=0A>> Recent=
ly, I had also been thinking about splitting more static configure data (=
like the action)=0A>> from highly transactional data like the activations=
. My reason was, however, to have an easier way=0A>> forward to multi-dat=
acenter deployments.=0A>> =0A>> Regardless of the motivation for the spli=
t into activation-db and "static-db" I would like to make=0A>> this comme=
nt:=0A>> Now, that we split out the activation-db into its own separate A=
PI we should take the opportunity=0A>> to design this API (interface) tha=
t truly allows pluggable implementations. I am particularly=0A>> interest=
ed in an Elastic Search impl for the activation-db (as I believe that ES =
lends itself well=0A>> to activations-workloads). There might be other in=
teresting impls. Point being: let's iterate a bit=0A>> over the interface=
 to make sure it can be implemented in non-CouchDB deployments. I am conv=
inced=0A>> this will be helpful in the future in order to evolve OW.=0A>>=
 In the light of the above it would be useful to design the activation-db=
 interface with a mindset=0A>> that does not assume that activation-db an=
d static-db are the same physical storage or even the=0A>> same storage t=
echnology. Mentioning this to avoid assumptions about ID-semantics, joins=
, etc.=0A>> =0A>> Wdyt?=0A>> Michael=0A>> =0A>> Sent from a mobile device=
=0A>> _____________________________=0A>> From: apache@cbickel.de<mailto:a=
pache@cbickel.de>=0A>> Sent: Wednesday, April 12, 2017 1:37 PM=0A>> Subje=
ct: Re: Save activations in a new activations DB=0A>> To: <dev@openwhisk.=
apache.org<mailto:dev@openwhisk.apache.org>>=0A>> =0A>> Hi OpenWhisk deve=
lopers,=0A>> =0A>> the first Pull Request I mentioned in the last mail=0A=
>> (https://github.com/openwhisk/openwhisk/pull/2123) is merged now.=0A>>=
 If you use one of the three environments in open, the default will still=
 be, that activations are=0A>> saved in the whisks-db. But now you can se=
t that flag to true.=0A>> The next Pull request is opened now: https://gi=
thub.com/openwhisk/openwhisk/pull/2134=0A>> It sets the default of the th=
ree environments to use the activations-db. It also requires this=0A>> va=
riable in your environment. It still can be set to true or false.=0A>> =
=0A>> If you have some OpenWhisk deployments, I'd suggest to set the vari=
able to false, do your migration=0A>> and set it to true afterwards.=0A>>=
 The migration is described here: https://gist.github.com/cbickel/37e6519=
65781b27de245eac0ce831a53=0A>> =0A>> My next, and last, PR will be to rem=
ove the ability to use that flag.=0A>> =0A>> If you have any problems wit=
h your migration or concerns just reach out to me.=0A>> =0A>> Greetings C=
hristian=0A>> =0A>> April 10, 2017 2:56 PM, apache@cbickel.de<mailto:apac=
he@cbickel.de> wrote:=0A> =0A> Hi OpenWhisk developers,=0A> =0A> some wee=
ks ago, I started working on some database performance and one item affec=
ts the=0A> whisks-database.=0A> All actions, rules, triggers, packages an=
d activations are stored in this database. Because of all=0A> the activat=
ions, this database becomes really huge over time.=0A> To keep the databa=
se with the artifacts, created by the users (actions, ...), small we had =
the idea=0A> to put all activations into another database. The activation=
s-db.=0A> =0A> To solve the problem of the migration of existing OpenWhis=
k deployments, my proposal is to make the=0A> split configurable with ans=
ible.=0A> PR 2123 (https://github.com/openwhisk/openwhisk/pull/2123) crea=
tes this flag. By default, the=0A> whisks database will be used to store =
activations.=0A> After some time, I'll open another PR, that changes the =
default to use the activations-db.=0A> Again, after some time, I'll open =
a third PR to remove this flag again. By default, all activations=0A> wil=
l go into the activations-db.=0A> The reason for removing this flag again=
 is, that it will be easier to maintain only one=0A> configuration: the c=
onfiguration to use two seperate dbs.=0A> And we think, that all owners o=
f OpenWhisk deployments agree that it would be good to split=0A> configur=
ation data of the user (actions, ...) and logs (activations, ...).=0A> =
=0A> If you need to migrate your deployment, it would work with the steps=
 on the following document:=0A> https://gist.github.com/cbickel/37e651965=
781b27de245eac0ce831a53=0A> (https://gist.github.com/cbickel/37e651965781=
b27de245eac0ce831a53)=0A> =0A> Does anyone have any concerns about this c=
hange or any comments?=0A> =0A> Greetings Christian