Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3DC6D200C48 for ; Thu, 6 Apr 2017 19:13:38 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 3C372160B91; Thu, 6 Apr 2017 17:13:38 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id DC422160B81 for ; Thu, 6 Apr 2017 19:13:36 +0200 (CEST) Received: (qmail 13031 invoked by uid 500); 6 Apr 2017 17:13:36 -0000 Mailing-List: contact user-help@predictionio.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@predictionio.incubator.apache.org Delivered-To: mailing list user@predictionio.incubator.apache.org Received: (qmail 13009 invoked by uid 99); 6 Apr 2017 17:13:36 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Apr 2017 17:13:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 36A79183991 for ; Thu, 6 Apr 2017 17:13:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.18 X-Spam-Level: *** X-Spam-Status: No, score=3.18 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_BADIPHTTP=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id ox2OVmC5uNup for ; Thu, 6 Apr 2017 17:13:32 +0000 (UTC) Received: from mail-qk0-f189.google.com (mail-qk0-f189.google.com [209.85.220.189]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id AB3815F1A1 for ; Thu, 6 Apr 2017 17:13:31 +0000 (UTC) Received: by mail-qk0-f189.google.com with SMTP id p22so5501710qka.1 for ; Thu, 06 Apr 2017 10:13:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:message-id:in-reply-to:references:subject :mime-version; bh=BtTsLwXKryoSLjHBUF9IoWp+7XGUWxkMZeyUDM4LXlA=; b=cqiPRz1OHO6bLfGzTElkZwgFFOL/tGTkYBa8a5XXcsn+XhMVb1ijzd0ptM5oZ4Tdbl xs0um4K8RWWZB4dZlsJwSHVN5xk+S+xGSUUZ4F2yWryI7rdUJYUc1OOFRaEVVBVdNt12 ZuN/9uOrBiVirVhoMinFooFoQjgwAo5jtkFf5GIfugPfbS+F/Bi29XbNrHoyY8MzoDYI TcJs7NFx6wufEmq/KovfUeIuseH2RYONZROFAnlXwFzET6zx8YKELVKTlbjv+AX3nW6f dFE+uzh6e2OwMmGhZBcktWiFH6YBBzRyOpLIz/fX+IUtru26jdwvq+FVpu0AidYLKKZ8 K4NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:message-id:in-reply-to :references:subject:mime-version; bh=BtTsLwXKryoSLjHBUF9IoWp+7XGUWxkMZeyUDM4LXlA=; b=ullvhTBNpiAHBT17btdE0jXPoHxw+K84LU6mq4ISAxzm3VZEnAn9ptmGCIwSMhtBWS iwu53gDOtjfyZK3FuZbBWfONVfzI6VU84N/mPUJyidMU2RwMpYcfkadzqnF1dkDg7Ovn okGYOP2kH8LHVU3oEzFj9YNw+NmCsTQwJDDCiAmWSG/lCeoPX1TWj3yXc2VXEp9Vt8FN 3DsdXGfNaxLRScTTg4HNFso9EBF+3lQ+FU+e/15QOlVxKl3m4NhhGNOYL8orgmLY+V3U 1rT6uW1D0E1HF7tP/sA904KV8ZlCZfosu7tlWMtl7EJxmKUDSJz8ttqYWJ2+xPfZK9oe tMQg== X-Gm-Message-State: AFeK/H1WJNXjaUPeHRNZRPDwyMZ5wMr5uGnfKZ5Vay3A0VgeTjQavbqQxT/A+GJqM1Al+jEIqUbw X-Received: by 10.157.55.247 with SMTP id x110mr1159995otb.6.1491498811405; Thu, 06 Apr 2017 10:13:31 -0700 (PDT) X-Google-Already-Archived: Yes X-Google-Already-Archived-Group-Id: 62ef9726c8 X-Google-Doc-Id: d8776cb0d3f43 X-Google-Thread-Id: 87af3da2fdec1e3f X-Google-Message-Url: http://groups.google.com/group/actionml-user/msg/d8776cb0d3f43 X-Google-Thread-Url: http://groups.google.com/group/actionml-user/t/87af3da2fdec1e3f X-Google-Web-Client: true Date: Thu, 6 Apr 2017 10:13:30 -0700 (PDT) From: juha.syrjala@gmail.com To: actionml-user Cc: user@predictionio.incubator.apache.org Message-Id: In-Reply-To: References: <6a38952f-6f23-445f-bea8-e97fbc04aa4b@googlegroups.com> <76ce9fa7-c6b0-4695-8b2f-724dee38653a@googlegroups.com> Subject: Re: Adding a new application is slow MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_4439_1136049101.1491498810917" X-Google-Token: ELrumccF2vG2dC1FLvA0 X-Google-IP: 62.78.145.39 archived-at: Thu, 06 Apr 2017 17:13:38 -0000 ------=_Part_4439_1136049101.1491498810917 Content-Type: multipart/alternative; boundary="----=_Part_4440_1811535565.1491498810918" ------=_Part_4440_1811535565.1491498810918 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I'll try make a patch for this to PredicitionIO during the weekend if I=20 have the time. On Thursday, April 6, 2017 at 8:07:31 PM UTC+3, pat wrote: > > Hmm, interesting. I have heard of this being a problem in PredictionIO bu= t=20 > haven=E2=80=99t heard this explanation before IIRC. > > > On Apr 6, 2017, at 9:47 AM, juha.s...@gmail.com wrote: > > Ah, I just found the problem: a fresh AWS instance takes a long time to= =20 > generate enough entropy so that a random access key can be generated.=20 > > Related github issue:=20 > https://github.com/apache/incubator-predictionio/issues/253 > > More discussion about the a similar problem:=20 > https://groups.google.com/forum/#!searchin/actionml-user/aws%7Csort:relev= ance/actionml-user/N9o2jBkyF_w/bUYr5U8zFQAJ > > > On Thursday, April 6, 2017 at 7:33:49 PM UTC+3, juha.s...@gmail.com wrote= : >> >> Hi, >> >> Yes, it is clear that the AWS image is for dev only. I was just wonderin= g=20 >> the slowness of the a newly created instance when it has no data.=20 >> >> However can't be that the HDFS is out of space. I ran those commands=20 >> right after creating a new AWS instance. I don't have any engines runnin= g=20 >> or configured, and I have not added any data yet to the system. The mach= ine=20 >> seems to be completely idle while adding the new application. >> >> Current HDFS usage: >> >> $ ./hadoop fs -df -h >> >> Filesystem Size Used Available Use% >> >> hdfs://127.0.0.1:9000 7.7 G 220 K 1.9 G 0% >> >> Also system has plenty of free memory >> >> $ free -h >> >> total used free shared buff/cache =20 >> available >> >> Mem: 14G 2.0G 12G 8.6M 404M = =20 >> 12G >> >> Swap: 0B 0B 0B >> >> >> >> On Thursday, April 6, 2017 at 6:47:53 PM UTC+3, pat wrote: >>> >>> Yes, this is sometimes true and it could be a couple things. Some=20 >>> operations in HDFS are slow to complete since they require replication.= =20 >>> Even starting a cluster can take minutes. Is it a problem? Is it possib= le=20 >>> HBase is getting full (check remaining data in HDFS). or needs to be=20 >>> scaled? The all-in-one instance on AWS only uses vertical scaling by nu= mber=20 >>> of cores, amount of memory, and disk space since all services run on on= e=20 >>> machine. You can =E2=80=9Cstop=E2=80=9D the instance and change the typ= e to a larger=20 >>> instance then =E2=80=9Cstart=E2=80=9D it to get it scaled. You have to = snapshot the disk to=20 >>> scale disk by copying to a new one and attaching the new volume. >>> >>> I hope it was clear that the AWS AMI was meant for dev, not production. >>> >>> On Apr 6, 2017, at 6:12 AM, juha.s...@gmail.com wrote: >>> >>> Hi, >>> >>> I am using ready-made AWS image from=20 >>> http://actionml.com/docs/awssetupguide with r3.large instance in=20 >>> eu-central-1 region. >>> >>> I followed installation instructions to the letter. Integration test=20 >>> passes but takes many minutes. Probably because creating a new a new ap= p=20 >>> takes so long. >>> >>> Then I tried creating a new application and that takes about 10 minutes= .=20 >>> Is this normal?=20 >>> >>> What this command is actually doing? I had assumed that it would just= =20 >>> create few records to the databases, which should take just a dozen sec= onds=20 >>> or so. >>> >>> Here is a log for creating a new application. Most of the time is spent= =20 >>> lines 'HBLEvents' and '[App$] Initialized Event...' >>> >>> $ time pio app new foobar123 >>> [INFO] [HBLEvents] The table pio_event:events_3 doesn't exist yet.=20 >>> Creating now... >>> [INFO] [App$] Initialized Event Store for this app ID: 3. >>> [INFO] [App$] Created new app: >>> [INFO] [App$] Name: foobar123 >>> [INFO] [App$] ID: 3 >>> [INFO] [App$] Access Key: .... >>> >>> real 9m58.891s >>> user 0m9.632s >>> sys 0m0.352s >>> >>> >>> --=20 >>> Juha Syrj=C3=A4l=C3=A4 >>> >>> --=20 >>> You received this message because you are subscribed to the Google=20 >>> Groups "actionml-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send= =20 >>> an email to actionml-use...@googlegroups.com. >>> To post to this group, send email to action...@googlegroups.com. >>> To view this discussion on the web visit=20 >>> https://groups.google.com/d/msgid/actionml-user/6a38952f-6f23-445f-bea8= -e97fbc04aa4b%40googlegroups.com=20 >>> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> > --=20 > You received this message because you are subscribed to the Google Groups= =20 > "actionml-user" group. > To unsubscribe from this group and stop receiving emails from it, send an= =20 > email to actionml-use...@googlegroups.com . > To post to this group, send email to action...@googlegroups.com=20 > . > To view this discussion on the web visit=20 > https://groups.google.com/d/msgid/actionml-user/76ce9fa7-c6b0-4695-8b2f-7= 24dee38653a%40googlegroups.com=20 > > . > For more options, visit https://groups.google.com/d/optout. > > ------=_Part_4440_1811535565.1491498810918 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
I'll try make a patch for this to PredicitionIO during= the weekend if I have the time.

On Thursday, April 6, 2017 at 8:07:= 31 PM UTC+3, pat wrote:
Hmm, interesting. I have heard of this being a= problem in PredictionIO but haven=E2=80=99t heard this explanation before = IIRC.


On Apr 6, 2017, at 9:47 AM, juha.s...@gm= ail.com wrote:

Ah, I just found the prob= lem: a fresh AWS instance takes a long time to generate enough entropy so t= hat a random access key can be generated.=C2=A0


Mor= e discussion about the a similar problem:=C2=A0https://groups.google.com/forum/#!searchin/acti= onml-user/aws%7Csort:relevance/actionml-user/N9o2jBkyF_w/bUYr5U8z= FQAJ


On Thursday, April 6, 2017 at 7:33:49 PM UTC+3, juha.s.= ..@gmail.com wrote:
Hi,

Ye= s, it is clear that the AWS image is for dev only. I was just wondering the= slowness of the a newly created instance when it has no data.=C2=A0
<= div>
However can't be that the HDFS is out of space. I ra= n those commands right after creating a new AWS instance. I don't have = any engines running or configured, and I have not added any data yet to the= system. The machine seems to be completely idle while adding the new appli= cation.

Current HDFS usage:

$ .= /hadoop fs -df -h

Filesystem=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 Size =C2=A0 Used=C2=A0 Available=C2=A0 Use%

=

hdfs://127.0.0.1:9000=C2=A0 7.7 G=C2=A0 220 K=C2=A0 =C2=A0 =C2=A0 1.9 G=C2=A0= =C2=A0 0%


Also system has plenty of fr= ee memory

$ free -h

=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 total=C2=A0 =C2=A0 =C2=A0 =C2=A0 used=C2=A0= =C2=A0 =C2=A0 =C2=A0 free=C2=A0 =C2=A0 =C2=A0 shared=C2=A0 buff/cache =C2= =A0 available

Mem:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 14G=C2=A0 =C2=A0 =C2=A0 =C2=A0 2.0G =C2=A0 =C2=A0 =C2=A0 =C2=A0 12G=C2= =A0 =C2=A0 =C2=A0 =C2=A0 8.6M=C2=A0 =C2=A0 =C2=A0 =C2=A0 404M =C2=A0 =C2=A0= =C2=A0 =C2=A0 12G

Swap:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 0B=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0B=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 0B



On Thursday, Apri= l 6, 2017 at 6:47:53 PM UTC+3, pat wrote:
Yes, this is sometimes true and i= t could be a couple things. Some operations in HDFS are slow to complete si= nce they require replication. Even starting a cluster can take minutes. Is = it a problem? Is it possible HBase is getting full (check remaining data in= HDFS). or needs to be scaled? The all-in-one instance on AWS only uses ver= tical scaling by number of cores, amount of memory, and disk space since al= l services run on one machine. You can =E2=80=9Cstop=E2=80=9D the instance = and change the type to a larger instance then =E2=80=9Cstart=E2=80=9D it to= get it scaled. You have to snapshot the disk to scale disk by copying to a= new one and attaching the new volume.

I hope it was cle= ar that the AWS AMI was meant for dev, not production.

On Apr 6, 2017, at 6:12 AM, juha.s...@gmail.com w= rote:

Hi,

I am using ready-mad= e AWS image from=C2=A0http://actionml.com/d= ocs/awssetupguide=C2=A0with r3.large instance in eu-c= entral-1 region.

I followed installa= tion instructions to the letter.=C2=A0Integration test passes = but takes many minutes. Probably because creating a new a new app takes so = long.

Then I tried creating a new ap= plication and that takes about 10 minutes. Is this normal?=C2=A0

What this command is actually doing? I had ass= umed that it would just create few records to the databases, which should t= ake just a dozen seconds or so.

Here= is a log for creating a new application. Most of the time is spent lines &= #39;HBLEvents' and '[App$] Initialized Ev= ent...'

$ time pio app new fooba= r123
[INFO] [HBLEvents] The table pio_event:events_3 doesn= 't exist yet. Creating now...
[INFO] [App$] Initialize= d Event Store for this app ID: 3.
[INFO] [App$] Created ne= w app:
[INFO] [App$] =C2=A0 =C2=A0 =C2=A0 Name: foobar123<= /span>
[INFO] [App$] =C2=A0 =C2=A0 =C2=A0 =C2=A0 ID: 3
[INFO] [App$] Access Key: ....

real=099m58.891s
user=090m9.632s
sys=090m0.352s


--=C2=A0
<= div style=3D"margin:0px;font-style:normal;font-weight:normal;font-size:11px= ;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">Juha Syrj=C3=A4l=C3=A4

--=C2=A0
You received this m= essage because you are subscribed to the Google Groups "actionml-user&= quot; group.
To unsubscribe from this group and stop receiving emails from it, send= an email to=C2=A0actionml-use...@googlegroups.com.
To post to this group, send email to=C2=A0action...@googlegroups= .com.
To view this discussion on the web visit=C2=A0https://groups.google.com/d/msgi= d/actionml-user/6a38952f-6f23-445f-bea8-e97fbc04aa4b%40googlegrou= ps.com.
For more options, visit=C2=A0https://gr= oups.google.com/d/optout.

=
--
You received this message because you are subscribed to the Google Groups &= quot;actionml-user" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to actionml-use...@googlegroups.com.
To post to this group, send email to action...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionm= l-user/76ce9fa7-c6b0-4695-8b2f-724dee38653a%40googlegroups.com.
For more options, visit https://groups.google.com/= d/optout.

------=_Part_4440_1811535565.1491498810918-- ------=_Part_4439_1136049101.1491498810917--