Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5AB8F200C2B for ; Thu, 2 Mar 2017 22:10:58 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 59247160B6F; Thu, 2 Mar 2017 21:10:58 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5272E160B6A for ; Thu, 2 Mar 2017 22:10:57 +0100 (CET) Received: (qmail 10106 invoked by uid 500); 2 Mar 2017 21:10:56 -0000 Mailing-List: contact user-help@predictionio.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@predictionio.incubator.apache.org Delivered-To: mailing list user@predictionio.incubator.apache.org Received: (qmail 10095 invoked by uid 99); 2 Mar 2017 21:10:56 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Mar 2017 21:10:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 0A2771A0257 for ; Thu, 2 Mar 2017 21:10:56 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.8 X-Spam-Level: * X-Spam-Status: No, score=1.8 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_SORBS_SPAM=0.5, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=phoenix-opsgroup-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id xtRaQGspdXRd for ; Thu, 2 Mar 2017 21:10:54 +0000 (UTC) Received: from mail-qk0-f172.google.com (mail-qk0-f172.google.com [209.85.220.172]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 4039A5FDD4 for ; Thu, 2 Mar 2017 21:10:54 +0000 (UTC) Received: by mail-qk0-f172.google.com with SMTP id m67so30270123qkf.2 for ; Thu, 02 Mar 2017 13:10:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=phoenix-opsgroup-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=2gghV/TQsfzDverVKdXcte4AtrQmDg+0+fwy8j9vLGg=; b=KtVMxA8YRKJTdKBCGdXLyf8vFnNj+ztYAFGWNv0sNITPMBuuK4pecpFmqhBkfzSx/q Sa1tDSlIOYTnOSaSOorJU0tyBrL7rqUyas695RwMi/L0arQvrt7jVicPodhGHIO9IKiL ioUPoJD1dOQe8N0wlFDFDiKDhvxOv6e5HO5qSOkinf98jzIkvT5dhW+nUARHeOhb/Wm9 Wl/0yfgm39lA/e3eQqwzrW6dYIDuhmLc4fUG3AG/+qya2NlemG3JtANRMNEU/QqoFrPY EJA/NQ4chy2KQlM6sYGBsqJWbLvAXj939j2Gel/79GpxkXiBGSa1AvBjlmBlxfwWFmil 3ZWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=2gghV/TQsfzDverVKdXcte4AtrQmDg+0+fwy8j9vLGg=; b=dh+HIPjIl1sx/Ci3E2eJeEGXCkoeDp7SQbHFbEGGfz78f7j56hhS8TtQzXaQK1pxd5 oZP5ttqzSvNB0KkG4OlG9gGCpQu1N7xkzlRCeDXX2vq3XqSNB2Rmt3Peu5dRNoZIFFS3 VWS3rzZn6lM0I/VRr4gXWNwyHNnCwrQmTJccRNxjdBaD7C7UJAAcSMzkXXrnvjwKQXK0 XtmPKunzYNdATMkz+A2ZZzniVOQlqtJzM9WyK6dCG1Vz1/RFOOeu7y0N2UQ7e14Yl2s2 kdowUi0fKOUGuoEsNr6tsIAyNijYMcrMFFd67sE6f7UMgVUVrEyNB/Rm+XV8SgPiu9Pv F5cw== X-Gm-Message-State: AMke39nla/DDP9TjzW+gc9RvbsCCDaoWYguF+fxwPvR3IxzZqvnLPBVVslrXH7RKMJwe934g71np2GUVzAs7Mg== X-Received: by 10.200.53.237 with SMTP id l42mr19096995qtb.264.1488489053596; Thu, 02 Mar 2017 13:10:53 -0800 (PST) MIME-Version: 1.0 Received: by 10.12.178.139 with HTTP; Thu, 2 Mar 2017 13:10:53 -0800 (PST) In-Reply-To: References: From: "Miller, Clifford" Date: Thu, 2 Mar 2017 16:10:53 -0500 Message-ID: Subject: Re: PredictionIO with remote Spark and Elasticsearch To: user@predictionio.incubator.apache.org Content-Type: multipart/alternative; boundary=001a113d419a4af54f0549c5dd1e archived-at: Thu, 02 Mar 2017 21:10:58 -0000 --001a113d419a4af54f0549c5dd1e Content-Type: text/plain; charset=UTF-8 I found some old references of folks having the same issue as me. They indicated that the AWS Elasticsearch Service only supports HTTP and not TCP. If this is true then it means that AWS Elasticsearch has very limited usefulness. Has anyone else ran into this? On Thu, Mar 2, 2017 at 1:26 PM, Miller, Clifford < clifford.miller@phoenix-opsgroup.com> wrote: > I'm able run pio train although the pio train -- --master > spark://your_master_url did not work. I'm using Spark on Yarn so I was > able to get pio train -- --master yarn://URL to work after I copied the > elastic search configuration from my CDH cluster. > > I'm still struggling with integrating this with AWS elasticsearch. Does > anyone have an example of how this should be configured. > > FYI, the EC2 instance that I'm running PredictionIO on can access it from > the command line: "curl -X GET ". > > > On Wed, Mar 1, 2017 at 11:44 AM, Donald Szeto wrote: > >> Hi Clifford, >> >> To use a remote Spark cluster, use passthrough command line arguments on >> the CLI, e.g. >> >> pio train -- --master spark://your_master_url >> >> Anything after a lone -- will be passed to spark-submit verbatim. For >> more information try "pio help". >> >> To use a remote Elasticsearch cluster, please refer to examples in >> "conf/pio-env.sh" where you could find a variable to set the remote host >> name or IP of your ES cluster. >> >> Regards, >> Donald >> >> On Tue, Feb 28, 2017 at 12:57 PM Miller, Clifford < >> clifford.miller@phoenix-opsgroup.com> wrote: >> >>> I currently have Cloudera cluster (Hadoop, Spark, Hbase...) setup on >>> AWS. I have PredictionIO installed on a different EC2 instance. I've been >>> able to successfully configure it to use HDFS for model storage and to >>> store events in Hbase from the cluster. Spark and Elasticsearch are >>> installed locally on the PredictionIO EC2 instance. I have the following >>> questions: >>> >>> How can I configure PredictionIO to utilize the Spark on the Cloudera >>> cluster? >>> How can I configure PredictionIO to utilize a remote Elasticsearch >>> domain? I'd like to use the AWS Elasticsearch service if possible. >>> >>> Thanks >>> >>> >>> -- >>> Clifford Miller >>> Mobile | 321.431.9089 >>> >> > > > -- > Clifford Miller > Mobile | 321.431.9089 > -- Clifford Miller Mobile | 321.431.9089 --001a113d419a4af54f0549c5dd1e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I found some old references of folks having the same issue= as me.=C2=A0 They indicated that the AWS Elasticsearch Service only suppor= ts HTTP and not TCP.=C2=A0 If this is true then it means that AWS Elasticse= arch has very limited usefulness.=C2=A0 Has anyone else ran into this?
=

O= n Thu, Mar 2, 2017 at 1:26 PM, Miller, Clifford <cliffo= rd.miller@phoenix-opsgroup.com> wrote:
I'm able run pio train although the pio = train -- --master spark://your_master_url did not work.=C2=A0 I'm using= Spark on Yarn so I was able to get pio train -- --master yarn://URL to wor= k after I copied the elastic search configuration from my CDH cluster.
=
I'm still struggling with integrating this with AWS elas= ticsearch.=C2=A0 Does anyone have an example of how this should be configur= ed. =C2=A0

FYI, the EC2 instance that I'm runn= ing PredictionIO on can access it from the command line: "curl -X GET <AWS = Elasticsearch endpoint URL>".=C2=A0
=C2=A0

On Wed, Mar 1, 2017 at 11:44 AM, Donald Szeto <do= nald@apache.org> wrote:
Hi Clifford,

To use a remote Spark cl= uster, use passthrough command line arguments on the CLI, e.g.
pio train -- --master spark://your_master_url

Anything after a lone -- will be passed to spark-submit verbatim. = For more information try "pio help".

To = use a remote Elasticsearch cluster, please refer to examples in "conf/= pio-env.sh" where you could find a variable to set the remote host nam= e or IP of your ES cluster.

Regards,
Don= ald

On Tue, Feb 28, = 2017 at 12:57 PM Miller, Clifford <clifford.miller@phoenix-opsgroup.= com> wrote:
I currently have Cloud= era cluster (Hadoop, Spark, Hbase...) setup on AWS.=C2=A0 I have Prediction= IO installed on a different EC2 instance.=C2=A0 I've been able to succe= ssfully configure it to use HDFS for model storage and to store events in H= base from the cluster.=C2=A0 Spark and Elasticsearch are installed locally = on the PredictionIO EC2 instance.=C2=A0 I have the following questions:
How can I config= ure PredictionIO to utilize the Spark on the Cloudera cluster? =C2=A0
=
How ca= n I configure PredictionIO to utilize a remote Elasticsearch domain?=C2=A0 = I'd like to use the AWS Elasticsearch service if possible.

Thanks


--
Clifford Miller
Mobile | 321.431.9089
<= div class=3D"m_-3207218328229934054m_-3109020664592786846gmail_msg">



--
=
Clifford Miller
=
Mobile | 321.431.9089



--
=
Clifford Miller
Mobile | 321.431.9089
--001a113d419a4af54f0549c5dd1e--