Mailing-List: contact dev-help@brooklyn.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@brooklyn.apache.org
MIME-Version: 1.0
In-Reply-To: <CAJUGTHwJnuT4OL9RTem+nG4S8xDGPebC1-y339Wcxysja7krrA@mail.gmail.com>
References: <CAJUGTHwJnuT4OL9RTem+nG4S8xDGPebC1-y339Wcxysja7krrA@mail.gmail.com>
From: Brian Long <brianblong@gmail.com>
Date: Thu, 5 Oct 2017 16:00:17 -0400
Message-ID: <CAJUGTHy7_398mEQgete1uSWxzTXX-L=13jcHg1q-zZ6b-dexug@mail.gmail.com>
Subject: Re: Issue running Brooklyn in Vagrant
To: dev@brooklyn.apache.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 05 Oct 2017 20:00:54 -0000

Hi all,

Sorry for the spam -- I'm just getting used to the plain text mailing
list.  If you prefer HTML formatting, I've uploaded my message here:
https://gist.github.com/b-long/9156a3696d9c30c14a092fe4b0a01381

Thanks,
b-long

On Thu, Oct 5, 2017 at 3:12 PM, Brian Long <brianblong@gmail.com> wrote:
> Hi Thomas,
>
> Sorry for my delayed response and thanks for your all of your work.
>
> Vagrant related
>
> I see that the brooklyn-dist PR that you referenced [0] was indeed merged
> and I agree it appears that it would fix the symlink issue I=E2=80=99ve o=
bserved.
> However, when I download the Vagrant tar archive, I=E2=80=99m still getti=
ng the same
> MD5 sum that was produced back on September 27th:
> 331ab054e08a0b8c0480621b2f2adfe4 . To download the Vagrant archive, I=E2=
=80=99m
> using the command found at https://brooklyn.apache.org/#get-started :
> curl -SL --output apache-brooklyn-0.12.0-vagrant.tar.gz
> "https://www.apache.org/dyn/closer.lua?action=3Ddownload&filename=3Dbrook=
lyn/apache-brooklyn-0.12.0/apache-brooklyn-0.12.0-vagrant.tar.gz"
>
> So, this yields a new question about the update process for mirrors. Is i=
t
> working? Or perhaps I didn=E2=80=99t understand the release model with
> brooklyn-dist.
>
> Anyhow, I still need to apply the fix I mentioned previously (linking and
> restarting the service after vagrant up) [1]. Unfortunately, I=E2=80=99m =
still
> experiencing issues with the bento/centos-7.3 boxes too, so I=E2=80=99m c=
ontinuing
> to change the box variable to geerlingguy/centos7. I appreciate your help=
 in
> debugging this. You referenced an error message with the text =E2=80=9CCo=
uld not
> find the X.Org or XFree86 Window System, skipping.=E2=80=9D . Is this exp=
ected to
> work on macOS? Is there a simple method of testing the integration?
>
> My approach is still working for me, and once the service is running I ca=
n
> access Brooklyn from my host, at http://localhost:8081 .
>
> Deployment related
>
> After getting Brooklyn started, the thing I=E2=80=99m eager to use more t=
han
> anything is Ambari. Here are the steps I=E2=80=99ve done, attempting depl=
oyment:
>
> # Install the Brooklyn command line tool
> brew install apache-brooklyn-cli
>
> # Authenticate to Brooklyn
> br login http://localhost:8081/
>
> # Get the Brooklyn Ambari repo
> git clone https://github.com/brooklyncentral/brooklyn-ambari.git
>
> cd brooklyn-ambari
>
> # Add Ambari to Brooklyn's catalog
> br add-catalog catalog.bom
>
> Next, in AWS, I had to establish my Security Group. I first created a
> security group called test-ambari and opened ports 8080 according to the
> brooklyn-ambari README. This failed, reasonably, since I know Ambari need=
s
> 8440 for coordinating agent nodes. In a third or fourth iteration, I saw =
an
> error that referenced port 22. At that point, I decided to just open thin=
gs
> up a lot wider, in hopes that the networking would get everything working
> properly. My final Security Group, with speculative rules for TCP
> connections from anywhere is the following:
>
> * 8080   # The Ambari web UI
> * 24007  # I saw mention of GlusterFS in the logs
> * 24008  # Again, for GlusterFS
> * 8441   # Ambari Registration and Heartbeat Port
> * 22     # It seems the Brooklyn control machine has to SSH to Ambari nod=
es
> * 8440   # Ambari Agent orchestration
> * 2181   # ZooKeeper
>
> Next, from the Brooklyn web UI, I navigate to the Composer / editor and
> enter this YAML:
>
> location:
>   jclouds:aws-ec2:
>     # edit these to use your credential (or delete if credentials specifi=
ed
> in brooklyn.properties)
>     identity:     <redacted>
>     credential:   <redacted>
>
>     region:       us-east-2
>
>     # we want Ubuntu, with a lot of RAM
>     osFamily:     ubuntu
>     minRam:       8gb
>
>     # set up this user and password (default is to authorize a public key=
)
>     user:         sample
>     password:     s4mpl3
>
> services:
> - type: ambari-cluster-application
>   name: Ambari Cluster
>   brooklyn.config:
>     securityGroup: test-ambari
>     initialSize: 3
>     services:
>     - FALCON
>     - FLUME
>     - GANGLIA
>     - HBASE
>     - HDFS
>     - KAFKA
>     - KERBEROS
>     - MAPREDUCE2
>     - OOZIE
>     - PIG
>     - SLIDER
>     - SQOOP
>     - YARN
>     - ZOOKEEPER
>
> After clicking the Deploy button, 4 instances are created in AWS. Here=E2=
=80=99s a
> screenshot:
> https://drive.google.com/file/d/0B91hmcyKP8SzLTRHcTdTZFkzWG8/view
>
> I can watch, in the Brooklyn UI, that there is communication between the
> Brooklyn Vagrant VM and these EC2 hosts. Screenshot:
> https://drive.google.com/file/d/0B91hmcyKP8SzR0R2bFFDYWhCUk0/view
>
> Eventually, all 3 Ambari agent nodes seem to be in a happy state.
> Unfortunately, the Ambari Server is not:
>
> Screenshot:
> https://drive.google.com/file/d/0B91hmcyKP8SzZzZ1XzF1UmZQZkU/view
>
> Screenshot:
> https://drive.google.com/file/d/0B91hmcyKP8SzYkQ4N1dXeVVnUEU/view
>
> When I attempt to open port 8080 (Ambari web UI) on the Ambari server, I=
=E2=80=99m
> receiving an error. Screenshot:
> https://drive.google.com/file/d/0B91hmcyKP8SzUkF2MGJWeFNCNDQ/view
>
> I know that at one point, things were working to a greater degree than I =
am
> seeing now. Unfortunately, I don=E2=80=99t recall how I managed to accomp=
lish that
> (maybe using the newer BOM file from this PR [2] or perhaps it was the BY=
ON
> / Vagrant nodes). I was, at one point, able to login to Ambari. I found i=
t
> really great that there was a Brooklyn generated password for the service=
.
> As a last ditch effort for today, I did try the former on AWS and didn=E2=
=80=99t
> have success.
>
> Wrapping up
>
> My team and I need move quickly, and unfortunately, if I can=E2=80=99t ge=
t things
> working with Brooklyn soon - I=E2=80=99ll need to change my approach.
>
> I think the Brooklyn team has a very serious opportunity if you can suppo=
rt
> Ambari. I say that, since I=E2=80=99m not totally satisfied with the job =
that
> Hortonworks is doing supporting FLOSS deployments of the Hadoop ecosystem
> and Ambari. Presumably, if you can support the baseline, you=E2=80=99ll i=
nherit a
> variety of other services.
>
> Furthermore, since Brooklyn has first-class support for load balancing an=
d
> resizing, Hadoop users get serious value in being able to scale workloads=
.
> The possibility of developing / testing distributed workloads on local VM=
s
> is another value not so well supported in the open source.
>
> Lastly, if we could get Apache Ranger working (via an Ambari + Brooklyn
> configuration), then Brooklyn could provide a very rich feature set for
> securing clusters, data, and custom endpoints. Perhaps some other Apache
> folks would be willing to help with this integration?
>
> Thanks for all your help,
> b-long
>
> 0: https://github.com/apache/brooklyn-dist/pull/111
> 1: https://gist.github.com/b-long/ab096f45a7867574b74f01adff9f6c22
> 2: https://github.com/brooklyncentral/brooklyn-ambari/pull/126