brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Long <brianbl...@gmail.com>
Subject Re: Issue running Brooklyn in Vagrant
Date Thu, 05 Oct 2017 20:00:17 GMT
Hi all,

Sorry for the spam -- I'm just getting used to the plain text mailing
list.  If you prefer HTML formatting, I've uploaded my message here:
https://gist.github.com/b-long/9156a3696d9c30c14a092fe4b0a01381

Thanks,
b-long

On Thu, Oct 5, 2017 at 3:12 PM, Brian Long <brianblong@gmail.com> wrote:
> Hi Thomas,
>
> Sorry for my delayed response and thanks for your all of your work.
>
> Vagrant related
>
> I see that the brooklyn-dist PR that you referenced [0] was indeed merged
> and I agree it appears that it would fix the symlink issue I’ve observed.
> However, when I download the Vagrant tar archive, I’m still getting the same
> MD5 sum that was produced back on September 27th:
> 331ab054e08a0b8c0480621b2f2adfe4 . To download the Vagrant archive, I’m
> using the command found at https://brooklyn.apache.org/#get-started :
> curl -SL --output apache-brooklyn-0.12.0-vagrant.tar.gz
> "https://www.apache.org/dyn/closer.lua?action=download&filename=brooklyn/apache-brooklyn-0.12.0/apache-brooklyn-0.12.0-vagrant.tar.gz"
>
> So, this yields a new question about the update process for mirrors. Is it
> working? Or perhaps I didn’t understand the release model with
> brooklyn-dist.
>
> Anyhow, I still need to apply the fix I mentioned previously (linking and
> restarting the service after vagrant up) [1]. Unfortunately, I’m still
> experiencing issues with the bento/centos-7.3 boxes too, so I’m continuing
> to change the box variable to geerlingguy/centos7. I appreciate your help in
> debugging this. You referenced an error message with the text “Could not
> find the X.Org or XFree86 Window System, skipping.” . Is this expected to
> work on macOS? Is there a simple method of testing the integration?
>
> My approach is still working for me, and once the service is running I can
> access Brooklyn from my host, at http://localhost:8081 .
>
> Deployment related
>
> After getting Brooklyn started, the thing I’m eager to use more than
> anything is Ambari. Here are the steps I’ve done, attempting deployment:
>
> # Install the Brooklyn command line tool
> brew install apache-brooklyn-cli
>
> # Authenticate to Brooklyn
> br login http://localhost:8081/
>
> # Get the Brooklyn Ambari repo
> git clone https://github.com/brooklyncentral/brooklyn-ambari.git
>
> cd brooklyn-ambari
>
> # Add Ambari to Brooklyn's catalog
> br add-catalog catalog.bom
>
> Next, in AWS, I had to establish my Security Group. I first created a
> security group called test-ambari and opened ports 8080 according to the
> brooklyn-ambari README. This failed, reasonably, since I know Ambari needs
> 8440 for coordinating agent nodes. In a third or fourth iteration, I saw an
> error that referenced port 22. At that point, I decided to just open things
> up a lot wider, in hopes that the networking would get everything working
> properly. My final Security Group, with speculative rules for TCP
> connections from anywhere is the following:
>
> * 8080   # The Ambari web UI
> * 24007  # I saw mention of GlusterFS in the logs
> * 24008  # Again, for GlusterFS
> * 8441   # Ambari Registration and Heartbeat Port
> * 22     # It seems the Brooklyn control machine has to SSH to Ambari nodes
> * 8440   # Ambari Agent orchestration
> * 2181   # ZooKeeper
>
> Next, from the Brooklyn web UI, I navigate to the Composer / editor and
> enter this YAML:
>
> location:
>   jclouds:aws-ec2:
>     # edit these to use your credential (or delete if credentials specified
> in brooklyn.properties)
>     identity:     <redacted>
>     credential:   <redacted>
>
>     region:       us-east-2
>
>     # we want Ubuntu, with a lot of RAM
>     osFamily:     ubuntu
>     minRam:       8gb
>
>     # set up this user and password (default is to authorize a public key)
>     user:         sample
>     password:     s4mpl3
>
> services:
> - type: ambari-cluster-application
>   name: Ambari Cluster
>   brooklyn.config:
>     securityGroup: test-ambari
>     initialSize: 3
>     services:
>     - FALCON
>     - FLUME
>     - GANGLIA
>     - HBASE
>     - HDFS
>     - KAFKA
>     - KERBEROS
>     - MAPREDUCE2
>     - OOZIE
>     - PIG
>     - SLIDER
>     - SQOOP
>     - YARN
>     - ZOOKEEPER
>
> After clicking the Deploy button, 4 instances are created in AWS. Here’s a
> screenshot:
> https://drive.google.com/file/d/0B91hmcyKP8SzLTRHcTdTZFkzWG8/view
>
> I can watch, in the Brooklyn UI, that there is communication between the
> Brooklyn Vagrant VM and these EC2 hosts. Screenshot:
> https://drive.google.com/file/d/0B91hmcyKP8SzR0R2bFFDYWhCUk0/view
>
> Eventually, all 3 Ambari agent nodes seem to be in a happy state.
> Unfortunately, the Ambari Server is not:
>
> Screenshot:
> https://drive.google.com/file/d/0B91hmcyKP8SzZzZ1XzF1UmZQZkU/view
>
> Screenshot:
> https://drive.google.com/file/d/0B91hmcyKP8SzYkQ4N1dXeVVnUEU/view
>
> When I attempt to open port 8080 (Ambari web UI) on the Ambari server, I’m
> receiving an error. Screenshot:
> https://drive.google.com/file/d/0B91hmcyKP8SzUkF2MGJWeFNCNDQ/view
>
> I know that at one point, things were working to a greater degree than I am
> seeing now. Unfortunately, I don’t recall how I managed to accomplish that
> (maybe using the newer BOM file from this PR [2] or perhaps it was the BYON
> / Vagrant nodes). I was, at one point, able to login to Ambari. I found it
> really great that there was a Brooklyn generated password for the service.
> As a last ditch effort for today, I did try the former on AWS and didn’t
> have success.
>
> Wrapping up
>
> My team and I need move quickly, and unfortunately, if I can’t get things
> working with Brooklyn soon - I’ll need to change my approach.
>
> I think the Brooklyn team has a very serious opportunity if you can support
> Ambari. I say that, since I’m not totally satisfied with the job that
> Hortonworks is doing supporting FLOSS deployments of the Hadoop ecosystem
> and Ambari. Presumably, if you can support the baseline, you’ll inherit a
> variety of other services.
>
> Furthermore, since Brooklyn has first-class support for load balancing and
> resizing, Hadoop users get serious value in being able to scale workloads.
> The possibility of developing / testing distributed workloads on local VMs
> is another value not so well supported in the open source.
>
> Lastly, if we could get Apache Ranger working (via an Ambari + Brooklyn
> configuration), then Brooklyn could provide a very rich feature set for
> securing clusters, data, and custom endpoints. Perhaps some other Apache
> folks would be willing to help with this integration?
>
> Thanks for all your help,
> b-long
>
> 0: https://github.com/apache/brooklyn-dist/pull/111
> 1: https://gist.github.com/b-long/ab096f45a7867574b74f01adff9f6c22
> 2: https://github.com/brooklyncentral/brooklyn-ambari/pull/126

Mime
View raw message