Return-Path: X-Original-To: apmail-bigtop-user-archive@www.apache.org Delivered-To: apmail-bigtop-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5D4D711ECA for ; Wed, 16 Jul 2014 16:35:06 +0000 (UTC) Received: (qmail 98900 invoked by uid 500); 16 Jul 2014 16:35:06 -0000 Delivered-To: apmail-bigtop-user-archive@bigtop.apache.org Received: (qmail 98837 invoked by uid 500); 16 Jul 2014 16:35:06 -0000 Mailing-List: contact user-help@bigtop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@bigtop.apache.org Delivered-To: mailing list user@bigtop.apache.org Received: (qmail 98827 invoked by uid 99); 16 Jul 2014 16:35:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jul 2014 16:35:06 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of grover.markgrover@gmail.com designates 209.85.220.178 as permitted sender) Received: from [209.85.220.178] (HELO mail-vc0-f178.google.com) (209.85.220.178) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jul 2014 16:35:01 +0000 Received: by mail-vc0-f178.google.com with SMTP id la4so2199304vcb.9 for ; Wed, 16 Jul 2014 09:34:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=4NWDUBeQsAVGS1+BDsLvWMOGkg1BUWWL4Hp1vOK06F0=; b=qTfHtxDtCmOUuY6fctn4uzUrSGGl+sTbLpwqDx2e75yx1XaHXMSNw+dL7y7GCFeOhW AEF85PgsWK5+YEat/UwcR0OoBXot9B9Q5OzeIDk5Q7c2j9R+7dwpv8XQpMGL3cAW5TjN 8sEpO9ElbFJ9FPjtsK4dtHkfofzxJfw7B5nwD2dOsMFxZ2Eui/K41RXV+9wsSUr8e39f d9o4Hayk2yk1/mBhYNlDWYy0/b/PRXulodbiRccoHnlAhED1cH7DCEKQiLcxpZp1XYSp mhozE3QFzrNVq66zNYYdy/C1QJOa8BGSHVZ+2XWEGtP3iqEXk3VLlWzGPaR3on9ZTS5s oG0g== MIME-Version: 1.0 X-Received: by 10.52.33.202 with SMTP id t10mr2830362vdi.97.1405528480835; Wed, 16 Jul 2014 09:34:40 -0700 (PDT) Sender: grover.markgrover@gmail.com Received: by 10.58.217.227 with HTTP; Wed, 16 Jul 2014 09:34:40 -0700 (PDT) In-Reply-To: References: Date: Wed, 16 Jul 2014 09:34:40 -0700 X-Google-Sender-Auth: IvU2j1ayYjCFXLWZZl1N0nBtwjk Message-ID: Subject: Re: New to Bigtop, where to start? From: Mark Grover To: user@bigtop.apache.org Content-Type: multipart/alternative; boundary=20cf3079bc30d2887104fe521958 X-Virus-Checked: Checked by ClamAV on apache.org --20cf3079bc30d2887104fe521958 Content-Type: text/plain; charset=UTF-8 Possibly, can you check what packages you have installed related to hadoop. rpm -qa | grep hadoop On Wed, Jul 16, 2014 at 9:28 AM, David Fryer wrote: > Hi Mark, > I'm trying to follow those instructions on a CentOS 6 machine, and after > running "yum install hadoop\*", I can't find anything related to hadoop in > /etc/init.d. Is there something I'm missing? > > -David > > > On Wed, Jul 16, 2014 at 11:34 AM, Mark Grover wrote: > >> Welcome, David. >> >> For physical machines, I personally always use instructions like these: >> >> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+0.6.0 >> >> These for Bigtop 0.6.0, the latest Bigtop release is 0.7.0 but we don't >> have a page for that unfortunately (we should and if you could help with >> that, that'd be much appreciated!). We are tying up lose ends for Bigtop >> 0.8, so we hope to release it soon. >> >> Mark >> >> >> On Wed, Jul 16, 2014 at 8:20 AM, jay vyas >> wrote: >> >>> one more note : by "look at the csv file" above i meant, "edit it so >>> that it reflects your >>> environment". >>> >>> Make sure and read the puppet README file as well under >>> bigtop-deploy/puppet. >>> >>> >>> On Wed, Jul 16, 2014 at 11:15 AM, jay vyas >>> wrote: >>> >>>> Hi david . >>>> >>>> Glad to hear the vagrant stuff worked for you. Now , the next step >>>> will be to port it to bare metal, like you say. >>>> >>>> The Vagrantfile does two things >>>> >>>> 1) It creates a shared folder for all machines. >>>> 2) It spins up centos boxes . >>>> >>>> >>>> So in the "real world" you will need to obviously set up ssh between >>>> machines to start. >>>> After that , roughly, will need to do the following: >>>> >>>> - clone bigtop onto each of your machines >>>> - install puppet 2.x on each of the machines >>>> - look at the csv file created in the vagrant provisioner, and read the >>>> puppet README file (in bigtop-deploy) >>>> - run puppet apply on the head node >>>> Once that works >>>> - run puppet apply on each slave. >>>> now on any node that you use as client, (i just use the master usually) >>>> you can yum install your favorite ecosystem components: >>>> yum install -y pig mahout >>>> >>>> And you have a working hadoop cluster. >>>> >>>> one idea as I know your on the east coast, if your company is >>>> interested in hosting/sponsoring a bigtop meetup, we could possibly bring >>>> some folks from the boston / nyc area together to walk through building a >>>> bigtop cluster on bare metal. Let us know if any other questions. These >>>> directions are admittedly a little bit rough. >>>> >>>> Also, once you get this working, you can help us to update the wiki >>>> pages. >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Jul 16, 2014 at 10:39 AM, David Fryer >>>> wrote: >>>> >>>>> Hi Bigtop! >>>>> >>>>> I'm looking to use bigtop to help set up a small hadoop cluster. I'm >>>>> currently messing about with the hadoop tarball and all of the associated >>>>> xml files, and I don't really have the time or expertise to get it up and >>>>> working. >>>>> >>>>> Jay suggested that bigtop may be a good solution, so I've decided to >>>>> give it a shot. Unfortunately, documentation is fairly sparse and I'm not >>>>> quite sure where to start. I've cloned the github repo and used the >>>>> startup.sh script found in bigtop/bigtop-deploy/vm/vagrant-puppet to set up >>>>> a virtual cluster, but I am unsure how to apply this to physical machines. >>>>> I'm also not quite sure how to get hadoop and hdfs up and working. >>>>> >>>>> Any help would be appreciated! >>>>> >>>>> Thanks, >>>>> David Fryer >>>>> >>>> >>>> >>>> >>>> -- >>>> jay vyas >>>> >>> >>> >>> >>> -- >>> jay vyas >>> >> >> > --20cf3079bc30d2887104fe521958 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Possibly, can you check what packages you have installed r= elated to hadoop.

rpm -qa | grep hadoop


On Wed, Jul 16, 20= 14 at 9:28 AM, David Fryer <dfryer1193@gmail.com> wrote:<= br>
Hi Mark,
I'm trying= to follow those instructions on a CentOS 6 machine, and after running &quo= t;yum install hadoop\*", I can't find anything related to hadoop i= n /etc/init.d. Is there something I'm missing?

-David
<= div class=3D"h5">


On Wed, Jul 16, 2014 at 11:34 AM, Mark Grover <mark@apache.org> wrote:
Welcome, David.

For physical machines, I personally always use instructions like th= ese:

These for Bigtop 0.6.0, the latest Bigtop release= is 0.7.0 but we don't have a page for that unfortunately (we should an= d if you could help with that, that'd be much appreciated!). We are tyi= ng up lose ends for Bigtop 0.8, so we hope to release it soon.

Mark


On Wed, Jul 16, 2014 at 8:20= AM, jay vyas <jayunit100.apache@gmail.com> wrote:=
one more note : b= y "look at the csv file" above i meant, "edit it so that it = reflects your
environment".=C2=A0

Make sure and read=C2=A0 the puppet README= file as well under bigtop-deploy/puppet.=C2=A0


On Wed, Jul 16, 2014 at 11:15 AM, jay vyas <jay= unit100.apache@gmail.com> wrote:
Hi david .=C2=A0 =

Glad to hear the vagrant stuff worked for you.=C2=A0 Now= , the next step will be to port it to bare metal, like you say.

The Vagrantfile does two things

1) It creates a shared folder for all machines.
2) It spins up centos boxes .


So in the "rea= l world" you will need to obviously set up ssh between machines to sta= rt.=C2=A0
After that , roughly, will need to do the following:

- cl= one bigtop onto each of your=C2=A0 machines
- install puppet = 2.x on each of the machines
- look at the csv file created in= the vagrant provisioner, and read the puppet README file (in bigtop-deploy= )
- run puppet apply on the head node
Once that work= s
- run puppet apply on each slave.
now on any= node that you use as client, (i just use the master usually) you can yum i= nstall your favorite ecosystem components:
yum install -y pig mahout

And you have a worki= ng hadoop cluster.

one idea as I know your on = the east coast, if your company is interested in hosting/sponsoring a bigto= p meetup, we could possibly bring some folks from the boston / nyc area tog= ether to walk through building a bigtop cluster on bare metal.=C2=A0 Let us= know if any other questions.=C2=A0=C2=A0 These directions are admittedly a= little bit rough.=C2=A0

Also, once you get this working, you can help us to update the wiki pag= es.






On Wed, Jul 16, 2014 at 10= :39 AM, David Fryer <dfryer1193@gmail.com> wrote:
Hi Bigtop!

I'm looking to use bigtop to help set up a small hadoop cluster. I&#= 39;m currently messing about with the hadoop tarball and all of the associa= ted xml files, and I don't really have the time or expertise to get it = up and working.

Jay suggested that bigtop may be a good solution, so I&= #39;ve decided to give it a shot. Unfortunately, documentation is fairly sp= arse and I'm not quite sure where to start. I've cloned the github = repo and used the startup.sh script found in bigtop/bigtop-deploy/vm/vagran= t-puppet to set up a virtual cluster, but I am unsure how to apply this to = physical machines. I'm also not quite sure how to get hadoop and hdfs u= p and working.

Any help would be appreciated!

Thanks,
David Fryer



--
jay vyas



--
jay vyas



--20cf3079bc30d2887104fe521958--