Return-Path: X-Original-To: apmail-mesos-user-archive@www.apache.org Delivered-To: apmail-mesos-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7665617943 for ; Fri, 31 Oct 2014 21:31:14 +0000 (UTC) Received: (qmail 57315 invoked by uid 500); 31 Oct 2014 21:31:14 -0000 Delivered-To: apmail-mesos-user-archive@mesos.apache.org Received: (qmail 57267 invoked by uid 500); 31 Oct 2014 21:31:14 -0000 Mailing-List: contact user-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@mesos.apache.org Delivered-To: mailing list user@mesos.apache.org Received: (qmail 57243 invoked by uid 99); 31 Oct 2014 21:31:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2014 21:31:14 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of stratos.dimopoulos@gmail.com designates 209.85.212.173 as permitted sender) Received: from [209.85.212.173] (HELO mail-wi0-f173.google.com) (209.85.212.173) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2014 21:31:10 +0000 Received: by mail-wi0-f173.google.com with SMTP id n3so2387885wiv.6 for ; Fri, 31 Oct 2014 14:30:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=9UQLly0IETtW13j2sXcoNbumSiTQxFRAeY8gmSEwJVA=; b=gS4qpCuWo9SFB1PFfcg5h3JCOflCPxO5gQe/n/snwy7Tin0jsHHId7nd1hkI1AGR44 HEkj9LgESCXcLWAa7huwHIbhOT11Ueh0Zs5Kvv3iMoB1bYc1u10Xzg0LroXlWkEtOedW XLUW7ef82Aq5+NnricAkLUSxGRHZGlenM9Wuy+7irtKwM9dcuNmW3csnkUg/sVljiOKS wKUPmOtqTRab4dMxMfpn/Z2Tk7Jga81SDc0jqtIExLsrWL9BLsoV9e5u5+H6zsJCCnqQ sWdthL7D07ey+k+jcXLnGTiaIYNVNim8vtqBFRXlGL3WJ8BzlIQ87NuHuubHm4gw1tQ9 7hew== X-Received: by 10.194.76.202 with SMTP id m10mr8173726wjw.42.1414791003961; Fri, 31 Oct 2014 14:30:03 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.79.230 with HTTP; Fri, 31 Oct 2014 14:29:23 -0700 (PDT) In-Reply-To: References: <8E441407-129E-47FA-A60A-39E9ACBE7C43@malloc64.com> <1414486674158.f19b69d8@Nodemailer> <72578FD2-E01E-41FB-B189-F6D015156FE5@malloc64.com> From: Stratos Dimopoulos Date: Fri, 31 Oct 2014 14:29:23 -0700 Message-ID: Subject: Re: CDH5.2.3 on mesos To: user@mesos.apache.org Content-Type: multipart/alternative; boundary=047d7bf0c1b23902420506beb3ed X-Virus-Checked: Checked by ClamAV on apache.org --047d7bf0c1b23902420506beb3ed Content-Type: text/plain; charset=UTF-8 High Availability you mean? Why not Zookeeper? On Thu, Oct 30, 2014 at 7:07 PM, Ankur Chauhan wrote: > Thanks this is great. I actually ended up using the vanilla Hadoop > distribution and it worked just fine. I will try out your tutorial. Side > question, is there a solution around making the jobtracker ha? > > -- ankur > > Sent from my iPhone > > On Oct 30, 2014, at 5:05 PM, Stratos Dimopoulos < > stratos.dimopoulos@gmail.com> wrote: > > Hi Ankur, > > I recently went through the process of installing Hadoop on Mesos over > cdh5.1.2 > > I created a post here - shouldn't be much different for your version: > http://strat0sphere.wordpress.com/2014/10/30/hadoop-on-mesos-installation-guide/ > > You can also find an other post about configuring CDH5.1.2 specifically to > use with Mesos: useful: > http://strat0sphere.wordpress.com/2014/10/30/cloudera-hdfs-cdh5-installation-to-use-with-mesos/ > > Have in mind that when using Mesos you don't need to start the jobtracker. > Mesos will do this for you. > You also mentioned that you are trying to start hadoop as ubuntu user. > This is not the right thing to do. Either add root to cloudera's root list > or (recommended) use the root user (mapred? hdfs?) that your cloudera > version considers as root - you have to check the documentation for this. > > Regarding the error you are seeing "Does not contain a valid host:port > authority: local" - I've seen this error when my worker version was > different than the jobtracker version (happened because I was using a > hadoop-on-mesos tar file compiled with a different version than the one my > cluster was using). To fix this you can do the obvious, which is making > sure the installed version is the same with the one you ship to the > executors through HDFS or you can hack this by adding the property hadoop.skip.worker.version.check > to True - In the later case I wish you good luck... Neverhteless, I am not > sure if this error can also appear in other cases. > > Hope this helps. > > Stratos > > > On Tue, Oct 28, 2014 at 12:30 PM, Ankur Chauhan > wrote: > >> Anyone else have something to add on this? >> -- Ankur Chauhan >> >> On 28 Oct 2014, at 02:10, Ankur Chauhan wrote: >> >> Hi tom, >> >> I was basically following the readme. This gist has the list of commands >> how i am setting up things >> https://gist.github.com/ankurcha/a9504b0e423b1a40d756 so first of all if >> possible if you could help me verify if my process of setting up core-site, >> hdfs-site and mapred-site is correct. I was starting the node with >> $ /opt/hadoop/bin/hadoop jobtracker >> >> There are two errors that i was working through. It seems that hadoop >> doesn't like running as root (which is good) but despite starting the >> process as ubuntu i kept getting >> >> Does not contain a valid host:port authority: local >> >> -- Ankur >> >> On 28 Oct 2014, at 01:57, Tom Arnfeld wrote: >> >> Hi Ankur, >> >> There aren't any getting started resources other than the documention >> there as far as I know. Could you share your hadoop configuration and >> perhaps a description of the problems you're having? >> >> Tom. >> >> >> >> On Tue, Oct 28, 2014 at 8:53 AM, Ankur Chauhan >> wrote: >> >>> H, >>> >>> I was trying to setup mesos/hadoop with the latest CDH version (MR1) and >>> it seems like the instructions are sort of out of date and I also tried the >>> suggestions in https://github.com/mesos/hadoop/issues/25 but after 4 >>> hours of flailing around I am still kind of stuck :-/ >>> >>> It seems like the configuration/installation instructions aren't >>> complete and I am just too new to hadoop to figure out what's missing or >>> going wrong. Does anyone know of a good resource I can use to get going? >>> >>> -- Ankur >>> >> >> >> >> > --047d7bf0c1b23902420506beb3ed Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
High Availability you mean? Why not Zookeeper?

On Thu, Oct 30, 2014 at= 7:07 PM, Ankur Chauhan <ankur@malloc64.com> wrote:
Thanks this is great. I a= ctually ended up using the vanilla Hadoop distribution and it worked just f= ine. I will try out your tutorial. Side question, is there a solution aroun= d making the jobtracker ha?

-- ankur=C2=A0

= Sent from my iPhone

On Oct 30, 2014, a= t 5:05 PM, Stratos Dimopoulos <stratos.dimopoulos@gmail.com> wrote:
Hi Ankur,

I recently went through the process of installing Hadoop on Mesos = over cdh5.1.2

I created a post here - shouldn'= t be much different for your version:=C2=A0http://strat0sphere.wordpress.com/2014/10/30/hadoop-on-mesos-installati= on-guide/

You can also find an other post abou= t configuring CDH5.1.2 specifically to use with Mesos: useful:=C2=A0http://strat0sphere.wordpress.c= om/2014/10/30/cloudera-hdfs-cdh5-installation-to-use-with-mesos/
<= div>
Have in mind that when using Mesos you don't need to= start the jobtracker. Mesos will do this for you.=C2=A0
You also= mentioned that you are trying to start hadoop as ubuntu user. This is not = the right thing to do. Either add root to cloudera's root list or =C2= =A0(recommended) use the root user (mapred? hdfs?) that your cloudera versi= on considers as root - you have to check the documentation for this.
<= div>
Regarding the error you are seeing "Does not contain a valid host:port authority: local" - I've seen = this error when my worker version was different than the jobtracker version= (happened because I was using a hadoop-on-mesos tar file compiled with a d= ifferent version than the one my cluster was using). To fix this you can do= the obvious, which is making sure the installed version is the same with t= he one you ship to the executors through HDFS or you can hack this by addin= g the property=C2=A0hadoop.skip.worker.version.check to True - In the later case= I wish you good luck... Neverhteless, I am not sure if this error can also= appear in other cases.

Hope this helps.

Stratos


On Tue, Oct 28, 2014 at 12:30 PM, A= nkur Chauhan <ankur@malloc64.com> wrote:
Anyone else have someth= ing to add on this?
-- Ankur Chauhan

On 28= Oct 2014, at 02:10, Ankur Chauhan <ankur@malloc64.com> wrote:

Hi tom,

I was basically fol= lowing the readme. This gist has the list of commands how i am setting up t= hings
https://gist.github.com/ankurcha/a9504b0e423b1a40d= 756=C2=A0so first of all if possible if you could help me verify if my = process of setting up core-site, hdfs-site and mapred-site is correct. I wa= s starting the node with=C2=A0
$=C2=A0/= opt/hadoop/bin/hadoop jobtracker

There are two err= ors that i was working through. It seems that hadoop doesn't like runni= ng as root (which is good) but despite starting the process as ubuntu i kep= t getting=C2=A0

Does not contain a valid host:port authority: local

-- Ankur

On 28 = Oct 2014, at 01:57, Tom Arnfeld <tom@duedil.com> wrote:

Hi Ankur,

There aren't any getting started resources other than the document= ion there as far as I know. Could you share your hadoop configuration and p= erhaps a description of the problems you're having?

Tom.



On Tue, Oct 28, 2014 at 8:53 AM, Anku= r Chauhan <ankur@malloc64.com> wrote:

H,


I was trying to setup mesos/hadoop with the latest CDH version (MR1) a= nd it seems like the instructions are sort of out of date and I also tried = the suggestions in=C2=A0https://github.com/mesos/hadoop/issues/25=C2=A0but= after 4 hours of flailing around I am still kind of stuck :-/

It seems like the configuration/installation instructions aren't c= omplete and I am just too new to hadoop to figure out what's missing or= going wrong. Does anyone know of a good resource I can use to get going?

-- Ankur





--047d7bf0c1b23902420506beb3ed--