From user-return-22739-apmail-mahout-user-archive=mahout.apache.org@mahout.apache.org Tue Oct 3 06:00:45 2017 Return-Path: X-Original-To: apmail-mahout-user-archive@www.apache.org Delivered-To: apmail-mahout-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 27AAE10188 for ; Tue, 3 Oct 2017 06:00:45 +0000 (UTC) Received: (qmail 23625 invoked by uid 500); 3 Oct 2017 06:00:42 -0000 Delivered-To: apmail-mahout-user-archive@mahout.apache.org Received: (qmail 23568 invoked by uid 500); 3 Oct 2017 06:00:42 -0000 Mailing-List: contact user-help@mahout.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@mahout.apache.org Delivered-To: mailing list user@mahout.apache.org Received: (qmail 23556 invoked by uid 99); 3 Oct 2017 06:00:42 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Oct 2017 06:00:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 60A7BDB055 for ; Tue, 3 Oct 2017 06:00:41 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.4 X-Spam-Level: X-Spam-Status: No, score=-0.4 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id iPRWXa8VmYgI for ; Tue, 3 Oct 2017 06:00:39 +0000 (UTC) Received: from mail-io0-f173.google.com (mail-io0-f173.google.com [209.85.223.173]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 55E905F238 for ; Tue, 3 Oct 2017 06:00:38 +0000 (UTC) Received: by mail-io0-f173.google.com with SMTP id i197so6648411ioe.9 for ; Mon, 02 Oct 2017 23:00:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=I0Gfd1XCR0fry1tvSI1vQ5NuqmVNY4N/gNZzoDJ8Wrg=; b=MgunilfpgYGFZqLOpqrgsgYkz2W0X45k+jg0/xad2ApYL+v37wbs/iU+plrSpQRGcG X/795OybdthpWgDTm8kOEb1nBUxrUwymIgKFUwvrFbKtSnB97Jgp6X8AVWv9ak5R6i2U zPkvNIKSPLUNLG41ksIl+VMuhjxLj8VW2iEzmpvNMCwoajtHlwlu4IMFSSJdtmVJ4VGB TwNNgW20d/DObKGqoeELv2+KSwVtuO2s6CoH8paB9WJptLEhFtVMA8EFtP3FtZcEpPoJ kUGdQNkSe7e+kYFwwXUdFvi6TJoe2Ns2VnYrPFDl+W4zHS8viG7iL6ONAgADa+13P5+R kBWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=I0Gfd1XCR0fry1tvSI1vQ5NuqmVNY4N/gNZzoDJ8Wrg=; b=ZlKwvX9QFnB7KOcjO9VMJxCj0qTkcuOCGUhvBFkgmXOfrNL/ZjfuLaS+J1Xi1+epqe ZM+06jVB60fE/a9Gx2nLNZJUxH6/IVBpb+2VCXIzHOISNwrOII20T0jblsNhrJ/omIJI HcQx7HX8MbzsKH6SvJQuLcATmJL84xrjgJjQJFQXFo6CvIhZrFEDpdbWroYHX7kVd3mG RX51uizdZolpMEKvuF5KXsN/eZ0Ar6yD42pi6nC5OwIhH5P+cAGauPEM3Km151HD0LQD FzKnxSvKHgxcLL374bvKdtV9r+LkXu3LqWZ/F9MsdsHN7oUeHXLm3m5Eq7FU7IdHWsrJ BPcg== X-Gm-Message-State: AMCzsaV459pK8PAivBpIZDItdX6VjIOKKk2xnfiY0rsNFfuBPux4RTrW a3bwyAxSqPT4PIDomthSAY7U/Qa6YOCpebx1Ti5HRg== X-Google-Smtp-Source: AOwi7QBQG3eGplLeTadJtaS751GrPvDEN56HKfyhtsAh4AroGSZ/lS1hhAgx8M4aHAKvcz+O2IS1GX8wNviPzX2M6nU= X-Received: by 10.107.11.214 with SMTP id 83mr28016411iol.254.1507010430947; Mon, 02 Oct 2017 23:00:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.36.20.197 with HTTP; Mon, 2 Oct 2017 23:00:30 -0700 (PDT) In-Reply-To: References: From: Trevor Grant Date: Tue, 3 Oct 2017 01:00:30 -0500 Message-ID: Subject: Re: Running Mahout on a Spark cluster To: user@mahout.apache.org Content-Type: multipart/alternative; boundary="001a113f8bbe68ee6d055a9e35c0" --001a113f8bbe68ee6d055a9e35c0 Content-Type: text/plain; charset="UTF-8" Hey- sorry for long delay. I've been traveling. Pat Ferrel was telling me he was having some simlar issues with Spark+Mahout+SBT recently, and that we need to re-examine our naming conventions on JARs. Fwiw- I have several project that use Spark+Mahout in Spark 2.1/Scala-2.11, and we even test this in our Travis CI tests, but the trick is- we use Maven for the build. Any chance you could use maven? If not, maybe Pat can chime in here, I'm just not an SBT user, so I'm not 100% sure what to tell you. On Fri, Sep 22, 2017 at 10:06 PM, Hoa Nguyen wrote: > Hey all, > > Thanks for the offers of help. I've been able to narrow down some of the > problems to version incompatibility and I just wanted to give an update. > Just to back track a bit, my initial goal was to run Mahout on a > distributed cluster whether that was running Hadoop Map Reduce or Spark. > > I started out trying to get it to run on Spark, which I have some > familiarity, but that didn't seem to work. While the error messages seem to > indicate there weren't enough resources on the workers ("WARN > scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; > check your cluster UI to ensure that workers are registered and have > sufficient memory"), I'm pretty sure that wasn't the case, not only because > it's a 4 node cluster of m4.xlarges, I was able to run another, simpler > Spark batch job on that same distributed cluster. > > After a bit of wrangling, I was able to narrow down some of the issues. It > turns out I was kind of blindly using this repo https://github.com/ > pferrel/3-input-cooc as a guide without fully realizing that it was from > several years ago and based on Mahout 0.10.0, Scala 2.10 and Spark 1.1.1 > That is significantly different from my environment, which has Mahout > 0.13.0 and Spark 2.1.1 installed, which also means I have to use Scala > 2.11. After modifying the build.sbt file to account for those versions, I > now have compile type mismatch issues that I'm just not that savvy to fix > (see attached screenshot if you're interested). > > Anyway, the good news that I was able to finally get Mahout code running > on Hadoop map-reduce, but also after a bit wrangling. It turned out my > instances were running Ubuntu 14 and apparently that doesn't play well with > Hadoop 2.7.4, which prevented me from running any sample Mahout code (from > here: https://github.com/apache/mahout/tree/master/examples/bin) that > relied on map-reduce. Those problems went away after I installed Hadoop > 2.8.1 instead. Now I'm able to get the shell scripts running on a > distributed Hadoop cluster (yay!). > > Anyway, if anyone has more recent and working Spark Scala code that uses > Mahout that they can point me to, I'd appreciate it. > > Many thanks! > Hoa > > On Fri, Sep 22, 2017 at 1:09 AM, Trevor Grant > wrote: > >> Hi Hoa, >> >> A few things could be happening here, I haven't run across that specific >> error. >> >> 1) Spark 2.x - Mahout 0.13.0: Mahout 0.13.0 WILL run on Spark 2.x, however >> you need to build from source (not the binaries). You can do this by >> downloading mahout source or cloning the repo and building with: >> mvn clean install -Pspark-2.1,scala-2.11 -DskipTests >> >> 2) Have you setup spark with Kryo serialization? How you do this depends >> on >> if you're in the shell/zeppelin or using spark submit. >> >> However, for both of these cases- it shouldn't have even run local afaik >> so >> the fact it did tells me you probably have gotten this far? >> >> Assuming you've done 1 and 2, can you share some code? I'll see if I can >> recreate on my end. >> >> Thanks! >> >> tg >> >> On Thu, Sep 21, 2017 at 9:37 PM, Hoa Nguyen >> wrote: >> >> > I apologize in advance if this is too much of a newbie question but I'm >> > having a hard time running any Mahout example code in a distributed >> Spark >> > cluster. The code runs as advertised when Spark is running locally on >> one >> > machine but the minute I point Spark to a cluster and master url, I >> can't >> > get it to work, drawing the error: "WARN scheduler.TaskSchedulerImpl: >> > Initial job has not accepted any resources; check your cluster UI to >> ensure >> > that workers are registered and have sufficient memory" >> > >> > I know my Spark cluster is configured and working correctly because I >> ran >> > non-Mahout code and it runs on a distributed cluster fine. What am I >> doing >> > wrong? The only thing I can think of is that my Spark version is too >> recent >> > -- 2.1.1 -- for the Mahout version I'm using -- 0.13.0. Is that it or >> am I >> > doing something else wrong? >> > >> > Thanks for any advice, >> > Hoa >> > >> > > --001a113f8bbe68ee6d055a9e35c0--