Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 72B6F1858D for ; Wed, 7 Oct 2015 17:38:28 +0000 (UTC) Received: (qmail 64737 invoked by uid 500); 7 Oct 2015 17:38:22 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 64628 invoked by uid 500); 7 Oct 2015 17:38:21 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 64614 invoked by uid 99); 7 Oct 2015 17:38:21 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Oct 2015 17:38:21 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 6BCD4C42B5 for ; Wed, 7 Oct 2015 17:38:21 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.02 X-Spam-Level: X-Spam-Status: No, score=-0.02 tagged_above=-999 required=6.31 tests=[RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id RgdfGmS8a4l2 for ; Wed, 7 Oct 2015 17:38:12 +0000 (UTC) Received: from mail-wi0-f173.google.com (mail-wi0-f173.google.com [209.85.212.173]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 98DD120562 for ; Wed, 7 Oct 2015 17:38:11 +0000 (UTC) Received: by wicge5 with SMTP id ge5so223709305wic.0 for ; Wed, 07 Oct 2015 10:38:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=5/WUFt8nLOdNeBepxm0PQs1rHPqXWEpX5ganjMwrAvs=; b=SlMjf8DCN6dQn4/1Jjr0V2KyfBpSLDHnHEjfs2cS7uCWlJYTHh7gj0CF0jwfO+HzAU 7EHTKhcz7XK9miV0W6eQd2rxZvuQcskn6+j4QaOJyFa5QgWRYzmnf4SLmwnMHk3ZnI4C JF3zAUPQ5gH6ySaDKfYxP3xv68gCFrTp/aKExUS+VkDJ2ZkMuPpP8JRsBlOft6UnZFSm J6TEAhmEGxD5j9SzOxf0d0B+uLpTXQmAFCGSwdSn2nEMjzEQd0mMdiSk4rrEFPbk3VdY FL2CUvBUvTf0O2U/4RTxJh89877OQ5rrFiZe3LjzP82YsL3cA9m5/yVMv06sVhkPjBQv wnhQ== X-Gm-Message-State: ALoCoQn7z2iZDAS9jbA/dmYihtgt1fa2gdzWgef+S3VT8kVNOy+QVKeDIC8ma24dwH3yTlbzBXqh X-Received: by 10.180.37.113 with SMTP id x17mr23631831wij.33.1444239490357; Wed, 07 Oct 2015 10:38:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.27.39.196 with HTTP; Wed, 7 Oct 2015 10:37:51 -0700 (PDT) In-Reply-To: References: From: Sean Owen Date: Wed, 7 Oct 2015 18:37:51 +0100 Message-ID: Subject: Re: Spark standalone hangup during shuffle flatMap or explode in cluster To: Saif.A.Ellafi@wellsfargo.com Cc: user Content-Type: text/plain; charset=UTF-8 -dev Is r.getInt(ind) very large in some cases? I think there's not quite enough info here. On Wed, Oct 7, 2015 at 6:23 PM, wrote: > When running stand-alone cluster mode job, the process hangs up randomly > during a DataFrame flatMap or explode operation, in HiveContext: > > -->> df.flatMap(r => for (n <- 1 to r.getInt(ind)) yield r) > > This does not happen either with SQLContext in cluster, or Hive/SQL in local > mode, where it works fine. > > A couple minutes after the hangup, executors start dropping. I am attching > the logs > Saif > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org > For additional commands, e-mail: user-help@spark.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org