Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6AD2718AA0 for ; Fri, 15 Jan 2016 20:11:30 +0000 (UTC) Received: (qmail 42308 invoked by uid 500); 15 Jan 2016 20:11:26 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 42195 invoked by uid 500); 15 Jan 2016 20:11:26 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 42185 invoked by uid 99); 15 Jan 2016 20:11:26 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Jan 2016 20:11:26 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 19E421801DA for ; Fri, 15 Jan 2016 20:11:26 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.213 X-Spam-Level: **** X-Spam-Status: No, score=4.213 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id xymWnwzVwOt7 for ; Fri, 15 Jan 2016 20:11:17 +0000 (UTC) Received: from mail-lb0-f179.google.com (mail-lb0-f179.google.com [209.85.217.179]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id DDEFB42BA7 for ; Fri, 15 Jan 2016 20:11:16 +0000 (UTC) Received: by mail-lb0-f179.google.com with SMTP id oh2so321409499lbb.3 for ; Fri, 15 Jan 2016 12:11:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=9ALwIrVUzTOI6vGotTcR4A6TguhTq3fnbQJkl8MhdzY=; b=Bxuu8KbRYBkefz6/Rn8CMHIckuu1U765pmAocrqsnYfIhEYy6iudaVH6Rvt+27h264 WMPDCiwwydRQl9ECPKZLNZ9yPAhjthcwh5MD5YVwhHIAA0/zp6WZ1cLoYYp4cdFm8a2V qKSUTv1Hj70pqYj4phy+ngDrD75/IY0iEzCdqNZW18wk2/RQluxLCXmh1+GenfGMxsP6 8hoGUm7NmRriMnkch6cX2xJf4mAh+CdmG8tLYnMyBnk2lQNhY8Q1kEVThpWK986mPFgt HQFcutf7oy0KK3fIIIDxTwQLrcjTnRSDyFIwaADqopABEoy9gFwNZ9qnfF54t5RksAAz +wOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=9ALwIrVUzTOI6vGotTcR4A6TguhTq3fnbQJkl8MhdzY=; b=gtlViziyvEO2IWDj74TLP9a9tftM0V3MxshVKzWOZZVvomzSeKp94sIAXfzdgqOcUH PBwm/8FvhnDEtdCFv06zJoAhVyCDkvIv3O1Coy1eLdl165qvXm8GV/y1mJbtz/En0nAz dMNclahUOdXOvSNOIml4hzGGi5PPubsB5WwPmxOfT2d9qk69mIhMpRxKwUqdINT4qd0p dNREPaPq3vOm6K9a8kpWo3Ca+Tj2ZV6pCxcAUaI8I/H75zwmJTWIavUGct0geFy3pMum iEIx6HNYCmsjfbl1zQ63J3bEhJfuBi0LEEh6bZtwp4/pAqkMKl9kgpCb5v8KlSHkWaSO XSLw== X-Gm-Message-State: ALoCoQmrsWe4zBM7D7IQw7gdmhtw1DX8T3aIpmUaGjU/1qwIwPyapyC5pMkqB3Xz9ROoG5m5ACQHJlw4JpqCId74zgT57GnMsA== MIME-Version: 1.0 X-Received: by 10.112.160.232 with SMTP id xn8mr3583851lbb.22.1452888668856; Fri, 15 Jan 2016 12:11:08 -0800 (PST) Received: by 10.112.138.163 with HTTP; Fri, 15 Jan 2016 12:11:08 -0800 (PST) In-Reply-To: References: <1452865922577-25977.post@n3.nabble.com> <3A3F1285-5383-4853-A39D-61FBB3E2A050@gmail.com> Date: Fri, 15 Jan 2016 12:11:08 -0800 Message-ID: Subject: Re: simultaneous actions From: Jakob Odersky To: Koert Kuipers Cc: Matei Zaharia , Jonathan Coveney , Kira , "user@spark.apache.org" Content-Type: multipart/alternative; boundary=001a11c38e50016edc0529650107 --001a11c38e50016edc0529650107 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I stand corrected. How considerable are the benefits though? Will the scheduler be able to dispatch jobs from both actions simultaneously (or on a when-workers-become-available basis)? On 15 January 2016 at 11:44, Koert Kuipers wrote: > we run multiple actions on the same (cached) rdd all the time, i guess in > different threads indeed (its in akka) > > On Fri, Jan 15, 2016 at 2:40 PM, Matei Zaharia > wrote: > >> RDDs actually are thread-safe, and quite a few applications use them thi= s >> way, e.g. the JDBC server. >> >> Matei >> >> On Jan 15, 2016, at 2:10 PM, Jakob Odersky wrote: >> >> I don't think RDDs are threadsafe. >> More fundamentally however, why would you want to run RDD actions in >> parallel? The idea behind RDDs is to provide you with an abstraction for >> computing parallel operations on distributed data. Even if you were to c= all >> actions from several threads at once, the individual executors of your >> spark environment would still have to perform operations sequentially. >> >> As an alternative, I would suggest to restructure your RDD >> transformations to compute the required results in one single operation. >> >> On 15 January 2016 at 06:18, Jonathan Coveney wrote= : >> >>> Threads >>> >>> >>> El viernes, 15 de enero de 2016, Kira escribi=C3= =B3: >>> >>>> Hi, >>>> >>>> Can we run *simultaneous* actions on the *same RDD* ?; if yes how can >>>> this >>>> be done ? >>>> >>>> Thank you, >>>> Regards >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/simultaneous-actio= ns-tp25977.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.co= m >>>> . >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org >>>> For additional commands, e-mail: user-help@spark.apache.org >>>> >>>> >> >> > --001a11c38e50016edc0529650107 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I stand corrected. How considerable are the benefits thoug= h? Will the scheduler be able to dispatch jobs from both actions simultaneo= usly (or on a when-workers-become-available basis)?

On 15 January 2016 at 11:44, Ko= ert Kuipers <koert@tresata.com> wrote:
we run multiple actions on the same (cached) r= dd all the time, i guess in different threads indeed (its in akka)
<= br>
On Fri, Jan 15, 2016 at 2:40 PM, Matei Zahari= a <matei.zaharia@gmail.com> wrote:
RDDs actually are thre= ad-safe, and quite a few applications use them this way, e.g. the JDBC serv= er.

Matei

On Jan 15, 2016, = at 2:10 PM, Jakob Odersky <jodersky@gmail.com> wrote:

I don't think RDDs are threadsafe.
More fundamenta= lly however, why would you want to run RDD actions in parallel? The idea be= hind RDDs is to provide you with an abstraction for computing parallel oper= ations on distributed data. Even if you were to call actions from several t= hreads at once, the individual executors of your spark environment would st= ill have to perform operations sequentially.

As an altern= ative, I would suggest to restructure your RDD transformations to compute t= he required results in one single operation.

On 15 January 2016 at 06:18, Jon= athan Coveney <jcoveney@gmail.com> wrote:
Threads


El viernes, 15 de en= ero de 2016, Kira <mennour.r@gmail.com> escribi=C3=B3:
Hi,

Can we run *simultaneous* actions on the *same RDD* ?; if yes how can this<= br> be done ?

Thank you,
Regards



--
View this message in context: http= ://apache-spark-user-list.1001560.n3.nabble.com/simultaneous-actions-tp2597= 7.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org




--001a11c38e50016edc0529650107--