Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F0F3F115A7 for ; Thu, 3 Jul 2014 05:48:40 +0000 (UTC) Received: (qmail 22452 invoked by uid 500); 3 Jul 2014 05:48:36 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 22318 invoked by uid 500); 3 Jul 2014 05:48:36 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 22307 invoked by uid 99); 3 Jul 2014 05:48:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2014 05:48:36 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dechouxb@gmail.com designates 209.85.215.42 as permitted sender) Received: from [209.85.215.42] (HELO mail-la0-f42.google.com) (209.85.215.42) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2014 05:48:32 +0000 Received: by mail-la0-f42.google.com with SMTP id pn19so7768858lab.15 for ; Wed, 02 Jul 2014 22:48:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=WtHuQySLynfn+M4V/NN87bQg7svKbBf/+qtUEqPvt/Y=; b=cw3hNvGXy/GbNsHZA+sFoO7oIYgjnuOJS2eYfC46I1+rplCBLTeCwvz8/RXYhFfZ/a Os09/4xSCKhiQ2jPDxL9LhqBAyP/qPckCgH8zq2Zu+rHvQyZ9a+k0Mv5mEniOEXoxMAP aVzOLVeq3KpApDQ6Gp91DG0Y+CfoqC29J3n5onbF4rO0LqTgSwCpAnpyScnQeNW/INQD PqU2zBL0iOySyFMtku00OivEdt3AKzGE8Y2UjHdTfdFmLy/bTes64s7wrs6A/0GA9VMz L+LtaYBKTQe6rQ95SzWO5QuqikpkFPkCeZPJGZo1FacKSK0MlNo9GVGHIxIVy1NLeuQ9 7UOw== MIME-Version: 1.0 X-Received: by 10.112.210.9 with SMTP id mq9mr1797443lbc.47.1404366488184; Wed, 02 Jul 2014 22:48:08 -0700 (PDT) Received: by 10.112.228.199 with HTTP; Wed, 2 Jul 2014 22:48:08 -0700 (PDT) In-Reply-To: References: Date: Thu, 3 Jul 2014 07:48:08 +0200 Message-ID: Subject: Re: Big Data tech stack (was Spark vs. Storm) From: Bertrand Dechoux To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=001a11c3c842a9ade004fd438d74 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c3c842a9ade004fd438d74 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I will second Stephen. At best you will arrive at a point where you can tell "I don't care about your problems here is the solution". Even though it sounds attractive if you are paid to set up the solution, that's really not the position a 'client' would want you to hold. Bertrand Dechoux On Thu, Jul 3, 2014 at 1:10 AM, Gavin Yue wrote: > Isn't this what Yarn or Mesos are trying to do? Separate the resources > management and applications. Run whatever suitable above them. Spark als= o > could run above yanr or mesos. Spark was designed for iteration intensive > computing like Machine learning algorithms. > > Storm is quite different. It is not designed for big data stored in the > hard disk. It is inspired by the stream data like tweets. On the other > side, Map-Reduce/HDFS was initially designed to handle stored webpage to > build up index. > > Hadoop is on the way to become a generic Big Data analysis framework. > HontonWorks and Cloudera are trying to make it much easier on management > and deployment. > > > > On Wed, Jul 2, 2014 at 4:25 PM, Adaryl "Bob" Wakefield, MBA < > adaryl.wakefield@hotmail.com> wrote: > >> You know what I=E2=80=99m really trying to do? I=E2=80=99m trying to c= ome up with a >> best practice technology stack. There are so many freaking projects it i= s >> overwhelming. If I were to walk into an organization that had no Big Dat= a >> capability, what mix of projects would be best to implement based on >> performance, scalability and easy of use/implementation? So far I=E2=80= =99ve got: >> Ubuntu >> Hadoop >> Cassandra (Seems to be the highest performing NoSQL database out there.) >> Storm (maybe?) >> Python (Easier than Java. Maybe that shouldn=E2=80=99t be a concern.) >> Hive (For people to leverage their existing SQL skillset.) >> >> That would seem to cover transaction processing and warehouse storage an= d >> the capability to do batch and real time analysis. What am I leaving out= or >> what do I have incorrect in my assumptions? >> >> B. >> >> >> >> *From:* Stephen Boesch >> *Sent:* Wednesday, July 02, 2014 3:07 PM >> *To:* user@hadoop.apache.org >> *Subject:* Re: Spark vs. Storm >> >> Spark Streaming discretizes the stream by configurable intervals of no >> less than 500Milliseconds. Therefore it is not appropriate for true real >> time processing.So if you need to capture events in the low 100's of >> milliseonds range or less than stick with Storm (at least for now). >> >> If you can afford one second+ of latency then spark provides advantages >> of interoperability with the other Spark components and capabilities. >> >> >> 2014-07-02 12:59 GMT-07:00 Shahab Yunus : >> >>> Not exactly. There are of course major implementation differences and >>> then some subtle and high level ones too. >>> >>> My 2-cents: >>> >>> Spark is in-memory M/R and it simulated streaming or real-time >>> distributed process for large datasets by micro-batching. The gain in s= peed >>> and performance as opposed to batch paradigm is in-memory buffering or >>> batching (and I am here being a bit naive/crude in explanation.) >>> >>> Storm on the other hand, supports stream processing even at a single >>> record level (known as tuple in its lingo.) You can do micro-batching o= n >>> top of it as well (using Trident API which is good for state maintenanc= e >>> too, if your BL requires that). This is more applicable where you want >>> control to a single record level rather than set, collection or batch o= f >>> records. >>> >>> Having said that, Spark Streaming is trying to simulate Storm's extreme >>> granular approach but as far as I recall, it still is built on top of c= ore >>> Spark (basically another level of abstraction over core Spark construct= s.) >>> >>> So given this, you can pick the framework which is more attuned to your >>> needs. >>> >>> >>> On Wed, Jul 2, 2014 at 3:31 PM, Adaryl "Bob" Wakefield, MBA < >>> adaryl.wakefield@hotmail.com> wrote: >>> >>>> Do these two projects do essentially the same thing? Is one better >>>> than the other? >>>> >>> >>> >> >> > > --001a11c3c842a9ade004fd438d74 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I will second Stephen. At best you will arrive at a point = where you can tell "I don't care about your problems here is the s= olution". Even though it sounds attractive if you are paid to set up t= he solution, that's really not the position a 'client' would wa= nt you to hold.

Bertrand Dechoux


On Thu, Jul 3, 2014 at 1:10 AM, Gavin Yu= e <yue.yuanyuan@gmail.com> wrote:
Isn't this what Yarn or Mesos are trying to do?= =C2=A0 Separate the resources management and applications. Run whatever sui= table above them.=C2=A0 Spark also could run above yanr or mesos. Spark was= designed for iteration intensive computing like Machine learning algorithm= s.

Storm is quite different.=C2=A0 It is not designed for = big data stored in the hard disk. It is inspired by the stream data like tw= eets. On the other side,=C2=A0Map-Reduce/HDFS was=C2=A0initially designed= =C2=A0to handle stored webpage to build up index.=C2=A0

Hadoop is on the way to become a generic Big Data analy= sis framework. HontonWorks and Cloudera are trying to make it much easier o= n management and deployment.



On Wed, Jul 2, 2014 at 4:25 PM, Adaryl &= quot;Bob" Wakefield, MBA <adaryl.wakefield@hotmail.com= > wrote:
You know what I=E2=80=99m really trying to do? I=E2=80=99m trying to c= ome up with a best=20 practice technology stack. There are so many freaking projects it is=20 overwhelming. If I were to walk into an organization that had no Big Data= =20 capability, what mix of projects would be best to implement based on=20 performance, scalability and easy of use/implementation? So far I=E2=80=99v= e=20 got:
Ubuntu
Hadoop
Cassandra (Seems to be the highest performing NoSQL database out=20 there.)
Storm (maybe?)
Python (Easier than Java. Maybe that shouldn=E2=80=99t be a concern.)<= /div>
Hive (For people to leverage their existing SQL skillset.)
=C2=A0
That would seem to cover transaction processing and warehouse storage = and=20 the capability to do batch and real time analysis. What am I leaving out or= what=20 do I have incorrect in my assumptions?
=C2=A0
B.
=C2=A0
=C2=A0
=C2=A0
Sent: Wednesday, July 02, 2014 3:07 PM
Subject: Re: Spark vs. Storm
=C2=A0
Spark Streaming discretizes the stream by configurable int= ervals of=20 no less than 500Milliseconds. Therefore it is not appropriate for true real= time=20 processing.So if you need to capture events in the low 100's of millise= onds=20 range or less than stick with Storm (at least for now).=20
=C2=A0
If you can afford one second+ of latency then spark provides advantage= s of=20 interoperability with the other Spark components and capabilities.


2014-07-02 12:59 GMT-07:00 Shahab Yunus <s= hahab.yunus@gmail.com>:
Not exactly. There are of course=C2=A0 major implementat= ion=20 differences and then some subtle and high level ones too.=20
=C2=A0
My 2-cents:
=C2=A0
Spark is in-memory M/R and it simulated streaming or real-time=20 distributed process for large datasets by micro-batching. The gain in spe= ed=20 and performance as opposed to batch paradigm is in-memory buffering or=20 batching (and I am here being a bit naive/crude in explanation.)
=C2=A0
Storm on the other hand, supports stream processing even at a single= =20 record level (known as tuple in its lingo.) You can do micro-batching on = top=20 of it as well (using Trident API which is good for state maintenance too,= if=20 your BL requires that). This is more applicable where you want control to= a=20 single record level rather than set, collection or batch of records.
=C2=A0
Having said that, Spark Streaming is trying to simulate Storm's = extreme=20 granular approach but as far as I recall, it still is built on top of cor= e=20 Spark (basically another level of abstraction over core Spark=20 constructs.)
=C2=A0
So given this, you can pick the framework which is more attuned to y= our=20 needs.


On Wed, Jul 2, 2014 at 3:31 PM, Adaryl "B= ob" Wakefield,=20 MBA <adaryl.wakefield@hotmail.com> wrote:
Do these two projects do essentially the same thing? Is one better= than=20 the other?
=C2=A0
=C2=A0


--001a11c3c842a9ade004fd438d74--