spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From haihar nahak <harihar1...@gmail.com>
Subject Re: how to make a spark cluster ?
Date Tue, 21 Apr 2015 22:45:17 GMT
I did some performance check on socLiveJournal PageRank b/w my local
machine (8 cores, 16 gb ) in local mode and my small cluster (4 nodes, 12
cores, 40 gb) and i found cluster mode is way faster than local mode. So I
confused.
no. of iterations ---> Local mode(in mins) --> cluster mode(in mins)
1
         20     1 2    31.3 1.2 3   39.5 1.3 5    56.4 1.6 10   117.26 2.6
with the help of this , I think , might be installing spark cluster on the
same machine and instead of giving local[no. of cores] , I'll set to
spark://host:7070.

Please let me know If I wrong somewhere.


On Tue, Apr 21, 2015 at 6:27 PM, Reynold Xin <rxin@databricks.com> wrote:

> Actually if you only have one machine, just use the Spark local mode.
>
> Just download the Spark tarball, untar it, set master to local[N], where N
> = number of cores. You are good to go. There is no setup of job tracker or
> Hadoop.
>
>
> On Mon, Apr 20, 2015 at 3:21 PM, haihar nahak <harihar1987@gmail.com>
> wrote:
>
>> Thank you :)
>>
>> On Mon, Apr 20, 2015 at 4:46 PM, Jörn Franke <jornfranke@gmail.com>
>> wrote:
>>
>>> Hi, If you have just one physical machine then I would try out Docker
>>> instead of a full VM (would be waste of memory and CPU).
>>>
>>> Best regards
>>> Le 20 avr. 2015 00:11, "hnahak" <harihar1987@gmail.com> a écrit :
>>>
>>>> Hi All,
>>>>
>>>> I've big physical machine with 16 CPUs , 256 GB RAM, 20 TB Hard disk. I
>>>> just
>>>> need to know what should be the best solution to make a spark cluster?
>>>>
>>>> If I need to process TBs of data then
>>>> 1. Only one machine, which contain driver, executor, job tracker and
>>>> task
>>>> tracker everything.
>>>> 2. create 4 VMs and each VM should consist 4 CPUs , 64 GB RAM
>>>> 3. create 8 VMs and each VM should consist 2 CPUs , 32 GB RAM each
>>>>
>>>> please give me your views/suggestions
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-make-a-spark-cluster-tp22563.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>
>>
>> --
>> {{{H2N}}}-----(@:
>>
>
>


-- 
{{{H2N}}}-----(@:

Mime
View raw message