incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Or Yanay ...@peer39.com>
Subject RE: Map-Reduce on top of cassandra
Date Thu, 24 Mar 2011 13:55:45 GMT
Worked like a charm!
I have installed hadoop on my Cassandra nodes and ran the MR using Hadoop job tracker.

A simple key count improved from ~2 hours to about 25 minutes (150M keys and ~100G on each
node)

Thanks Jeremy.

-----Original Message-----
From: Jeremy Hanna [mailto:jeremy.hanna1234@gmail.com] 
Sent: Monday, March 14, 2011 8:42 PM
To: user@cassandra.apache.org
Subject: Re: Map-Reduce on top of cassandra

Just for the sake of updating this thread - Orr didn't yet have task trackers on the Cassandra
nodes so most of the time was likely due to copying the ~100G of data to the hadoop cluster
prior to processing.  They're going to try after installing task trackers on the nodes.

On Mar 14, 2011, at 10:06 AM, Or Yanay wrote:

> Hi All,
>  
> I am trying to write some map-reduce tasks so I can find out stuff like - how many records
have X status?
> I am using 0.7.0 and have 5 nodes with ~100G of data on each node.
>  
> I have written the code based on the word_count example and the map-reduce is running
successfully BUT is extremely slow (about 2 hours for the simplest key count).
>  
> I am now looking to track down the slowness and tune my process, or explore alternative
ways to achieve the same goal.
>  
> Can anyone point me to a way to tune my map-reduce job?
> Does anyone have any experience exploring Cassandra data with Hadoop cluster configuration?
( As suggested inhttp://wiki.apache.org/cassandra/HadoopSupport#ClusterConfig)
>  
> Thanks,
> Orr
>  


Mime
View raw message