incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ralph Romanos <matgan...@hotmail.com>
Subject Slow Reads in Cassandra with Hadoop
Date Thu, 06 Dec 2012 15:29:23 GMT

Hello Cassandra users,
I am trying to read and process data in Cassandra using Hadoop. I have a 4-node Cassandra
cluster, and an 8-node Hadoop cluster:- 1 Namenode/Jobtracker- 7 Datanodes/Tasktrackers (4
of them are also hosting Cassandra)
I am using Cassandra 1.2 beta, Hadoop 0.20.2, java 1.6_u_34, 7 of my nodes are on SLES 10
(Linux kernel: 2.6.16.60-0.76.8-smp) and the last one is on SLES 11 (Linux kernel: 2.6.32.12-0.7-default).
They are all 24 cores with 33 GB ram, but for some reasons, the node running on SLES 11 is
running Hadoop jobs significantly faster then the others (two to three times faster); any
explanation for this is welcome as well.
In my Hadoop job, I am using ColumnFamilyInputFormat and ColumnFamilyOutputFormat.Here is
my mapper: Mapper<ByteBuffer, SortedMap<ByteBuffer, IColumn>, Text, Text>,and
my reducer: Reducer<Text, Text, ByteBuffer, List<Mutation>>.
The input of my mapper is the values of the columns given in input. In output of my map, I
write those values in the Text format separated by comas. I ran the task on about 400 million
rows in my database so the map function is called one time for each row. When I run the job
with 6 concurrent map tasks on each server and 7 Hadoop servers, the job takes about an hour
and a half (the reduce step is done in about 5 seconds, so the problem is in map task), which
is too long...
So I set some timers between each call to the map function, and here is what I get:
After mapping about 4150 - 4160 rows (each row has 8 columns and values are strings or long)
in Cassandra in 60 ms approximately, there is a gap in time.This gap is not the same for all
the machines:- it is 200 ms on the node Cassandra + Hadoop that is running on SLES 11 (Cassandra
is using 400% cpu on this node)- it is 4200 ms on the 3 nodes that are hadoop only- it is
900 ms on two nodes that are Cassandra + Hadoop and running on SLES 10 (Cassandra is using
400% cpu on this node)- it is 4200 ms on the last Cassandra + Hadoop node (Cassandra is using
2300% cpu on this node and I get a lot of Garbage collection messages in the cassandra logs
of this node only)
When I run only 1 concurrent map task per node (instead of 6 above), I get the following results:-
it is 200 ms on the node Cassandra + Hadoop that is running on SLES 11 (Cassandra is using
150% cpu on this node)- it is 600 ms on the 3 nodes that are hadoop only- it is 600 ms on
two nodes that are Cassandra + Hadoop and running on SLES 10 (Cassandra is using 150% cpu
on this node)- it is 600 ms on the last Cassandra + Hadoop node (Cassandra is using 400% cpu
on this node and I don't get Garbage collection messages anymore in the cassandra logs)
I do not really know what is happening during this gap; my guess would be that Hadoop is reading
data in Cassandra, streaming it to the Hadoop nodes and finally writing it to the Hadoop Distributed
File System.Does anyone understand how reads are done when using Hadoop and Cassandra? and
what is exactly happening during this gap in time? and why there is such a difference in time
between nodes running on SLES10 and the node running on SLES 11?Why does it seem like this
gap in time is smaller on nodes running Cassandra + Hadoop?
Finally, does anyone know why this gap in time occurs after approximately 4160 rows which
represent about 32 KB in my case? Is there any parameter I am not aware of to change this?
Thanks in advance,Ralph 		 	   		  
Mime
View raw message