incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Slow Reads in Cassandra with Hadoop
Date Tue, 11 Dec 2012 21:37:52 GMT
First I would try to simplify your architecture. Get everything onto the same OS. 

Then change the topology so you have 1 job tracker, and 4 nodes that ran both Cassandra and
Hadoop tasks. So that reading and mapping the data is happening on the same nodes. Reads from
cassandra happen as range scans, reading all the rows on the node. 

If you are seeing a lot of GC messages it may be because of the JVM config or the workload.
If the JVM config is the default look at to modify the workload, try reducing the batch size
http://wiki.apache.org/cassandra/HadoopSupport?highlight=%28hadoop%29

Hope that helps. 

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 7/12/2012, at 4:29 AM, Ralph Romanos <matganine@hotmail.com> wrote:

> Hello Cassandra users,
> 
> I am trying to read and process data in Cassandra using Hadoop. I have a 4-node Cassandra
cluster, and an 8-node Hadoop cluster:
> - 1 Namenode/Jobtracker
> - 7 Datanodes/Tasktrackers (4 of them are also hosting Cassandra)
> 
> I am using Cassandra 1.2 beta, Hadoop 0.20.2, java 1.6_u_34, 7 of my nodes are on SLES
10 (Linux kernel: 2.6.16.60-0.76.8-smp) and the last one is on SLES 11 (Linux kernel: 2.6.32.12-0.7-default).
They are all 24 cores with 33 GB ram, but for some reasons, the node running on SLES 11 is
running Hadoop jobs significantly faster then the others (two to three times faster); any
explanation for this is welcome as well.
> 
> In my Hadoop job, I am using ColumnFamilyInputFormat and ColumnFamilyOutputFormat.
> Here is my mapper: Mapper<ByteBuffer, SortedMap<ByteBuffer, IColumn>, Text,
Text>,
> and my reducer: Reducer<Text, Text, ByteBuffer, List<Mutation>>.
> 
> The input of my mapper is the values of the columns given in input. In output of my map,
I write those values in the Text format separated by comas. I ran the task on about 400 million
rows in my database so the map function is called one time for each row. When I run the job
with 6 concurrent map tasks on each server and 7 Hadoop servers, the job takes about an hour
and a half (the reduce step is done in about 5 seconds, so the problem is in map task), which
is too long...
> 
> So I set some timers between each call to the map function, and here is what I get:
> 
> After mapping about 4150 - 4160 rows (each row has 8 columns and values are strings or
long) in Cassandra in 60 ms approximately, there is a gap in time.
> This gap is not the same for all the machines:
> - it is 200 ms on the node Cassandra + Hadoop that is running on SLES 11 (Cassandra is
using 400% cpu on this node)
> - it is 4200 ms on the 3 nodes that are hadoop only
> - it is 900 ms on two nodes that are Cassandra + Hadoop and running on SLES 10 (Cassandra
is using 400% cpu on this node)
> - it is 4200 ms on the last Cassandra + Hadoop node (Cassandra is using 2300% cpu on
this node and I get a lot of Garbage collection messages in the cassandra logs of this node
only)
> 
> When I run only 1 concurrent map task per node (instead of 6 above), I get the following
results:
> - it is 200 ms on the node Cassandra + Hadoop that is running on SLES 11 (Cassandra is
using 150% cpu on this node)
> - it is 600 ms on the 3 nodes that are hadoop only
> - it is 600 ms on two nodes that are Cassandra + Hadoop and running on SLES 10 (Cassandra
is using 150% cpu on this node)
> - it is 600 ms on the last Cassandra + Hadoop node (Cassandra is using 400% cpu on this
node and I don't get Garbage collection messages anymore in the cassandra logs)
> 
> I do not really know what is happening during this gap; my guess would be that Hadoop
is reading data in Cassandra, streaming it to the Hadoop nodes and finally writing it to the
Hadoop Distributed File System.
> Does anyone understand how reads are done when using Hadoop and Cassandra? and what is
exactly happening during this gap in time? and why there is such a difference in time between
nodes running on SLES10 and the node running on SLES 11?
> Why does it seem like this gap in time is smaller on nodes running Cassandra + Hadoop?
> 
> Finally, does anyone know why this gap in time occurs after approximately 4160 rows which
represent about 32 KB in my case? Is there any parameter I am not aware of to change this?
> 
> Thanks in advance,
> Ralph


Mime
View raw message