cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: Need ur expertise on Cassandra issue!!
Date Fri, 17 Jan 2014 04:00:20 GMT
Look at the logs for the cassandra servers, are nodes going down ? 
Are there any other errors ? 
Check for log messages about GCInspector, if there is a lot of GC nodes will start to flap
up and down. 

It sounds like there is stability issue with cassandra, look there first to make sure it is
always available.

If you want to load 150GB of data from Hadoop to Cassandra a day I would suggest creating
SSTables in Hadoop and bulk loading them into cassandra. This article is old buy it’s still
relevant http://www.datastax.com/dev/blog/bulk-loading

Hope that helps. 

 

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 12/01/2014, at 3:53 pm, Arun <toarunkv5@gmail.com> wrote:

> Hi , 
> 
> I need your help & suggestions for our production issue. 
> 
> Details: 
> ---------- 
> we have 40 CF's in cassandra cluster for each datasource like below 
> MusicData--Keyspace 
> spotify_1-column family-Active 
> spotify_2-column family-standby 
> Daily we load data into this cluster using as below process: 
> 1.Astyanix library to delete inactive version of CF datahere spotify_2 
> 2. Hadoop Bulkload JAR -pushes data from Hadoop to cassandra into spotify_2 
> Data inflow rate 150GB per day . 
> Datastax community version 1.1.9 with 9 nodes of 4 TB which are built on 
> openstack with high end config. 
> 
> Problem: 
> ----------- 
> we're encountering the problem every week, the hadoop bulkload program is 
> failing with 
> java.io.IOException: Too many hosts failed: [/10.240.171.80, /10.240.171.76, 
> /10.240.171.74, /10.240.171.73] 
> 
> at 
> org.apache.cassandra.hadoop.BulkRecordWriter.close(BulkRecordWriter.java:243
> ) 
> I can provide more details about the error if you need.with our initial 
> analysis we came to know if we're deleting the deleted space for tombstoned 
> blocks will be reclaimed in compaction process so we have increased storage 
> capacity by adding new nodes but problem still persists. 
> we need your expertise to comment on this production issue.please let me 
> know if you need any information!! 
> I will wait for your response !! 
> 
> -Arun
> 
> 


Mime
View raw message