cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sasha Dolgy <sdo...@gmail.com>
Subject Re: after a while nothing happening with repair
Date Thu, 09 Jun 2011 08:33:12 GMT
I recall having this issue when one of the nodes wasn't available ... or
there was a problem during the repair process.  Cancelling the repair job
and rerunning it would complete successfully.  I believe there is a bug open
for this

https://issues.apache.org/jira/browse/CASSANDRA-2290

On Thu, Jun 9, 2011 at 10:28 AM, Jonathan Colby <jonathan.colby@gmail.com>wrote:

> When I run repair on a node in my 0.7.6-2 cluster, the repair starts to
> stream data and activity is seen in the logs.
>
> However, after a while (a day or so) it seems like everything freezes up.
> The repair command is still running (the command prompt has not returned)
> and netstats shows output similar to below.  All streams at 0% and nothing
> happening.  The logs indicate that things were started but there is no
> indication if anything is in fact still active.
>
> For example, this is the last log entry related to repair, just this
> morning:
>
>  INFO [StreamStage:1] 2011-06-09 07:13:21,423 StreamOut.java (line 173)
> Stream context metadata [/var/lib/cassandra/data/DFS/main-f-144-Data.db
> sections=2 progress=0/31947748 - 0%,
> /var/lib/cassandra/data/DFS/main-f-145-Data.db section
> s=2 progress=0/25786564 - 0%,
> /var/lib/cassandra/data/DFS/main-f-143-Data.db sections=2
> progress=0/5830103399 - 0%], 9 sstables.
>  INFO [StreamStage:1] 2011-06-09 07:13:21,423 StreamOutSession.java (line
> 174) Streaming to /10.46.108.104
>
>
> However, netstats on all related notes looks something like this.  The
> nodes continue to handle read/write requests just  fine. They are not
> overloaded at all.
>
> Any advice would be greatly appreciated.  Because repairs seem like they
> never finish, I have a feeling we have a lot of garbage data in our cluster.
>
> /opt/cassandra/bin/nodetool -h $HOSTNAME -p 35014 netstats
> Mode: Normal
> Not sending any streams.
> Streaming from: /10.46.108.104
>   DFS: /var/lib/cassandra/data/DFS/main-f-209-Data.db sections=2 progress=0/276461810
> - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-153-Data.db sections=2
> progress=0/100340568 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-40-Data.db sections=2
> progress=0/62726190502 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-180-Data.db sections=1
> progress=0/158898493 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-109-Data.db sections=2
> progress=0/87250515569 - 0%
> Streaming from: /10.47.108.102
>   DFS: /var/lib/cassandra/data/DFS/main-f-304-Data.db sections=2
> progress=0/13563864214 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-350-Data.db sections=1
> progress=0/2877129955 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-379-Data.db sections=2
> progress=0/143804948 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-370-Data.db sections=2
> progress=0/683716174 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-371-Data.db sections=2
> progress=0/56650 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-368-Data.db sections=2
> progress=0/4005533616 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-369-Data.db sections=2
> progress=0/155515922 - 0%
> Streaming from: /10.46.108.103
>   DFS: /var/lib/cassandra/data/DFS/main-f-888-Data.db sections=2
> progress=0/158096259 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-828-Data.db sections=1
> progress=0/29508276 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-886-Data.db sections=2
> progress=0/133704150 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-759-Data.db sections=2
> progress=0/83629797522 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-889-Data.db sections=2
> progress=0/96903803 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-751-Data.db sections=2
> progress=0/17944852950 - 0%
> Streaming from: /10.46.108.101
>   DFS: /var/lib/cassandra/data/DFS/main-f-1318-Data.db sections=2
> progress=0/60617216778 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-1179-Data.db sections=2
> progress=0/11870790009 - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-1324-Data.db sections=2 progress=0/710603722
> - 0%
>   DFS: /var/lib/cassandra/data/DFS/main-f-1322-Data.db sections=2
> progress=0/5844992187 - 0%
>
>


-- 
Sasha Dolgy
sasha.dolgy@gmail.com

Mime
View raw message