cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stan Lemon <sle...@salesforce.com>
Subject Re: Long joining node
Date Wed, 05 Aug 2015 12:33:01 GMT
I set the stream timeout to 1 hour this morning and started fresh trying to
join this node.  It took about an hour to stream over 230gb of data, and
then into hour 2 I wound up back where I was yesterday, the node's load is
slowly reducing and the netstats does not show sending or receiving
anything.  I'm not sure how long I should wait before I throw the towel in
on this attempt. I'm also not really sure what to try next...

The only thing in the logs currently are three entries like this:

ERROR 07:39:44,447 Exception in thread Thread[CompactionExecutor:31,1,main]
java.lang.RuntimeException: Last written key
DecoratedKey(8633837336094175369,
003076697369746f725f706167655f766965623936636232346331623661313935313634346638303838393465313132373700004930303030663264632d303030302d303033302d343030302d3030303030303030663264633a66376436366166382d383564352d313165342d383030302d30303030303035343764623600)
>= current key DecoratedKey(-6568345298384940765,
003076697369746f725f706167655f766965623936636232346331623661313935313634346638303838393465313132373700004930303030376464652d303030302d303033302d343030302d3030303030303030376464653a64633930336533382d643766342d313165342d383030302d30303030303730626338386300)
writing into
/var/lib/cassandra/data/pi/__shardindex/pi-__shardindex-tmp-jb-644-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)



ANY help is greatly appreciated.

Thanks,
Stan





On Tue, Aug 4, 2015 at 2:23 PM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:

> That's the one. I set it to an hour to be safe (if a stream goes above the
> timeout it will get restarted) but it can probably be lower.
>
> All the best,
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
>
>
> <http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Tue, Aug 4, 2015 at 2:21 PM, Stan Lemon <slemon@salesforce.com> wrote:
>
>> Sebastian,
>> You're referring to streaming_socket_timeout_in_ms correct?  What value
>> do you recommend?  All of my nodes are currently at the default 0.
>>
>> Thanks,
>> Stan
>>
>>
>> On Tue, Aug 4, 2015 at 2:16 PM, Sebastian Estevez <
>> sebastian.estevez@datastax.com> wrote:
>>
>>> It helps to set stream socket timeout in the yaml so that you don't hang
>>> forever on a lost / broken stream.
>>>
>>> All the best,
>>>
>>>
>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>
>>> Sebastián Estévez
>>>
>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>
>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>> <https://twitter.com/datastax> [image: g+.png]
>>> <https://plus.google.com/+Datastax/about>
>>> <http://feeds.feedburner.com/datastax>
>>>
>>>
>>> <http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature>
>>>
>>> DataStax is the fastest, most scalable distributed database technology,
>>> delivering Apache Cassandra to the world’s most innovative enterprises.
>>> Datastax is built to be agile, always-on, and predictably scalable to any
>>> size. With more than 500 customers in 45 countries, DataStax is the
>>> database technology and transactional backbone of choice for the worlds
>>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>
>>> On Tue, Aug 4, 2015 at 2:14 PM, Robert Coli <rcoli@eventbrite.com>
>>> wrote:
>>>
>>>> On Tue, Aug 4, 2015 at 11:02 AM, Stan Lemon <slemon@salesforce.com>
>>>> wrote:
>>>>
>>>>> I am attempting to add a 13th node in one of the datacenters. I have
>>>>> been monitoring this process from the node itself with nodetool netstats
>>>>> and from one of the existing nodes using nodetool status.
>>>>>
>>>>> On the existing node I see the new node as UJ. I have watched the load
>>>>> steadily climb up to about 203.4gb, and then over the last two hours
it has
>>>>> fluctuated a bit and has been steadily dropping to about 203.1gb
>>>>>
>>>>
>>>> It's probably hung. If I were you I'd probably wipe the node and
>>>> re-bootstrap.
>>>>
>>>> (what version of cassandra/what network are you on (AWS?)/etc.)
>>>>
>>>> =Rob
>>>>
>>>>
>>>
>>>
>>
>

Mime
View raw message