incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ruslan usifov <ruslan.usi...@gmail.com>
Subject Re: Warn while bootstrap
Date Thu, 21 Oct 2010 13:08:10 GMT
Thanks for replay


How did you start the bootstrap and how are you checking if it has
> completed?
>
>
I start new node with <AutoBootstrap>true</AutoBootstrap> conf parameter,
and than check ring state with follow command:

freebsd#  nodetool -h 192.168.0.37 -p 8080 ring, if bootstrap this command
must give my two nodes, but it give only one:

Address       Status     Load
Range                                      Ring

128553582115057755121346842736115269027
192.168.0.37 Up         1.7 GB
43443562429493948342285492631024919735     |<--|



> What do nodetool info and nodetool streams say?
>

freebsd#  nodetool -h 192.168.0.37 -p 8080 stream
Starting NodeTool
Mode: Normal
Not sending any streams.
Not receiving any streams.

freebsd#  nodetool -h 192.168.0.220 -p 8080 stream
Starting NodeTool
Mode: Bootstrapping
Not sending any streams.
Not receiving any streams.


>
> The dropped message log usually indicates the node is under load.
>
> The log messages normally include the thread and file and line number with
> them. That's makes it a bit easier to follow along.
>
>
No, my node is not under heavy load, hear is top of system (192.167.0.37 -
on this node warn happens):

freebsd#  top

last pid: 70295;  load averages:  0.21,  0.77,
0.98                                                        up 16+01:02:56
17:07:28
299 processes: 1 running, 297 sleeping, 1 stopped
CPU:  1.1% user,  0.0% nice,  1.5% system,  0.4% interrupt, 97.0% idle
Mem: 1664M Active, 1299M Inact, 165M Wired, 99M Cache, 112M Buf, 28M Free
Swap: 4096M Total, 1132K Used, 4095M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
83355 www         1   4    0 48060K 23788K accept 1   0:50  0.20% php-cgi
  701 zabbix      1   4    5  5092K  1724K accept 0   4:52  0.00%
zabbix_agentd
  702 zabbix      1   4    5  5092K  1724K accept 0   4:52  0.00%
zabbix_agentd
  700 zabbix      1   4    5  5092K  1724K accept 0   4:50  0.00%
zabbix_agentd
  699 zabbix      1   8    5  5092K  1424K nanslp 1   1:41  0.00%
zabbix_agentd
83349 www         1   4    0 48060K 23280K accept 0   0:50  0.00% php-cgi
83356 www         1   4    0 48060K 22804K accept 1   0:50  0.00% php-cgi
83351 www         1   4    0 48060K 23244K accept 0   0:50  0.00% php-cgi
83359 www         1   4    0 48060K 23160K accept 1   0:50  0.00% php-cgi
83358 www         1   4    0 48060K 21152K accept 0   0:49  0.00% php-cgi
83348 www         1   4    0 48060K 21300K accept 0   0:49  0.00% php-cgi
83353 www         1   4    0 48060K 22812K accept 0   0:49  0.00% php-cgi
83354 www         1   4    0 48060K 22168K accept 1   0:49  0.00% php-cgi
83352 www         1   4    0 48060K 22340K accept 1   0:49  0.00% php-cgi
83350 www         1   4    0 48060K 21504K accept 0   0:49  0.00% php-cgi
83357 www         1   4    0 48060K 22476K accept 1   0:49  0.00% php-cgi
57523 www         1   4    0  6644K  4536K kqread 0   0:38  0.00% nginx
  598 root        1  44    0  4672K  1688K select 0   0:38  0.00% ntpd
  703 zabbix      1   8    5  5092K  1200K nanslp 0   0:36  0.00%
zabbix_agentd
57522 www         1   4    0  5620K  4284K kqread 0   0:36  0.00% nginx
83347 root        1   4    0 47036K  6880K kqread 0   0:33  0.00% php-cgi
  835 root        1  44    0  5876K  2348K select 1   0:30  0.00% sendmail
81287 root        1  44    0  9596K  4760K select 0   0:09  0.00% mc
  451 root        1  44    0  3184K   876K select 1   0:09  0.00% syslogd
  848 root        1   8    0  3212K   976K nanslp 1   0:07  0.00% cron
  874 root        1  44    0  3240K  1052K select 0   0:05  0.00% inetd
77864 root        1  44    0  8428K  2528K select 1   0:04  0.00% sshd
30821 root        1  44    0  8428K  2556K select 0   0:01  0.00% sshd
68918 www         1   4    0 14020K 10900K kqread 1   0:01  0.00% python
28599 root        1  44    0  8428K  2556K select 1   0:01  0.00% sshd
57351 mysql      16  20    0  1124M   378M sigwai 1   0:01  0.00% mysqld
31652 www         1  44    0  9624K  6568K select 0   0:01  0.00% python
31653 www         1  44    0  9624K  6568K select 1   0:01  0.00% python
70029 root       26  44    0  1150M 59468K ucond  0   0:01  0.00% java
68919 www         1   4    0 12996K 10744K kqread 0   0:00  0.00% python


also no network heavy load.


I solve problem by recompile cassandra with folow patch on
src/java/org/apache/cassandra/net/MessageDeliveryTask.java

Index: MessageDeliveryTask.java
===================================================================
--- MessageDeliveryTask.java    (revision 1025860)
+++ MessageDeliveryTask.java    (working copy)
@@ -37,12 +37,6 @@

     public void run()
     {
-        if (System.currentTimeMillis() >  constructionTime_ +
DatabaseDescriptor.getRpcTimeout())
-        {
-            MessagingService.incrementDroppedMessages();
-            return;
-        }
-
         StorageService.Verb verb = message_.getVerb();
         IVerbHandler verbHandler =
MessagingService.instance.getVerbHandler(verb);
         assert verbHandler != null : "unknown verb " + verb;



but this patch doesn't looks good for me

PS: we use:
     FreeBSD 7.2-p8,
     java - diablo-jdk-1.6.0.07.02_10 Java Development Kit 1.6.0_07.02
     100Mbit network between nodes

Mime
View raw message