incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joyabrata Das <joy.luv.challen...@gmail.com>
Subject Re: Fwd: Re: Apache cassandra not joining cluster ring
Date Fri, 18 Apr 2014 16:39:07 GMT
Please refer replies embedded below.

Thanks in anticipation,

Joy

On 18 April 2014 18:18, Michael Shuler <michael@pbandjelly.org wrote:


On 04/18/2014 02:33 AM, Joyabrata Das wrote:


Let me explain sequentially.

I’ve four nodes in cluster; node1, node2 (seed), node3, node4.

1. Reinstalling 32-Bit Ubuntu problem didn’t solve the problem.



*The suggestion here was not to reinstall 32-bit, but to install 64-bit so
that your servers were all the same OS, taking that variable out of the
equation.* *>** Other three servers were 32-Bit only, so reinstalled the
same in Node4*


After explicitly enabling port 9160 by ufw command (Uncomplicated
FireWall in Ubuntu), node3 joined the ring though no firewall was set
among them.

Why only node3 required the modification while other three three
nodes were already joined the ring without that?



*Because your servers have different software installed and enabled by
default, like ufw? Dropping *all* firewall rules on *all* servers would be
a good starting point to taking this firewall variable out of the equation.
**>*
* Same is done via command $ ufw default allow Also I’m not sure about
other software except Java bit version was different which is now similar*


2. Now opscenter (http://node2:8888/opscenter/index.html) is detecting all
nodes in cluster though agent wasn’t getting installed with error "Unable
to find a java executable! Please install java or set JAVA_HOME. Exit code:
4", I’m sudoing to install agents.

Then I enabled port 61620, 61621 & agent got installed in node3 but not in
other nodes, please refer opscenter list view below.

*DATACENTER  HOSTNAME    TOKEN                             STATUS
LOAD (CPU)     DATA SIZE*

datacenter1     node1              -7564491331177403445    Active
            -

datacenter1     node2               0
Active              -

datacenter1     node4              -5882736283116946676    Active
            -

datacenter1     node4               5596871268197106065    Active
             0                      7.1 MB

 *Oops. You have 2 node4 servers. **>**That was a typo*


Why opscenter agent installed only in node3 & not in other nodes from
opscenter GUI?

*DETAILS REQUESTED*



*Node1*

*Node3*

*echo $JAVA_HOME*

/app/jdk1.6.0_16

/app/jdk1.6.0_16

*echo $PATH*

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/app/jdk1.6.0_16/bin

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/app/jdk1.6.0_16/bin

*which java*

/app/jdk1.6.0_16/bin/java

/app/jdk1.6.0_16/bin/java

*java -version*

java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) Server VM (build 14.2-b01, mixed mode)

java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) Server VM (build 14.2-b01, mixed mode)

*whoami (as there are diff sudo users from different nodes, I'm trying to
install agents one by one in each nodes)*

node1

node3

*ps aux | grep cassandra*

apacas   17440  4.3 30.1 1377336 1196792 ?     Sl   Mar19 1868:18
/app/jdk1.6.0_16/bin/java -ea -javaagent:./../lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M
-Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dlog4j.configuration=log4j-server.properties
-Dlog4j.defaultInitOverride=true -cp
./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-1.2.8.jar:./../lib/apache
cassandra-clientutil-1.2.8.jar:./../lib/apache-cassandra-thrift-1.2.8.jar:./../lib/avro-1.4.0-fixes.jar:./../lib/avro-1.4.0-sources-fixes.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.7.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.6.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar
org.apache.cassandra.service.CassandraDaemon

apacas   22642  1.5 25.4 1390740 1008932 ?     Sl   Apr15  66:28
/app/jdk1.6.0_16/bin/java -ea -javaagent:./../lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M
-Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dlog4j.configuration=log4j-server.properties
-Dlog4j.defaultInitOverride=true -cp
./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-1.2.8.jar:./../lib/apache-cassandra-clientutil-1.2.8.jar:./../lib/apache-cassandra-thrift-1.2.8.jar:./../lib/avro-1.4.0-fixes.jar:./../lib/avro-1.4.0-sources-fixes.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.7.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.6.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar
-Dcassandra.join_ring=true org.apache.cassandra.service.CassandraDaemon


1) how do you start Cassandra on the first 3 nodes? – using
/app/apache/cassandra/apache-cassandra-1.2.8/bin/cassandra script

2) how do you start it on the 4th node? – using
/app/apache/cassandra/apache-cassandra-1.2.8/bin/cassandra script

3) how did you start opscenter? – using
/app/apache/cassandra/opscenter-4.1.1/bin/opscenter script.

On 18 April 2014 18:18, Michael Shuler <michael@pbandjelly.org> wrote:

> On 04/18/2014 02:33 AM, Joyabrata Das wrote:
>
>> Let me explain sequentially.
>>  >
>>  > I’ve four nodes in cluster; node1, node2 (seed), node3, node4.
>>  >
>>  > 1. Reinstalling 32-Bit Ubuntu problem didn’t solve the problem.
>>
>
> The suggestion here was not to reinstall 32-bit, but to install 64-bit so
> that your servers were all the same OS, taking that variable out of the
> equation.
>
>
>   > After explicitly enabling port 9160 by ufw command (Uncomplicated
>> FireWall in Ubuntu), node3 joined the ring though no firewall was set
>> among them.
>>  >
>>  > Why only node3 required the modification while other three three
>> nodes were already joined the ring without that?
>>
>
> Because your servers have different software installed and enabled by
> default, like ufw? Dropping *all* firewall rules on *all* servers would be
> a good starting point to taking this firewall variable out of the equation.
>
>
>   >
>>  > 2. Now opscenter (http://node2:8888/opscenter/index.html) is
>> detecting all nodes in cluster though agent wasn’t getting installed
>> with error "Unable to find a java executable! Please install java or set
>> JAVA_HOME. Exit code: 4", I’m sudoing to install agents.
>>  >
>>  > Then I enabled port 61620, 61621 & agent got installed in node3 but
>> not in other nodes, please refer opscenter list view below.
>>  >
>>  > DATACENTER              HOSTNAME                 TOKEN
>>                                      STATUS          LOAD (CPU)     DATA
>> SIZE
>>  >
>>  > datacenter1                  node1
>> -7564491331177403445          Active              -
>>  >
>>  > datacenter1                  node2
>> 0                                              Active              -
>>  >
>>  > datacenter1                  node4
>> -5882736283116946676          Active              -
>>  >
>>  > datacenter1                  node4
>> 5596871268197106065            Active              0
>>              7.1 MB
>>
>
> Oops. You have 2 node4 servers.
>
>
>   >
>>  > Why opscenter agent installed only in node3 & not in other nodes from
>> opscenter GUI?
>>  >
>>  > DETAILS REQUESTED
>>  >
>>  >
>>  >
>>  > Node1
>>  >
>>  > Node3
>>  >
>>  > echo $JAVA_HOME
>>  >
>>  > /app/jdk1.6.0_16
>>  >
>>  > /app/jdk1.6.0_16
>>  >
>>  > echo $PATH
>>  >
>>  >
>>
>> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/app/jdk1.6.0_16/bin
>>  >
>>  >
>>
>> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/app/jdk1.6.0_16/bin
>>  >
>>  > which java
>>  >
>>  > /app/jdk1.6.0_16/bin/java
>>  >
>>  > /app/jdk1.6.0_16/bin/java
>>  >
>>  > java -version
>>  >
>>  > java version "1.6.0_16"
>>  > Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
>>  > Java HotSpot(TM) Server VM (build 14.2-b01, mixed mode)
>>  >
>>  > java version "1.6.0_16"
>>  > Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
>>  > Java HotSpot(TM) Server VM (build 14.2-b01, mixed mode)
>>  >
>>  > whoami (as there are diff sudo users from different nodes, I'm trying
>> to install agents one by one in each nodes)
>>
>
> So why are you running cassandra and opscenter as various users, instead
> of following the documentation and using the user "cassandra" on all
> servers?
>
>
>   >
>>  > node1
>>  >
>>  > node3
>>  >
>>  > ps aux | grep cassandra
>>  >
>>  > apacas   17440  4.3 30.1 1377336 1196792 ?     Sl   Mar19 1868:18
>> /app/jdk1.6.0_16/bin/java -ea -javaagent:./../lib/jamm-0.2.5.jar
>> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M
>> -Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:+UseTLAB -Djava.net.preferIPv4Stack=true
>> -Dcom.sun.management.jmxremote.port=7199
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dlog4j.configuration=log4j-server.properties
>> -Dlog4j.defaultInitOverride=true -cp
>>
>> ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-1.2.8.jar:./../lib/apache
>>
>> cassandra-clientutil-1.2.8.jar:./../lib/apache-cassandra-thrift-1.2.8.jar:./../lib/avro-1.4.0-fixes.jar:./../lib/avro-1.4.0-sources-fixes.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.7.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.6.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar
>> org.apache.cassandra.service.CassandraDaemon
>>  >
>>  > apacas   22642  1.5 25.4 1390740 1008932 ?     Sl   Apr15  66:28
>> /app/jdk1.6.0_16/bin/java -ea -javaagent:./../lib/jamm-0.2.5.jar
>> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M
>> -Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:+UseTLAB -Djava.net.preferIPv4Stack=true
>> -Dcom.sun.management.jmxremote.port=7199
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dlog4j.configuration=log4j-server.properties
>> -Dlog4j.defaultInitOverride=true -cp
>>
>> ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-1.2.8.jar:./../lib/apache-cassandra-clientutil-1.2.8.jar:./../lib/apache-cassandra-thrift-1.2.8.jar:./../lib/avro-1.4.0-fixes.jar:./../lib/avro-1.4.0-sources-fixes.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.7.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.6.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree
>>
> -
> 0.1.jar
>
>> -Dcassandra.join_ring=true org.apache.cassandra.service.CassandraDaemon
>>  >
>>  >
>>  >
>>  > 1) how do you start Cassandra on the first 3 nodes? – using
>> /app/apache/cassandra/apache-cassandra-1.2.8/bin/cassandra script
>>  >
>>  >
>>  >
>>  > 2) how do you start it on the 4th node? - using
>> /app/apache/cassandra/apache-cassandra-1.2.8/bin/cassandra script
>>  >
>>  >
>>  >
>>  > 3) how did you start opscenter? – using
>> /app/apache/cassandra/opscenter-4.1.1/bin/opscenter script.
>>  >
>>  > ---------- Forwarded message ----------
>>  > From: "Adrian Rodriguez" <oglowo@gmail.com <mailto:oglowo@gmail.com>>
>>  > Date: Apr 18, 2014 11:50 AM
>>  > Subject: Re: Apache cassandra not joining cluster ring
>>  > To: <user@cassandra.apache.org <mailto:user@cassandra.apache.org>>
>>  > Cc: <user@cassandra.apache.org <mailto:user@cassandra.apache.org>>
>>  >
>>  > If I were you, I would make sure all nodes were the same. The 32 / 64
>> thing might not be an issue, but you probably don't want that on a
>> production cluster. What would help is if your pasted the output of all
>> nodes for these commands:
>>  >
>>  >
>>  >
>>  > 1) echo $JAVA_HOME
>>  >
>>  > 2) echo $PATH
>>  >
>>  > 3) which java
>>  >
>>  > 4) java -version
>>  >
>>  > 5) whoami
>>  >
>>  > 6) ps aux | grep cassandra
>>  >
>>  >
>>  >
>>  > And if you could answer these questions:
>>  >
>>  >
>>  >
>>  > 1) how do you start Cassandra on the first 3 nodes?
>>  >
>>  >
>>  >
>>  > 2) how do you start it on the 4th node?
>>  >
>>  >
>>  >
>>  > 3) how did you start opscenter?
>>  >
>>  >
>>  >
>>  > I'm also a bit confused as to what your current problem is. Are you
>> having trouble installing the opscener agents from the opscener ui?
>>
>
> --
> Kind regards,
> Michael
>
>

Mime
View raw message