cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Shuler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table
Date Mon, 09 Feb 2015 21:12:35 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312887#comment-14312887
] 

Michael Shuler commented on CASSANDRA-7688:
-------------------------------------------

This caused a regression:
from: http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/470/testReport/
to: http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/472/testReport/ 

repro with the bootstrap_test.py dtest: vnodes vs no-vnodes:
{noformat}
(master)mshuler@hana:~/git/cassandra-dtest$ nosetests -vs bootstrap_test.py 
read_from_bootstrapped_node_test (bootstrap_test.TestBootstrap) ... Created keyspaces. Sleeping
1s for propagation.
Warming up WRITE with 50000 iterations...
INFO  19:17:03 Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is
incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
INFO  19:17:03 New Cassandra host /127.0.0.2:9042 added
Connected to cluster: test
Datatacenter: datacenter1; Host: /127.0.0.1; Rack: rack1
INFO  19:17:03 New Cassandra host /127.0.0.3:9042 added
Datatacenter: datacenter1; Host: /127.0.0.3; Rack: rack1
Datatacenter: datacenter1; Host: /127.0.0.2; Rack: rack1
INFO  19:17:03 New Cassandra host /127.0.0.1:9042 added
Failed to connect over JMX; not collecting these stats
Sleeping 2s...
Running WRITE with 8 threads for 10000 iteration
Failed to connect over JMX; not collecting these stats
total ops , adj row/s,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,
    max,   time,   stderr,  gc: #,  max ms,  sum ms,  sdv ms,      mb
2403      ,      2403,    2403,    2403,    2403,     3.3,     2.0,    10.0,    16.2,    23.8,
   27.3,    1.0,  0.00000,      0,       0,       0,       0,       0
4231      ,      1806,    1806,    1806,    1806,     4.4,     2.1,    16.2,    27.0,    67.2,
   72.5,    2.0,  0.00000,      0,       0,       0,       0,       0
6796      ,      2624,    2534,    2534,    2534,     3.1,     1.9,     9.0,    14.6,    49.3,
   50.5,    3.0,  0.10034,      0,       0,       0,       0,       0
9449      ,      2684,    2627,    2627,    2627,     3.0,     1.9,     8.8,    14.5,    35.1,
   36.7,    4.0,  0.08758,      0,       0,       0,       0,       0
10000     ,      2395,    2395,    2395,    2395,     3.3,     1.8,    10.0,    26.6,    48.2,
   48.2,    4.3,  0.07295,      0,       0,       0,       0,       0


Results:
op rate                   : 2345
partition rate            : 2345
row rate                  : 2345
latency mean              : 3.4
latency median            : 1.9
latency 95th percentile   : 10.4
latency 99th percentile   : 19.7
latency 99.9th percentile : 42.6
latency max               : 72.5
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:00:04
END
ok
simple_bootstrap_test (bootstrap_test.TestBootstrap) ... ok

----------------------------------------------------------------------
Ran 2 tests in 230.646s

OK
{noformat}

{noformat}
(master)mshuler@hana:~/git/cassandra-dtest$ export DISABLE_VNODES=true ; nosetests -vs bootstrap_test.py

read_from_bootstrapped_node_test (bootstrap_test.TestBootstrap) ... Created keyspaces. Sleeping
1s for propagation.
Warming up WRITE with 50000 iterations...
INFO  19:21:20 Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is
incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
Connected to cluster: test
INFO  19:21:20 New Cassandra host /127.0.0.3:9042 added
Datatacenter: datacenter1; Host: /127.0.0.1; Rack: rack1
Datatacenter: datacenter1; Host: /127.0.0.3; Rack: rack1
INFO  19:21:20 New Cassandra host /127.0.0.2:9042 added
Datatacenter: datacenter1; Host: /127.0.0.2; Rack: rack1
INFO  19:21:20 New Cassandra host /127.0.0.1:9042 added
Failed to connect over JMX; not collecting these stats
Sleeping 2s...
Running WRITE with 8 threads for 10000 iteration
Failed to connect over JMX; not collecting these stats
total ops , adj row/s,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,
    max,   time,   stderr,  gc: #,  max ms,  sum ms,  sdv ms,      mb
6145      ,      6143,    6143,    6143,    6143,     1.3,     0.9,     3.1,     6.9,    18.5,
   34.1,    1.0,  0.00000,      0,       0,       0,       0,       0
10000     ,      7485,    7485,    7485,    7485,     1.0,     0.7,     2.6,     4.9,    11.0,
   16.3,    1.5,  0.00000,      0,       0,       0,       0,       0


Results:
op rate                   : 6599
partition rate            : 6599
row rate                  : 6599
latency mean              : 1.2
latency median            : 0.8
latency 95th percentile   : 2.9
latency 99th percentile   : 6.2
latency 99.9th percentile : 14.9
latency max               : 34.1
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:00:01
END
ERROR
simple_bootstrap_test (bootstrap_test.TestBootstrap) ... ERROR

======================================================================
ERROR: read_from_bootstrapped_node_test (bootstrap_test.TestBootstrap)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mshuler/git/cassandra-dtest/dtest.py", line 426, in tearDown
    raise AssertionError('Unexpected error in %s node log: %s' % (node.name, errors))
AssertionError: Unexpected error in node1 node log: ['ERROR [OptionalTasks:1] 2015-02-09 13:21:38,131
CassandraDaemon.java:167 - Exception in thread Thread[OptionalTasks:1,5,main]\n']
-------------------- >> begin captured logging << --------------------
dtest: DEBUG: cluster ccm directory: /tmp/dtest-WychZv
cassandra.cluster: WARNING: [control connection] Error connecting to 127.0.0.4:
Traceback (most recent call last):
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 1891, in _reconnect_internal
    return self._try_connect(host)
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 1908, in _try_connect
    connection = self._cluster.connection_factory(host.address, is_control_connection=True)
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 708, in connection_factory
    return self.connection_class.factory(address, *args, **kwargs)
  File "/home/mshuler/git/python-driver/cassandra/io/asyncorereactor.py", line 162, in factory
    conn = cls(*args, **kwargs)
  File "/home/mshuler/git/python-driver/cassandra/io/asyncorereactor.py", line 195, in __init__
    raise socket.error(sockerr.errno, "Tried connecting to %s. Last error: %s" % ([a[4] for
a in addresses], sockerr.strerror))
error: [Errno 111] Tried connecting to [('127.0.0.4', 9042)]. Last error: Connection refused
cassandra.cluster: ERROR: Control connection failed to connect, shutting down Cluster:
Traceback (most recent call last):
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 754, in connect
    self.control_connection.connect()
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 1865, in connect
    self._set_new_connection(self._reconnect_internal())
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 1900, in _reconnect_internal
    raise NoHostAvailable("Unable to connect to any servers", errors)
NoHostAvailable: ('Unable to connect to any servers', {'127.0.0.4': error(111, "Tried connecting
to [('127.0.0.4', 9042)]. Last error: Connection refused")})
dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-WychZv
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: simple_bootstrap_test (bootstrap_test.TestBootstrap)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mshuler/git/cassandra-dtest/dtest.py", line 426, in tearDown
    raise AssertionError('Unexpected error in %s node log: %s' % (node.name, errors))
AssertionError: Unexpected error in node1 node log: ['ERROR [OptionalTasks:1] 2015-02-09 13:23:29,505
CassandraDaemon.java:167 - Exception in thread Thread[OptionalTasks:1,5,main]\n']
-------------------- >> begin captured logging << --------------------
dtest: DEBUG: cluster ccm directory: /tmp/dtest-VMfxCJ
cassandra.cluster: WARNING: [control connection] Error connecting to 127.0.0.1:
Traceback (most recent call last):
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 1891, in _reconnect_internal
    return self._try_connect(host)
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 1908, in _try_connect
    connection = self._cluster.connection_factory(host.address, is_control_connection=True)
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 708, in connection_factory
    return self.connection_class.factory(address, *args, **kwargs)
  File "/home/mshuler/git/python-driver/cassandra/io/asyncorereactor.py", line 162, in factory
    conn = cls(*args, **kwargs)
  File "/home/mshuler/git/python-driver/cassandra/io/asyncorereactor.py", line 195, in __init__
    raise socket.error(sockerr.errno, "Tried connecting to %s. Last error: %s" % ([a[4] for
a in addresses], sockerr.strerror))
error: [Errno 111] Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused
cassandra.cluster: ERROR: Control connection failed to connect, shutting down Cluster:
Traceback (most recent call last):
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 754, in connect
    self.control_connection.connect()
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 1865, in connect
    self._set_new_connection(self._reconnect_internal())
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 1900, in _reconnect_internal
    raise NoHostAvailable("Unable to connect to any servers", errors)
NoHostAvailable: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting
to [('127.0.0.1', 9042)]. Last error: Connection refused")})
cassandra.cluster: WARNING: Host 127.0.0.4 has been marked down
cassandra.cluster: WARNING: Host 127.0.0.3 has been marked down
cassandra.cluster: WARNING: Host 127.0.0.2 has been marked down
cassandra.cluster: WARNING: Host 127.0.0.1 has been marked down
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.4, scheduling retry in 2.0
seconds: [Errno 111] Tried connecting to [('127.0.0.4', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.3, scheduling retry in 2.0
seconds: [Errno 111] Tried connecting to [('127.0.0.3', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 2.0
seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.4, scheduling retry in 4.0
seconds: [Errno 111] Tried connecting to [('127.0.0.4', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.3, scheduling retry in 4.0
seconds: [Errno 111] Tried connecting to [('127.0.0.3', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 4.0
seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.3, scheduling retry in 8.0
seconds: [Errno 111] Tried connecting to [('127.0.0.3', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.4, scheduling retry in 8.0
seconds: [Errno 111] Tried connecting to [('127.0.0.4', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 8.0
seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.4, scheduling retry in 16.0
seconds: [Errno 111] Tried connecting to [('127.0.0.4', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.3, scheduling retry in 16.0
seconds: [Errno 111] Tried connecting to [('127.0.0.3', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 16.0
seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.4, scheduling retry in 32.0
seconds: [Errno 111] Tried connecting to [('127.0.0.4', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.3, scheduling retry in 32.0
seconds: [Errno 111] Tried connecting to [('127.0.0.3', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 32.0
seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused
cassandra.cluster: WARNING: Failed to create connection pool for new host 127.0.0.2:
Traceback (most recent call last):
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 1612, in run_add_or_renew_pool
    new_pool = HostConnection(host, distance, self)
  File "/home/mshuler/git/python-driver/cassandra/pool.py", line 294, in __init__
    self._connection = session.cluster.connection_factory(host.address)
  File "/home/mshuler/git/python-driver/cassandra/cluster.py", line 708, in connection_factory
    return self.connection_class.factory(address, *args, **kwargs)
  File "/home/mshuler/git/python-driver/cassandra/io/asyncorereactor.py", line 162, in factory
    conn = cls(*args, **kwargs)
  File "/home/mshuler/git/python-driver/cassandra/io/asyncorereactor.py", line 195, in __init__
    raise socket.error(sockerr.errno, "Tried connecting to %s. Last error: %s" % ([a[4] for
a in addresses], sockerr.strerror))
error: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused
cassandra.cluster: WARNING: Connection pool could not be created, not marking node 127.0.0.2
up
cassandra.cluster: WARNING: Host 127.0.0.2 has been marked down
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 2.0
seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 4.0
seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.4, scheduling retry in 64.0
seconds: [Errno 111] Tried connecting to [('127.0.0.4', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.3, scheduling retry in 64.0
seconds: [Errno 111] Tried connecting to [('127.0.0.3', 9042)]. Last error: Connection refused
cassandra.pool: WARNING: Error attempting to reconnect to 127.0.0.2, scheduling retry in 64.0
seconds: [Errno 111] Tried connecting to [('127.0.0.2', 9042)]. Last error: Connection refused
cassandra.cluster: WARNING: Host 127.0.0.1 has been marked down
cassandra.cluster: WARNING: Host 127.0.0.2 has been marked down
dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-VMfxCJ
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 2 tests in 220.323s

FAILED (errors=2)
{noformat}

Throws this in the node1.log:
{noformat}
ERROR [OptionalTasks:1] 2015-02-09 13:23:29,505 CassandraDaemon.java:167 - Exception in thread
Thread[OptionalTasks:1,5,main]
java.lang.IllegalStateException: null
        at org.apache.cassandra.utils.IntervalTree.max(IntervalTree.java:121) ~[main/:na]
        at org.apache.cassandra.db.DataTracker$View.sstablesInBounds(DataTracker.java:761)
~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore$6.apply(ColumnFamilyStore.java:1862)
~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore$6.apply(ColumnFamilyStore.java:1859)
~[main/:na]
        at org.apache.cassandra.db.SizeEstimatesRecorder.recordSizeEstimates(SizeEstimatesRecorder.java:79)
~[main/:na]
        at org.apache.cassandra.db.SizeEstimatesRecorder.run(SizeEstimatesRecorder.java:65)
~[main/:na]
        at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:82)
~[main/:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_76]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_76]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_76]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_76]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_76]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_76]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_76]
{noformat}

> Add data sizing to a system table
> ---------------------------------
>
>                 Key: CASSANDRA-7688
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
>             Project: Cassandra
>          Issue Type: New Feature
>            Reporter: Jeremiah Jordan
>            Assignee: Aleksey Yeschenko
>             Fix For: 2.1.3
>
>         Attachments: 7688.txt
>
>
> Currently you can't implement something similar to describe_splits_ex purely from the
a native protocol driver.  https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose
easily getting ownership information to a client in the java-driver.  But you still need the
data sizing part to get splits of a given size.  We should add the sizing information to a
system table so that native clients can get to it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message