cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajesh Radhakrishnan <>
Subject Cassandra Python Driver : execute_async consumes lots of memory?
Date Mon, 07 Nov 2016 16:51:05 GMT

We are trying to inject millions to data into a table by executing Batches of PreparedStatments.

We found that when we use 'session.execute(batch)', it write more data but very very slow.
However if we use  'session.execute_async(batch)' then its relatively fast but when it reaches
certain limit, its fillup the memory (python process)

Our implementation:
Cassandra 3.7.0 cluster  ring with 3 nodes (RedHat, 150GB Disk, 8GB of RAM each)

Python 2.7.12

Anybody know how to reduce the memory use of Cassandra-python driver API specifically for
execute_async? Thank you!

===CODE ======================================
      sqlQuery = "INSERT INTO tableV  (id, sample_name, pos, ref_base, var_base) values (?,?,?,?,?)"
       random_numbers_for_strains = random.sample(xrange(1,300), 200)
        random_numbers = random.sample(xrange(1,2000000), 200000)

        totalCounter  = 0
        c = 0
        time_init = time.time()
        for random_number_strain in random_numbers_for_strains:

            sample_name = None
            sample_name = 'sample'+str(random_number_strain)

            cassandraCluster = CassandraCluster.CassandraCluster()
            cluster = cassandraCluster.create_cluster_with_protocol2()
            session = cluster.connect();
            #session.default_timeout = 1800

            preparedStatement = session.prepare(sqlQuery)

            counter = 0
            c = c + 1

            for random_number in random_numbers:

                totalCounter += 1
                if counter == 0 :
                    batch = BatchStatement()

                counter += 1
                if totalCounter % 10000 == 0 :
                    print "Total Count "+ str(totalCounter)

                batch.add(preparedStatement.bind([ uuid.uuid1(), sample_name, random_number,
random.choice('GT'), random.choice('AC')]))
                if counter % 50 == 0:
                    batch = None
                    del batch
                    counter = 0

            random_number= None
            del random_number
            preparedStatement = None
            session = None
            del session
            cluster = None
            del cluster
            cassandraCluster = None
            del cassandraCluster

===CODE ======================================

Kind regards,
Rajesh Radhakrishnan

The information contained in the EMail and any attachments is confidential and intended solely
and for the attention and use of the named addressee(s). It may not be disclosed to any other
person without the express authority of Public Health England, or the intended recipient,
or both. If you are not the intended recipient, you must not disclose, copy, distribute or
retain this message or any part of it. This footnote also confirms that this EMail has been
swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening
or saving.
View raw message