cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Reynolds <blreynol...@gmail.com>
Subject Building SSTables for bulk load with composite column name going slow
Date Fri, 20 Jul 2012 14:48:06 GMT
Hi

I'm using Cassandra 1.1.0 from Datastax and attempting to build SST files
to bulk load for a ColumnFamily with a single column with a Composite name.
 For small inputs it seems to work and load correctly.  When I use larger
dataset it is noticeably slower than code that isn't using a composite
column.

Is there a problem using SSTableSimpleUnsortedWriter when a column has a
composite name?  Am I doing it correctly?

I tried building the Composite outside the while loop and just copying it
before adding components in each iteration but that didn't seem to help.

thanks
bri

Code based on bulk loading tutorial:

        List<AbstractType<?>> compositeList = new
ArrayList<AbstractType<?>>();
        compositeList.add(UTF8Type.instance);
        compositeList.add(UTF8Type.instance);

        SSTableSimpleUnsortedWriter usersWriter = new
SSTableSimpleUnsortedWriter(
                directory,
                new RandomPartitioner(),
                keyspace,
                columnFamily,
                CompositeType.getInstance(compositeList),
                null,
                32);

        String line;
        int lineNumber = 1;
        CsvEntry entry = new CsvEntry();
        // There is no reason not to use the same timestamp for every
column in that example.
        long timestamp = System.currentTimeMillis() * 1000;
        while ((line = reader.readLine()) != null)
        {
            if (entry.parse(line, lineNumber))
            {
                if ( (lineNumber % 10000) == 0) {
                    System.out.println("Line "+lineNumber);
                }
                CompositeType.Builder builder = new CompositeType.Builder(
CompositeType.getInstance(compositeList) );
                String siteKey = entry.key;
                usersWriter.newRow(bytes( siteKey ));
                builder.add(bytes(entry.part1));
                builder.add(bytes(entry.part2));
                usersWriter.addColumn(builder.build(), bytes(entry.date),
timestamp);
            }
            lineNumber++;
        }

Mime
View raw message