cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 翟玉勇 (JIRA) <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-12992) when mapreduce create sstables and load to cassandra cluster,then drop the table there are much data file not moved to snapshot
Date Wed, 06 Sep 2017 14:47:00 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-12992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

翟玉勇 updated CASSANDRA-12992:
----------------------------
    Description: 
{code}
when mapreduce create sstables and load to cassandra cluster,then drop the table there are
much data file not move to snapshot,

nodetool clearsnapshot can not free the disk,

wo must Manual delete the files 


cassandra table schema:

CREATE TABLE test.st_platform_api_restaurant_export (
    id_date text PRIMARY KEY,
    dt text,
    eleme_order_total double,
    order_amt bigint,
    order_date text,
    restaurant_id int,
    total double
) WITH bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = 'restaurant'
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 2592000
    AND gc_grace_seconds = 1800
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';


mapreduce job:
CREATE EXTERNAL TABLE st_platform_api_restaurant_export_h2c_sstable
(
    id_date string,
    order_amt bigint,
    total double,
    eleme_order_total double,
    order_date string,
    restaurant_id int,
    dt string)  STORED BY 'org.apache.hadoop.hive.cassandra.bulkload.CqlBulkStorageHandler'
TBLPROPERTIES (
'cassandra.output.keyspace.username' = 'cassandra',
'cassandra.output.keyspace'='test',
'cassandra.output.partitioner.class'='org.apache.cassandra.dht.Murmur3Partitioner',
'cassandra.output.keyspace.passwd'='cassandra',
'mapreduce.output.basename'='st_platform_api_restaurant_export',
'cassandra.output.thrift.address'='casandra cluster ips',
'cassandra.output.delete.source'='true',
'cassandra.columnfamily.insert.st_platform_api_restaurant_export'='insert into test.st_platform_api_restaurant_export(id_date,order_amt,total,eleme_order_total,order_date,restaurant_id,dt)values(?,?,?,?,?,?,?)',
'cassandra.columnfamily.schema.st_platform_api_restaurant_export'='CREATE TABLE test.st_platform_api_restaurant_export
(id_date text PRIMARY KEY,dt text,eleme_order_total double,order_amt bigint,order_date text,restaurant_id
int,total double)');
{code}

  was:
when mapreduce create sstables and load to cassandra cluster,then drop the table there are
much data file not move to snapshot,

nodetool clearsnapshot can not free the disk,

wo must Manual delete the files 


cassandra table schema:

CREATE TABLE test.st_platform_api_restaurant_export (
    id_date text PRIMARY KEY,
    dt text,
    eleme_order_total double,
    order_amt bigint,
    order_date text,
    restaurant_id int,
    total double
) WITH bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = 'restaurant'
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 2592000
    AND gc_grace_seconds = 1800
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';


mapreduce job:
CREATE EXTERNAL TABLE st_platform_api_restaurant_export_h2c_sstable
(
    id_date string,
    order_amt bigint,
    total double,
    eleme_order_total double,
    order_date string,
    restaurant_id int,
    dt string)  STORED BY 'org.apache.hadoop.hive.cassandra.bulkload.CqlBulkStorageHandler'
TBLPROPERTIES (
'cassandra.output.keyspace.username' = 'cassandra',
'cassandra.output.keyspace'='test',
'cassandra.output.partitioner.class'='org.apache.cassandra.dht.Murmur3Partitioner',
'cassandra.output.keyspace.passwd'='cassandra',
'mapreduce.output.basename'='st_platform_api_restaurant_export',
'cassandra.output.thrift.address'='casandra cluster ips',
'cassandra.output.delete.source'='true',
'cassandra.columnfamily.insert.st_platform_api_restaurant_export'='insert into test.st_platform_api_restaurant_export(id_date,order_amt,total,eleme_order_total,order_date,restaurant_id,dt)values(?,?,?,?,?,?,?)',
'cassandra.columnfamily.schema.st_platform_api_restaurant_export'='CREATE TABLE test.st_platform_api_restaurant_export
(id_date text PRIMARY KEY,dt text,eleme_order_total double,order_amt bigint,order_date text,restaurant_id
int,total double)');



> when mapreduce create sstables and load to cassandra cluster,then drop the table there
are much data file not moved to snapshot
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12992
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12992
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: cassandra 2.1.15
>            Reporter: 翟玉勇
>            Priority: Minor
>         Attachments: after-droptable.png, before-droptable.png
>
>
> {code}
> when mapreduce create sstables and load to cassandra cluster,then drop the table there
are much data file not move to snapshot,
> nodetool clearsnapshot can not free the disk,
> wo must Manual delete the files 
> cassandra table schema:
> CREATE TABLE test.st_platform_api_restaurant_export (
>     id_date text PRIMARY KEY,
>     dt text,
>     eleme_order_total double,
>     order_amt bigint,
>     order_date text,
>     restaurant_id int,
>     total double
> ) WITH bloom_filter_fp_chance = 0.01
>     AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
>     AND comment = 'restaurant'
>     AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
>     AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
>     AND dclocal_read_repair_chance = 0.1
>     AND default_time_to_live = 2592000
>     AND gc_grace_seconds = 1800
>     AND max_index_interval = 2048
>     AND memtable_flush_period_in_ms = 0
>     AND min_index_interval = 128
>     AND read_repair_chance = 0.0
>     AND speculative_retry = '99.0PERCENTILE';
> mapreduce job:
> CREATE EXTERNAL TABLE st_platform_api_restaurant_export_h2c_sstable
> (
>     id_date string,
>     order_amt bigint,
>     total double,
>     eleme_order_total double,
>     order_date string,
>     restaurant_id int,
>     dt string)  STORED BY 'org.apache.hadoop.hive.cassandra.bulkload.CqlBulkStorageHandler'
> TBLPROPERTIES (
> 'cassandra.output.keyspace.username' = 'cassandra',
> 'cassandra.output.keyspace'='test',
> 'cassandra.output.partitioner.class'='org.apache.cassandra.dht.Murmur3Partitioner',
> 'cassandra.output.keyspace.passwd'='cassandra',
> 'mapreduce.output.basename'='st_platform_api_restaurant_export',
> 'cassandra.output.thrift.address'='casandra cluster ips',
> 'cassandra.output.delete.source'='true',
> 'cassandra.columnfamily.insert.st_platform_api_restaurant_export'='insert into test.st_platform_api_restaurant_export(id_date,order_amt,total,eleme_order_total,order_date,restaurant_id,dt)values(?,?,?,?,?,?,?)',
> 'cassandra.columnfamily.schema.st_platform_api_restaurant_export'='CREATE TABLE test.st_platform_api_restaurant_export
(id_date text PRIMARY KEY,dt text,eleme_order_total double,order_amt bigint,order_date text,restaurant_id
int,total double)');
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message