cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "mck (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-14247) SASI tokenizer for simple delimiter based entries
Date Sun, 25 Feb 2018 21:28:00 GMT


mck commented on CASSANDRA-14247:

[~mkjellman], have forced pushed the branch again. (let me know if you want to be adding checkpoint
commits rather than overwriting the existing commit.)

This adds the test file {{test/resources/tokenization/world_cities_a.csv}}, and a unit test
to match. The other unit test methods have been updated to use different delimiters as appropriate
for the existing test data files.

Example corridor testing…
create table test ( one text, two int, three text, PRIMARY KEY (one,two) );

# insert a new row, with the contents of test/resources/tokenization/world_cities_a.csv going
into column 'three'.

create CUSTOM INDEX on test (three) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH
OPTIONS = { 'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer',
'delimiter': ',', 'mode': 'prefix', 'analyzed': 'true'};

select one,two from test where three LIKE 'azzazl' ALLOW FILTERING;

> SASI tokenizer for simple delimiter based entries
> -------------------------------------------------
>                 Key: CASSANDRA-14247
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: sasi
>            Reporter: mck
>            Assignee: mck
>            Priority: Major
>             Fix For: 4.0, 3.11.x
> Currently SASI offers only two tokenizer options:
>  - NonTokenizerAnalyser
>  - StandardAnalyzer
> The latter is built upon Snowball, powerful for human languages but overkill for simple
> A simple tokenizer is proposed here. The need for this arose as a workaround of CASSANDRA-11182,
and to avoid the disk usage explosion when having to resort to {{CONTAINS}}. See
> Example use of this would be:
> {code}
> CREATE CUSTOM INDEX span_annotation_query_idx 
>     ON zipkin2.span (annotation_query) USING 'org.apache.cassandra.index.sasi.SASIIndex'

>     WITH OPTIONS = {
>         'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer',

>         'delimiter': 'â–‘',
>         'case_sensitive': 'true', 
>         'mode': 'prefix', 
>         'analyzed': 'true'};
> {code}
> Original credit for this work goes to

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message