<a id="fts-working-with"></a>

# Working with Full Text Search

Full text search retrieves rows by matching the *terms* in a query against the
text stored in a column, and ranks the matches by relevance using the
[BM25](https://en.wikipedia.org/wiki/Okapi_BM25) algorithm. It is the right
tool when the exact words matter — searching articles, product descriptions,
support tickets, or log messages.

Full text search is one half of ScyllaDB’s
[Vector and Text Search](https://cloud.docs.scylladb.com/stable/vector-search/index.md) feature. It runs on the
same indexing nodes as [vector search](https://cloud.docs.scylladb.com/stable/vector-search/work-with-vector-search.md)
and is enabled by the same deployment steps, so everything in
[Vector and Text Search Deployments](https://cloud.docs.scylladb.com/stable/vector-search/vector-search-clusters.md)
applies here as well.

#### NOTE
Full text search requires **ScyllaDB 2026.3.0 or later**. Vector search is
available on clusters running ScyllaDB 2025.4.3 or later. See
[Feature Compatibility Matrix](https://cloud.docs.scylladb.com/stable/vector-search/vector-search-clusters.md#vs-feature-compatibility).

<a id="fts-prerequisites"></a>

## Prerequisites

* A cluster with Vector and Text Search enabled, running ScyllaDB 2026.3.0 or
  later. See [Creating a New Cluster with Vector and Text Search Enabled](https://cloud.docs.scylladb.com/stable/vector-search/vector-search-clusters.md#vs-create-cluster) to create one, or
  [Enabling Vector and Text Search on an Existing Cluster](https://cloud.docs.scylladb.com/stable/vector-search/vector-search-clusters.md#vs-enable-existing) to add indexing nodes to a cluster you already
  have.
* `cqlsh`, the web-based CQL console in the ScyllaDB Cloud UI, or any CQL
  driver.

<a id="fts-workflow"></a>

## Workflow

1. Create a table with a `text`, `varchar`, or `ascii` column holding the
   text you want to search.
2. Create a full-text index on that column.
3. Wait for the index to finish building.
4. Run `BM25()` queries against the indexed column.

<a id="fts-create-index"></a>

## Creating a Full-Text Index

Create a full-text index with `CREATE CUSTOM INDEX` and the
`fulltext_index` class:

```cql
CREATE KEYSPACE blog
  WITH replication = {'class': 'NetworkTopologyStrategy', 'replication_factor': 3};

CREATE TABLE blog.articles (
  id int PRIMARY KEY,
  author text,
  body text
);

CREATE CUSTOM INDEX articles_body_fts ON blog.articles(body)
  USING 'fulltext_index';
```

`IF NOT EXISTS` is supported, and the index name may be omitted — ScyllaDB
then derives one from the table and column names, such as
`articles_body_idx`.

Use `DESCRIBE INDEX` to inspect an existing index:

```cql
DESCRIBE INDEX blog.articles_body_fts;
```

<a id="fts-indexable-columns"></a>

### Which Columns Can Be Indexed

A full-text index can only be created on a text-typed column:

| Column                                     | Supported                                                                                                                                                          |
|--------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `text`, `varchar`, `ascii` regular column  | Yes.                                                                                                                                                               |
| Clustering key column                      | Yes.                                                                                                                                                               |
| Partition key column                       | No. ScyllaDB rejects the statement with `Cannot create secondary index<br/>on partition key column <name>`.                                                        |
| Any non-text type, such as `int` or `blob` | No. ScyllaDB rejects the statement with `Fulltext index is only<br/>supported on text, varchar, or ascii columns, but column <name> has an<br/>incompatible type`. |

A single table can carry several full-text indexes, as long as each one
targets a different column. A full-text index and a
[vector index](https://cloud.docs.scylladb.com/stable/vector-search/work-with-vector-search.md#vs-vector-index) can also coexist on the same table —
see [Combining Full Text and Vector Search](#fts-hybrid).

<a id="fts-index-options"></a>

### Index Options

Two options control how the index treats text. Both are set at creation time
with `WITH OPTIONS` and cannot be changed afterwards:

| Option      | Default    | Description                                                                                                                                                                                                                           |
|-------------|------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `analyzer`  | `standard` | How text is split into terms and normalized. See<br/>[Choosing an Analyzer](#fts-analyzers).                                                                                                                                          |
| `positions` | `true`     | Whether token positions are stored. **Phrase queries need them**: an<br/>index created with `false` answers term and boolean queries but<br/>rejects any phrase query. See [Phrase Queries and the positions Option](#fts-positions). |
```cql
CREATE CUSTOM INDEX articles_body_fts ON blog.articles(body)
  USING 'fulltext_index'
  WITH OPTIONS = {'analyzer': 'english'};
```

Option values are case-insensitive, so `'english'` and `'ENGLISH'` are
equivalent. An unknown option, or an unsupported value, is rejected when the
index is created:

```none
Invalid value in option 'analyzer' for fulltext index: 'klingon'. Supported
are case-insensitive: standard, english, german, french, spanish, italian,
portuguese, russian, simple, whitespace

Invalid value in option 'positions' for fulltext index: 'sometimes'.
Supported are case-insensitive: false, true

Unsupported option nonsense for fulltext index
```

There is no `ALTER INDEX` statement in CQL, so changing either option means
dropping the index and creating it again. The replacement index is rebuilt from
the base table, and queries against the column fail until it is serving again.

<a id="fts-index-build"></a>

### Index Building

Full-text indexes are populated through Change Data Capture, exactly like
vector indexes, so the same requirements apply: the keyspace must use tablets
and CDC is enabled automatically on the base table. See
[Tablets Requirement](https://cloud.docs.scylladb.com/stable/vector-search/work-with-vector-search.md#vector-search-tablets-info) and
[CDC Requirements](https://cloud.docs.scylladb.com/stable/vector-search/work-with-vector-search.md#vs-vector-index-cdc).

After you create the index, the indexing node discovers it and scans the base
table. Queries fail until that scan completes:

```none
# Immediately after CREATE, before the indexing node has picked up the index
Vector Store error: HTTP status 404 Not Found, message: missing index: blog.articles_body_fts

# While the base table is being scanned
Vector Store error: HTTP status 503 Service Unavailable, message: Index
blog.articles_body_fts is not available yet as it is still being constructed,
progress: 39.804%
```

Retry until the query succeeds. Build time scales with the number of rows and
the size of the indexed text.

#### NOTE
`Vector Store` in these messages refers to the **indexing node** that
serves both vector and full-text indexes. The messages are reproduced here
as the server emits them.

<a id="fts-queries"></a>

## Running Full Text Search Queries

A full text search query has exactly one valid shape:

```cql
SELECT <columns> FROM <table>
  WHERE BM25(<column>, '<query>') > 0
  ORDER BY BM25(<column>, '<query>')
  LIMIT <n>;
```

For example:

```cql
SELECT id, body FROM blog.articles
  WHERE BM25(body, 'photosynthesis') > 0
  ORDER BY BM25(body, 'photosynthesis')
  LIMIT 10;
```

Every part of that shape is mandatory, and ScyllaDB rejects any deviation. The
rules are:

| Rule                                                                                                              | Error when violated                                                                                                                                   |
|-------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| The query must include `ORDER BY BM25()`.                                                                         | `Full-text search queries require an ORDER BY BM25() clause`                                                                                          |
| The query must include a `LIMIT`.                                                                                 | `Full-text search queries require a LIMIT`                                                                                                            |
| `LIMIT` must not exceed 1000.                                                                                     | `Full-text search queries require a LIMIT that is not greater than<br/>1000. LIMIT was 1001`                                                          |
| The search term must be identical in `WHERE` and `ORDER BY`.                                                      | `Full-text search queries must use the same search term in both WHERE<br/>and ORDER BY clauses`                                                       |
| The comparison value must be the literal `0`.                                                                     | `BM25 function comparison value must be the literal 0`                                                                                                |
| The only supported operator is `>`.                                                                               | `Unsupported ">=" relation for BM25 function restriction, only ">" is<br/>supported`                                                                  |
| `BM25()` cannot appear in the `SELECT` list.                                                                      | `BM25() is not supported in the SELECT clause`                                                                                                        |
| No other `WHERE` restriction is allowed, including on the partition<br/>key, and `ALLOW FILTERING` does not help. | `Full-text search queries do not support additional WHERE<br/>restrictions`                                                                           |
| Only one `BM25()` restriction is allowed per query.                                                               | `Full-text search queries support only one WHERE BM25() restriction`                                                                                  |
| `BM25()` and `ANN OF` cannot be combined.                                                                         | `BM25 and ANN cannot be combined in the same query`                                                                                                   |
| The targeted column must have a full-text index.                                                                  | `No fulltext index found for full-text search query`                                                                                                  |
| The query must include a `WHERE BM25()` restriction — an<br/>`ORDER BY BM25()` on its own is not enough.          | `Full-text search queries require a WHERE BM25() > 0 clause`                                                                                          |
| `WHERE` and `ORDER BY` must target the same column, not only the<br/>same search term.                            | `Full-text search queries must reference the same column in both<br/>WHERE and ORDER BY clauses`                                                      |
| `PER PARTITION LIMIT` is not supported.                                                                           | `Full-text search queries do not support per-partition limits`                                                                                        |
| Aggregate functions and `GROUP BY` are not supported.                                                             | `Full-text search queries cannot be run with aggregation`                                                                                             |
| `ORDER BY BM25()` cannot be combined with any other ordering, whether<br/>a regular column or a second `BM25()`.  | `bm25 ordering does not support any other ordering` (or<br/>`Similarity ordering does not support any other ordering`, depending<br/>on clause order) |

`ORDER BY BM25()` does not accept `ASC` or `DESC`; results always come
back with the highest relevance first.

#### NOTE
Because a full text search query cannot carry any additional `WHERE`
restriction, it cannot be narrowed to a partition or filtered on another
column. Retrieve the top matches first, then apply any further filtering in
your application.

<a id="fts-paging"></a>

### Result Size and Paging

`LIMIT` is capped at 1000, and full text search results are never paged —
the server always returns the whole result set in a single response. If the
result set is larger than the page size your driver requested, ScyllaDB
returns all of the rows anyway and attaches a warning to the response:

```none
Paging is not supported for Full-Text Search queries. The entire result set
has been returned.
```

Size the `LIMIT` to what your application will actually use, since every
matching row is transferred at once. To reach beyond 1000 matches, narrow the
query with additional terms rather than trying to page through it.

<a id="fts-query-language"></a>

## Text Query Language

The string you pass to `BM25()` is written in its own text query language —
it is not CQL, and it is not treated as a literal phrase. The following
constructs are supported.

<a id="fts-terms"></a>

### Terms

A bare term matches rows whose indexed text contains that term. Under the
default `standard` analyzer — and every analyzer except `whitespace` —
matching is case-insensitive, because both the stored text and the query are
lowercased:

```cql
-- These two queries return the same rows.
SELECT id FROM blog.articles
  WHERE BM25(body, 'photosynthesis') > 0
  ORDER BY BM25(body, 'photosynthesis') LIMIT 10;

SELECT id FROM blog.articles
  WHERE BM25(body, 'PHOTOSYNTHESIS') > 0
  ORDER BY BM25(body, 'PHOTOSYNTHESIS') LIMIT 10;
```

With the `whitespace` analyzer, which lowercases nothing, the two queries
match different rows. See [Choosing an Analyzer](#fts-analyzers).

Several terms separated by whitespace are combined with an implicit **OR**,
so a row matching any one of them is returned — `jupiter saturn` is
equivalent to `jupiter OR saturn`. Rows matching more of the terms rank
higher. Use an explicit [AND](#fts-boolean) when every term must be
present.

<a id="fts-boolean"></a>

### Boolean Operators

`AND`, `OR`, and `NOT` combine terms, and parentheses group them.

#### IMPORTANT
**Operators must be uppercase.** Unlike CQL, which is case-insensitive, the
text query language only recognizes `AND`, `OR`, and `NOT` in upper
case. A lowercase `and`, `or`, or `not` is not an operator — it is
treated as an ordinary search term. Under `standard` and `english` it is
then dropped as a stop word; under `simple`, `whitespace`, and the
non-English language analyzers it stays in the query as a term to match.

Either way the effect is the same and it fails silently: the terms end up
joined by the implicit `OR`, so `database and distributed` does not
narrow anything — it returns the same rows as
`database OR distributed`.

```cql
-- Rows that contain both terms.
SELECT id FROM blog.articles
  WHERE BM25(body, 'database AND distributed') > 0
  ORDER BY BM25(body, 'database AND distributed') LIMIT 10;

-- Rows that contain either term.
SELECT id FROM blog.articles
  WHERE BM25(body, 'jupiter OR saturn') > 0
  ORDER BY BM25(body, 'jupiter OR saturn') LIMIT 10;

-- Rows that contain the first term but not the second.
SELECT id FROM blog.articles
  WHERE BM25(body, 'python NOT snake') > 0
  ORDER BY BM25(body, 'python NOT snake') LIMIT 10;

-- Grouping.
SELECT id FROM blog.articles
  WHERE BM25(body, '(jupiter OR saturn) AND planet NOT rings') > 0
  ORDER BY BM25(body, '(jupiter OR saturn) AND planet NOT rings') LIMIT 10;
```

<a id="fts-phrases"></a>

### Phrases

Double quotes match a sequence of adjacent terms in order. Because the CQL
string itself is single-quoted, the double quotes go inside it:

```cql
SELECT id FROM blog.articles
  WHERE BM25(body, '"theory of relativity"') > 0
  ORDER BY BM25(body, '"theory of relativity"') LIMIT 10;
```

Without the quotes the words become independent terms combined with an
implicit `OR`, so word order and adjacency stop mattering —
`theory of relativity` also matches text that says *relativity theory*,
while the quoted phrase does not.

Phrase matching relies on the `positions` option, which is on by default. An
index created with `'positions': 'false'` rejects phrase queries — see
[Phrase Queries and the positions Option](#fts-positions).

<a id="fts-analysis"></a>

### How Text Is Analyzed

The index applies the same analysis to the stored text when it is indexed and
to your query when it is parsed — which is why the two always agree on what a
term is. Analysis is made up of up to four steps:

| Step         | Behavior                                                                                                                                                                                                                                                                                                                                                                        |
|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Tokenization | Splits text into terms. Every analyzer except `whitespace` splits on<br/>non-alphanumeric characters, so punctuation separates terms:<br/>`wide-column` is indexed as `wide` and `column`, and each matches<br/>on its own, while `high-throughput,` matches `throughput`. The<br/>`whitespace` analyzer splits on whitespace only, keeping<br/>`wide-column` as a single term. |
| Lowercasing  | Makes matching case-insensitive, so `PHOTOSYNTHESIS` and<br/>`photosynthesis` return the same rows. Performed by every analyzer<br/>except `whitespace`.                                                                                                                                                                                                                        |
| Stop words   | Drops very common words that carry little search signal. Performed by<br/>`standard` and the language analyzers, each in its own language, and<br/>**not** by `simple` or `whitespace`. Under the default analyzer,<br/>searching for `the` alone matches nothing, and `the database`<br/>returns the same rows as `database`.                                                  |
| Stemming     | Reduces words to a common root so that different inflections of the<br/>same word match each other, such as `run` matching *running*.<br/>Performed only by the language analyzers.                                                                                                                                                                                             |

Which of those steps run — and in which language — is set by the
[analyzer](#fts-analyzers).

<a id="fts-analyzers"></a>

### Choosing an Analyzer

The `analyzer` option selects the pipeline. The default, `standard`, is
**English-only**: it removes English stop words and does no stemming at all,
which makes it a reasonable choice for English text that should match on exact
word forms — and a poor one for any other language. For text in another
language, pick that language’s analyzer:

| Analyzer                                                                         | Splits on       | Lowercases   | Stop words    | Stemming      |
|----------------------------------------------------------------------------------|-----------------|--------------|---------------|---------------|
| `standard` (default)                                                             | punctuation     | Yes          | English       | No            |
| `simple`                                                                         | punctuation     | Yes          | None          | No            |
| `whitespace`                                                                     | whitespace only | No           | None          | No            |
| `english`, `german`, `french`, `spanish`, `italian`,<br/>`portuguese`, `russian` | punctuation     | Yes          | that language | that language |

Pick by what you need:

* `standard` — the default. Exact word forms, English stop words removed.
  <br/>
* A language analyzer — when you want different inflections of a word to
  match each other. With `english`, a search for `run` matches *running*
  and *runners*, and `walk` matches *walks*:
  <br/>
  ```cql
  CREATE CUSTOM INDEX articles_body_fts ON blog.articles(body)
    USING 'fulltext_index'
    WITH OPTIONS = {'analyzer': 'english'};
  ```
* `simple` — like `standard` but keeps stop words, so a query for a
  common word such as `the` can still match.
  <br/>
* `whitespace` — no normalization at all beyond splitting on whitespace.
  Case and punctuation are preserved, so `Quick` matches only *Quick*, and
  `hello` does not match *hello,* — the stored term includes the comma.
  Use it for text where case or symbols are meaningful, such as identifiers
  or codes.
  <br/>

#### IMPORTANT
Choose the analyzer that matches the language of your text. The default
`standard` analyzer uses **English** stop words and does no stemming, so
on German text it indexes *die* (not an English stop word) and does not
match `lauf` against *laufen*. The `german` analyzer drops *die* and
stems *laufen* to `lauf`.

Because the analyzer is fixed for the life of the index, and because changing
it means a full rebuild, it is worth deciding before you load data.

<a id="fts-positions"></a>

### Phrase Queries and the positions Option

[Phrase queries](#fts-phrases) match on the recorded position of each
term, so they need the `positions` option, which is on by default. Setting it
to `false` makes the index smaller, at the cost of no longer being able to
answer phrase queries:

```cql
CREATE CUSTOM INDEX articles_body_fts ON blog.articles(body)
  USING 'fulltext_index'
  WITH OPTIONS = {'positions': 'false'};
```

Term and boolean queries work as usual against such an index, but a phrase
query is rejected rather than silently returning nothing:

```none
Vector Store error: HTTP status 400 Bad Request, message: index.bm25 request
error: fts: failed to parse query: The field 'body' does not have positions
indexed
```

Leave `positions` at its default unless you have measured the memory saving
and know that no query will ever need a phrase.

<a id="fts-unsupported-syntax"></a>

### Unsupported Query Syntax

Fuzzy matching (`term~1`) and prefix or wildcard matching (`term*`) are
parsed without error but match nothing. Do not rely on them.

Some characters are meaningful to the query parser and make the query fail
rather than match literally:

| Input                                                                     | Result                                                                                                                                                                            |
|---------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `:`                                                                       | Read as a field separator. `colon:term` fails with `failed to parse<br/>query: Field does not exist: 'colon'`.                                                                    |
| `(`, `)`                                                                  | Grouping. Balanced parentheses are valid syntax, but an unbalanced one<br/>is rejected with `failed to parse query: Syntax Error: <input>`, so<br/>`database (distributed` fails. |
| `[` `]` `{` `}` `^` and the quote characters `'` `"` and<br/>the backtick | Rejected with `failed to parse query: Syntax Error: <input>`.                                                                                                                     |
| A bare `AND`, `OR`, or `NOT`                                              | Rejected with `failed to parse query: Syntax Error: <input>`.                                                                                                                     |

Of the ASCII punctuation characters, exactly the eleven in `"'():[]^`{}` are
rejected when they appear inside a term. The rest are passed through to the
analyzer, which splits on them.

#### WARNING
Passing raw end-user input straight into `BM25()` can fail on these
characters. Using a bound parameter protects you from CQL injection, but the
value is still parsed as a query expression — so strip or replace the
characters above before searching. See [Using Full Text Search from a Driver](#fts-drivers).

<a id="fts-ranking"></a>

## Relevance Ranking

Matches are ordered by their BM25 score, highest first. BM25 scores a row on
how often the query terms occur in it, offset by how common those terms are
across the whole index and by the length of the indexed text: a term that
appears in nearly every row contributes little, and a short row that mentions
the term as often as a long one scores higher.

Scores are relative to the current contents of the index and are not
exposed — `BM25()` cannot appear in the `SELECT` list, so you get the
ordering rather than the numbers.

#### NOTE
Returning BM25 relevance scores in query results is planned for ScyllaDB
2026.4.0.

<a id="fts-write-latency"></a>

## Write-to-Query Latency

Inserts, updates, and deletes reach the full-text index asynchronously through
CDC, so a write becomes searchable a short time after it is acknowledged —
expect a few seconds. Deleting a row removes it from the index, and updating an
indexed column replaces the old text, so the previous terms stop matching.

If a query must reflect a write immediately, read the row by primary key
instead.

#### WARNING
**Cell-level TTL leaves stale entries in the index.** Expiration through
`USING TTL` on an `INSERT` or `UPDATE`, or through the
`default_time_to_live` table property, does not generate a CDC event. The
value becomes unreadable at its deadline but the indexing node never hears
about it, so the index keeps an entry for the expired row.

Query results stay correct — ScyllaDB filters the expired rows out — but
the stale entries still occupy memory and, more visibly, consume candidate
slots against your `LIMIT`. A query may then return fewer rows than both
the `LIMIT` and the number of live matches, and where expired rows
outnumber live ones it may return only a small fraction of them.

Use [per-row TTL](https://cloud.docs.scylladb.com/stable/vector-search/work-with-vector-search.md#vs-ttl) instead. It stores an absolute expiration
time in a dedicated column, and ScyllaDB deletes the row explicitly when
that time passes, which does emit a CDC event and does remove the entry
from the index. See [Cell-Level TTL Is Not Supported](https://cloud.docs.scylladb.com/stable/vector-search/work-with-vector-search.md#vs-ttl-cell-level).

<a id="fts-hybrid"></a>

## Combining Full Text and Vector Search

A table can carry both a full-text index and a vector index, which is what
makes hybrid search possible:

```cql
CREATE TABLE blog.docs (
  id int PRIMARY KEY,
  body text,
  embedding vector<float, 3>
);

CREATE CUSTOM INDEX docs_body_fts ON blog.docs(body) USING 'fulltext_index';
CREATE INDEX docs_embedding_ann ON blog.docs(embedding) USING 'vector_index';
```

The two indexes are queried separately — `BM25()` and `ANN OF` cannot
appear in the same statement:

```cql
-- Term matching.
SELECT id, body FROM blog.docs
  WHERE BM25(body, 'apple') > 0
  ORDER BY BM25(body, 'apple') LIMIT 10;

-- Semantic similarity.
SELECT id, body FROM blog.docs
  ORDER BY embedding ANN OF [1.0, 0.1, 0.0] LIMIT 10;
```

To build hybrid search, run both queries and merge the two result sets in your
application, for example with Reciprocal Rank Fusion, which combines the rank
of each row in each list without needing comparable scores.

#### NOTE
Native hybrid search — combining term matching and vector similarity in a
single query, with the fusion done by the database — is planned for
ScyllaDB 2026.4.0.

<a id="fts-drivers"></a>

## Using Full Text Search from a Driver

Full text search needs no driver-side support beyond ordinary CQL. Prepared
statements accept bind markers for the search term — in both the `WHERE`
and `ORDER BY` positions — and for the `LIMIT`.

The example below assumes an open `session`. Connecting to a ScyllaDB Cloud
cluster requires authentication, TLS, and a datacenter-aware load balancing
policy — see [Driver Examples](https://cloud.docs.scylladb.com/stable/vector-search/work-with-vector-search.md#vs-driver-python-example) for a complete connection
example.

```python
search = session.prepare(
    "SELECT id, body FROM articles "
    "WHERE BM25(body, ?) > 0 "
    "ORDER BY BM25(body, ?) "
    "LIMIT ?"
)


def full_text_search(query, limit=10):
    # The same term has to be bound to both markers; the server rejects the
    # statement if the two values differ.
    return session.execute(search, (query, query, limit))


for row in full_text_search("database AND distributed"):
    print(row.id, row.body)
```

Because the bound value is parsed as a query expression, sanitize end-user
input before passing it in — see [Unsupported Query Syntax](#fts-unsupported-syntax). To search for
the words a user typed without interpreting them as query syntax, keep only
word characters and whitespace, then drop anything the parser would read as an
operator:

```python
import re

# Keep letters, digits and underscore; replace every other character with a
# space. A whitelist is safer here than a list of reserved characters: it
# cannot fall behind the parser, and it cannot leave an unbalanced
# parenthesis or a stray operator behind.
NON_WORD = re.compile(r"[^\w\s]", re.UNICODE)
OPERATORS = {"AND", "OR", "NOT"}


def as_terms(user_input):
    """Reduce free-form input to a list of plain, safe search terms."""
    cleaned = NON_WORD.sub(" ", user_input)
    return [term for term in cleaned.split() if term not in OPERATORS]


terms = as_terms(user_input)
if terms:
    # Whitespace joins the terms with an implicit OR, which keeps the search
    # forgiving; join with " AND " instead when every term must be present.
    for row in full_text_search(" ".join(terms)):
        print(row.id, row.body)
```

Order matters in that function: characters are replaced *before* the operator
check. Stripping first turns `AND)` into `AND`, which the operator filter
then removes — whereas filtering first would leave `AND)` in place, and the
query would fail on the unbalanced parenthesis. Because `\w` is
Unicode-aware, accented and non-Latin words survive intact.

Stripping punctuation can split a word — `don't` becomes the two
terms `don` and `t` — but under the default `standard` analyzer that
costs you nothing on its own, because the indexed text is tokenized the same
way: a row containing *don’t* is indexed under `don` and `t` too, so it
still matches. Note that this does not hold for the `whitespace` analyzer,
which keeps punctuation, so there the indexed term is `don't` and neither
`don` nor `don t` matches it.

The reason to prefer the implicit `OR` for free-form input is that `AND`
requires *every* term to be present, so a single word the user typed that
appears in no document reduces the whole result set to nothing:

```cql
-- 'unicorn' appears in no document, so the AND query matches nothing
-- while the implicit OR still returns the rows that mention a database.
SELECT id FROM blog.articles
  WHERE BM25(body, 'database AND unicorn') > 0
  ORDER BY BM25(body, 'database AND unicorn') LIMIT 10;

SELECT id FROM blog.articles
  WHERE BM25(body, 'database unicorn') > 0
  ORDER BY BM25(body, 'database unicorn') LIMIT 10;
```

Note also that sanitizing changes only the surface form of the words, not
their meaning: a search for *don’t* will not reach text that says *do not*,
because those are different terms. Matching by meaning rather than by word is
what [vector search](https://cloud.docs.scylladb.com/stable/vector-search/work-with-vector-search.md) is for, and
[Combining Full Text and Vector Search](#fts-hybrid) combines the two.

See [ScyllaDB Drivers](https://cloud.docs.scylladb.com/stable/vector-search/work-with-vector-search.md#vs-drivers) for the list of supported drivers and
connection details.

<a id="fts-drop-index"></a>

## Dropping a Full-Text Index

```cql
DROP INDEX blog.articles_body_fts;
```

Dropping the index releases the memory it used on the indexing nodes and leaves
the base table untouched. Full text search queries against the column then fail
with `No fulltext index found for full-text search query`.

<a id="fts-limitations"></a>

## Limitations

* Full text search requires ScyllaDB 2026.3.0 or later.
* A full-text index can only target a `text`, `varchar`, or `ascii`
  column, and not a partition key column.
* `LIMIT` is mandatory and must not exceed 1000. Results are not paged.
* A full text search query cannot carry any other `WHERE` restriction, and
  cannot be combined with an `ANN OF` clause.
* Only one `BM25()` restriction is allowed per query.
* BM25 scores cannot be projected in the `SELECT` list.
* Fuzzy matching (`term~1`) and prefix matching (`term*`) are not
  available.
* Stemming is available only through the language analyzers, and the analyzer
  applies to the whole index — it cannot be chosen per query.
* Index options cannot be changed after creation — there is no
  `ALTER INDEX` statement. Drop the index and create it again instead.
* Cell-level TTL (`USING TTL`, `default_time_to_live`) is not supported:
  expired rows leave stale entries in the index. Use
  [per-row TTL](https://cloud.docs.scylladb.com/stable/vector-search/work-with-vector-search.md#vs-ttl) instead.

<a id="fts-next-steps"></a>

## Next Steps

* [Working with Vector and Text Search](https://cloud.docs.scylladb.com/stable/vector-search/work-with-vector-search.md)
  — the vector search half of the feature.
* [Full Text Search CQL Reference](https://cloud.docs.scylladb.com/stable/vector-search/reference-vector-search.md#fts-cql-reference) — the statements and index options on one page.
* [Troubleshooting](https://cloud.docs.scylladb.com/stable/vector-search/vector-search-troubleshooting.md) — common
  full text search problems.
* [FAQ](https://cloud.docs.scylladb.com/stable/vector-search/vector-search-faq.md)
