Revised FlexSearch worker

This document describes the current search implementation in src/scripts/search-worker.js.

The problem this revision fixes

The FlexSearch docs discuss a tag property that names a property that can later be used to constrain the search to that property.

As I understand it, the tag property in the FlexSearch index defintion below identifies the folder property as search constraint.

index = new Document({
    document: {
        id: "id",
        store: ["id", "title", "url", "tags", "folder", "description"],
        //store: true,
        index: ["title", "url", "content", "tags", "description"],
        tag: "folder",
    },
    tokenize: "forward",
});

At runtime, if the search options argument includes a tag property, that property should apply the constraint to the search results--searching only index objects where the folder property is post.

const rawSearchResult = index.search(searchTerm, {
    tag: { folder: 'post' },
    enrich: true,
    limit: 20,
});

If it worked, this feature makes for a very clean and direct way to filter search results (within the search itself). Alas, I'm either doing something or this feature doesn't work, because it worked only intermittently for me.

The revised search function

The revised search function is at the end of this article.

This revision's swindle is that it searches unconditionally across all search objects and then filters the search results to include only those optional values specified.

Beware that the FlexSearch search limit should probably be a little higher than it needs to be if optional search values are used. Consider a search limit of 20. Whatever search results removed by the after-search filtering come of out that 20, so there may be way less than 20, even zero, results to show.

The other downside is that this version is just butt ugly. Its use of flatMap, reduce, and filter are challenging to quickly grasp.

The revised function supports three behaviors:

1. Search across indexed content with FlexSearch Document.
2. Optionally constrain matches to one indexed field (title, content, or tags).
3. Optionally constrain results to a folder (post or link).

It also deduplicates results by document id and adds metadata (sourceFields) indicating which indexed field(s) matched.

Inputs

search(payload) expects:

• payload.query: search text.
• payload.property_constraint: one of all, title, content, tags.
• payload.folder_constraint: one of all, post, posts, link, links.

Helper Functions

normalizeFolderConstraint(folder)

Normalizes folder values from UI controls:

• all (or empty) -> null (no folder constraint)
• posts -> post
• links -> link
• otherwise lowercased value

groupByDocId(results)

Takes FlexSearch enriched field-grouped results and returns a map of:

• key: doc.id
• value: array of field names that matched (title, content, tags, etc.)

This is later written to each result as sourceFields.

Search Flow

1. Normalize constraints

The function extracts and normalizes constraints:

• folder -> normalizedFolderConstraint
• property -> normalizedPropertyConstraint

2. Run one FlexSearch query

It runs:

index.search(searchTerm, { enrich: true, limit: 20 })

This returns grouped results by indexed field.

3. Apply property constraint (field scope)

If property_constraint is one of allowed fields (title, content, tags), it keeps only the matching field group from the FlexSearch response.

If property is all (or invalid), it keeps all field groups.

4. Build source-field map

groupByDocId(scopedSearchResult) builds a lookup of matching fields per doc id.

5. Flatten + dedupe

The function flattens all matches into one list and deduplicates by id.

For each unique doc:

• adds sourceFields (from the lookup)
• pushes the enriched doc into final results

6. Apply folder constraint

Finally, it filters deduped docs by folder when requested:

• no folder constraint -> all docs pass
• folder constraint (post or link) -> only docs with matching doc.folder

Why Folder Filtering Is Post-Search

The current implementation intentionally applies folder filtering in JavaScript after enrichment/dedupe.

Reason: in this runtime, FlexSearch tag filtering did not return expected matches for this dataset/configuration. The post-filter path is deterministic and currently correct.

Output Shape

Returns an array of enriched doc objects from store:

• id
• title
• url
• tags
• folder
• sourceFields (added by this function)

Example Behavior (query = "json")

With current data:

• folder all + property all -> 6 hits
• folder post + property all -> 5 hits
• folder link + property all -> 1 hit

And property constraints narrow within those folder results (for example, tags only).

The revised search function

function search(payload) {
    const searchTerm = payload.query;
    const constrainedProperty = payload.property_constraint;
    const constrainedFolder = payload.folder_constraint;
    const normalizedFolderConstraint =
        normalizeFolderConstraint(constrainedFolder);
    const normalizedPropertyConstraint = constrainedProperty
        ?.trim()
        .toLowerCase();

    const searchArgs = {
        enrich: true,
        limit: 20,
    };

    const rawSearchResult = index.search(searchTerm, searchArgs);
    const scopedSearchResult =
        normalizedPropertyConstraint &&
        normalizedPropertyConstraint !== "all" &&
        availableSearchConstraintFields.includes(normalizedPropertyConstraint)
            ? rawSearchResult.filter(
                  ({ field }) => field === normalizedPropertyConstraint,
              )
            : rawSearchResult;

    const uniqueIdCombos = groupByDocId(scopedSearchResult);
    console.log(JSON.stringify(uniqueIdCombos, null, 4));

    const searchResult = scopedSearchResult
        .flatMap((item) => item.result)
        .reduce((acc, current) => {
            if (!acc.find((item) => item.id === current.id)) {
                current.doc["sourceFields"] = uniqueIdCombos[current.id];
                acc.push(current.doc);
            }
            return acc;
        }, [])
        .filter((doc) => {
            if (!normalizedFolderConstraint) return true;
            return doc.folder === normalizedFolderConstraint;
        });

    console.log("searchResult", JSON.stringify(searchResult, null, 4));

    return searchResult;
}

Deciphering this code:

const searchResult = scopedSearchResult
    .flatMap((item) => item.result)
    .reduce((acc, current) => {
        if (!acc.find((item) => item.id === current.id)) {
            current.doc["sourceFields"] = uniqueIdCombos[current.id];
            acc.push(current.doc);
        }
        return acc;
    }, [])
    .filter((doc) => {
        if (!normalizedFolderConstraint) return true;
        return doc.folder === normalizedFolderConstraint;
    });

flatmap

Applied against the search objects, this code

const test = scopedSearchResult.flatMap((item) => item.result);

produces this:

[
    {
        "id": 1000,
        "doc": {
            "id": 1000,
            "title": "https://github.com/wesbos/JSON-Alexander",
            "url": "https://github.com/wesbos/JSON-Alexander",
            "tags": "links,json,web-dev",
            "folder": "link",
            "description": "A really good JSON viewer browser Extension."
        }
    },
    ...
]

flatmap and reduce

Applied against the search objects, this code

const test = scopedSearchResult
    .flatMap((item) => item.result)
    .reduce((acc, current) => {
        if (!acc.find((item) => item.id === current.id)) {
            current.doc["sourceFields"] = uniqueIdCombos[current.id];
            acc.push(current.doc);
        }
        return acc;
    }, []);

produces this:

[
    {
        "id": 1000,
        "title": "https://github.com/wesbos/JSON-Alexander",
        "url": "https://github.com/wesbos/JSON-Alexander",
        "tags": "links,json,web-dev",
        "folder": "link",
        "description": "A really good JSON viewer browser extension.",
        "sourceFields": [
            "title",
            "url",
            "content",
            "tags",
            "description"
        ]
    },
    ...
]

See note on sourceFields below.

flatMap, reduce, and filter

When normalizedFolderConstraint contains a folder name, the filter method limits the results to only search objects where the folder property is equal to the normalizedFolderConstraint variable.

const searchResult = scopedSearchResult
    .flatMap((item) => item.result)
    .reduce((acc, current) => {
        if (!acc.find((item) => item.id === current.id)) {
            current.doc["sourceFields"] = uniqueIdCombos[current.id];
            acc.push(current.doc);
        }
        return acc;
    }, [])
    .filter((doc) => {
        if (!normalizedFolderConstraint) return true;
        return doc.folder === normalizedFolderConstraint;
    });

Identifying where the search term was found

The search results includes a sourceFields property that shows the properties of the search object in which the search term was found. The original intent of this was be to be able to show the user where the search term was found. I'm not sure that's of use, but it remains for now.

This code:

function groupByDocId(results) {
    return results.reduce((acc, { field, result }) => {
        for (const { doc } of result) {
            (acc[doc.id] ??= []).push(field);
        }
        return acc;
    }, {});
}

...

const uniqueIdCombos = groupByDocId(scopedSearchResult);

produces this object which provides, by search object id, the properties of the search object in which the search term was found.

{
    "2": [
        "content"
    ],
    "4": [
        "content"
    ],
    "5": [
        "content"
    ],
    "10": [
        "content"
    ],
    "12": [
        "tags"
    ],
    "16": [
        "content"
    ],
    "1000": [
        "title",
        "url",
        "content",
        "tags",
        "description"
    ]
}

The full original worker

This original doesn't include the code to search specific properties, but it does include the abiility to constrain the search by the property specified with the tag option (which I couldn't make work!)

import { Document } from "/scripts/flexsearch.bundle.module.min.js";

const availableSearchConstraintFields = ["title", "content", "tags"];

let index = null;

self.addEventListener("message", async (event) => {
    const { type, data } = event.data;

    if (type === "init") {
        /**
         * The 'store' property defines the search result output properties.
         * The 'index' property defines the properties that are indexed.
         */
        index = new Document({
            document: {
                id: "id",
                store: ["id", "title", "url", "tags", "folder"],
                //store: true,
                index: ["title", "url", "content", "tags"],
                tag: "folder",
            },
            tokenize: "forward",
        });

        /**
         * Search result properties are fetched from the index object.
         */
        for (const post of data.posts) {
            index.add({
                id: post.id,
                title: post.title,
                content: post.content,
                tags: (post.tags || []).join(","),
                url: post.url,
                folder: post.folder,
            });
        }

        self.postMessage({ type: "ready" });
    } else if (type === "search") {
        const results = search(data.payload);
        self.postMessage({ type: "results", data: results });
    }
});

function groupByDocId(results) {
    return results.reduce((acc, { field, result }) => {
        for (const { doc } of result) {
            (acc[doc.id] ??= []).push(field);
        }
        return acc;
    }, {});
}

function search(payload) {
    const searchTerm = payload.query;
    const constrainedProperty = payload.property_constraint;
    const constrainedFolder = payload.folder_constraint;

    const searchArgs = {
        enrich: true,
        limit: 20,
    };

    if (constrainedFolder !== "all") {
        searchArgs["tag"] = { folder: constrainedFolder };
    }

    const rawSearchResult = index.search(searchTerm, searchArgs);

    const uniqueIdCombos = groupByDocId(rawSearchResult);
    console.log(JSON.stringify(uniqueIdCombos, null, 4));

    const searchResult = rawSearchResult
        .flatMap((item) => item.result)
        .reduce((acc, current) => {
            if (!acc.find((item) => item.id === current.id)) {
                current.doc["sourceFields"] = uniqueIdCombos[current.id];
                acc.push(current.doc);
            }
            return acc;
        }, []);

    return searchResult;
}