roger pence

Potentially, each day is crucial in the total development

Search
Advanced Search
Folder to search:
Limit search to one of these properties:

    How to create a FlexSearch index

    install FlexSearch

    FlexSearch uses Json documents to define and populate its indices. Here is the markdown frontmatter that needs indexed for this project.

    ---
    title: Bump Version
    description: This Go program makes it easy to... 
    date_created: 2026-02-25
    date_updated: 2026-06-04
    date_published: false
    rank: 0
    pinned:
    tags:
      - go
      - git
      - cli
    ---
    
    Figure 1. Markdown example to be indexed
    index = new Document({
        document: {
            id: "id",
            store: [
                "id",
                "title",
                "url",
                "tags",
                "folder",
                "description",
            ],
            index: ["title", "url", "content", "tags", "description"],
            tag: [{ field: "folder" }],
        },
        tokenize: "forward",
        encoder: Charset.Normalize,
    });
    

    This article talks about creating a single FlexSearch index. However, if its prudent for your application, FlexSearch supports multiple indexes.

    For an elevently site, we'll get the Json with a Nunjucks template.

    How you convert your content into Json is up to you. For this example, the content is primary markdown documents, we so need a way to generate Json index data from these markdown documents.

    Markdown documents have YAML frontmatter in them at the start of the document, deliimited by three dashes (---).

    In markdown-friendly projects, (like Astro, Eleventy, Svelte, and many others)

    ---
    title: Bump Version
    description: This Go program makes it easy to increment a package version number and promote that number to GitHub. 
    date_created: 2026-02-25
    date_updated: 2026-06-04
    date_published:
    rank: 0
    pinned:
    tags:
      - go
      - git
      - cli
    ---
    
    [content here]
    

    Nunjucks template

    ---
    permalink: /data/search-index.json
    eleventyExcludeFromCollections: true
    ---
    [
      {%- for post in collections.posts -%}
        {%- if not loop.first -%},{%- endif -%}
        {
            "id": { loop.index0 | dump | safe }},
            "title": { post.data.title | dump | safe }},
            "description": { post.data.description | dump | safe }},
            "url": { post.url | dump | safe }},
            "tags": { post.data.tags | dump | safe }},
            "content": { post.templateContent | striptags | trim | dump | safe }},
            "folder": "post"
        }
      {%- endfor -%}
    ]
    

    document structure

      {
          "id": 11,
          "title": "Bump Version",
          "description": "This is a Go to increment a package ...",
          "url": "/posts/bump-version/",
          "tags": ["posts", "go", "git", "cli"],
          "content": "This Go program makes it easy...",
          "folder": "post"
      }
    

    /posts/flexsearch-worker/#search-worker-js

    To stress-test FlexSearch,

    • I created 5000 markdown documents where the content for each was randomly sized between 750 words and 2000 words.
    • Eleventy translated these 5000 documents into a 8mb Json file.
    • FlexSearch took about 2/3 of a second to index that Json document on each page load.
    • Chrome showed the tab using 200-300mb of memory (which was pretty much what my browser showed Amazon using)