Pular para o conteúdo

Path filtering

Atualizado em Ver como Markdown

Path filtering allows you to control which files or URLs are indexed by defining include and exclude patterns. Use this to limit indexing to specific content or to skip files you do not want searchable.

Path filtering works with both website and R2 data sources.

Configuration

You can configure path filters when creating or editing an AI Search instance. In the dashboard, open Path Filters and add your include or exclude rules. You can also update path filters at any time from the Settings page of your instance.

When using the REST API, specify include_items and exclude_items in the source_params of your configuration:

Parameter Type Limit Description
include_items string[] Maximum 10 patterns Only index items matching at least one of these patterns
exclude_items string[] Maximum 10 patterns Skip items matching any of these patterns

Both parameters are optional. If neither is specified, all items from the data source are indexed.

Filtering behavior

Wildcard rules

Exclude rules take precedence over include rules. Filtering is applied in this order:

  1. Exclude check: If the item matches any exclude pattern, it is skipped.
  2. Include check: If include patterns are defined and the item does not match any of them, it is skipped.
  3. Index: The item proceeds to indexing.
Scenario Behavior
No rules defined All items are indexed
Only exclude_items defined All items except those matching exclude patterns are indexed
Only include_items defined Only items matching at least one include pattern are indexed
Both defined Exclude patterns are checked first, then remaining items must match an include pattern

Pattern syntax

Patterns use a case-sensitive wildcard syntax based on micromatch ↗:

Wildcard Meaning
* Matches any characters except path separators (/)
** Matches any characters including path separators (/)

Patterns can contain:

  • Letters, numbers, and underscores (a-z, A-Z, 0-9, _)
  • Hyphens (-) and dots (.)
  • Path separators (/)
  • URL characters (?, :, =, &, %)
  • Wildcards (*, **)

Indexing job status

Items skipped by filtering rules are recorded in job logs with the reason:

  • Exclude match: Skipped by rule: {pattern}
  • No include match: Skipped by Include Rules

You can view these in the Jobs tab of your AI Search instance to verify your filters are working as expected.

Important notes

  • Case sensitivity: Pattern matching is case-sensitive. /Blog/* does not match /blog/post.html.
  • Full path matching: Patterns match the entire path or URL. Use ** at the beginning for partial matching. For example, docs/* matches docs/file.pdf but not site/docs/file.pdf, while **/docs/* matches both.
  • Single * does not cross directories: Use ** to match across path separators. For example, docs/* matches docs/file.pdf but not docs/sub/file.pdf, while docs/** matches both.
  • Trailing slashes matter: URLs are matched as-is without normalization. /blog/ does not match /blog.

Examples

R2 data source

Use case Pattern Indexed Skipped
Index only PDFs in docs Include: /docs/**/*.pdf /docs/guide.pdf, /docs/api/ref.pdf /docs/guide.md, /images/logo.png
Exclude temp and backup files Exclude: **/*.tmp, **/*.bak /docs/guide.md /data/cache.tmp, /old.bak
Exclude temp and backup folders Exclude: /temp/**, /backup/** /docs/guide.md /temp/file.txt, /backup/data.json
Index docs but exclude drafts Include: /docs/**, Exclude: /docs/drafts/** /docs/guide.md /docs/drafts/wip.md
Scope an instance to one tenant Include: /customers/acme/** /customers/acme/report.pdf /customers/globex/report.pdf

To give each tenant an isolated instance backed by a single shared bucket, refer to Multitenancy.

Website data source

Use case Pattern Indexed Skipped
Index only blog pages Include: **/blog/** example.com/blog/post, example.com/en/blog/article example.com/about
Exclude admin pages Exclude: **/admin/** example.com/blog/post example.com/admin/settings
Exclude login pages Exclude: **/login* example.com/blog/post example.com/login, example.com/auth/login-form
Index docs but exclude drafts Include: **/docs/**, Exclude: **/docs/drafts/** example.com/docs/guide example.com/docs/drafts/wip

API format

When using the API, specify patterns in source_params:

{
	"source_params": {
		"include_items": ["<PATTERN_1>", "<PATTERN_2>"],
		"exclude_items": ["<PATTERN_1>", "<PATTERN_2>"]
	}
}