BPlatform specification
Login
featureStandard

Pest parser generator

Compiler mod and host codegen that turns Pest .pest grammars into Beskid combinator parsers.

Document layout: Feature capability layoutconforms
  • ## Purpose
  • ## Requirements

Pest parser generator Specification

Purpose

Compiler mod and host codegen that turns Pest .pest grammars into Beskid combinator parsers.

Requirements

Requirement: Embedded Pest grammars carve-out: Decision [D-COMP-MODS-0020]

The Beskid standard SHALL enforce the following migrated contract section. Accepted ADR decisions are binding; uppercase requirement keywords retain their BCP-14 meaning.

Project-local .pest files compiled by GrammarGenerator parse text DSLs only. They must not replace or extend beskid.pest. Conformance fixtures must prove host surface unchanged.

Stable ID: BSP-REQ-DB38B6D44A0B
Legacy source: site/spec-content/platform-spec/compiler/compiler-mods/pest-parser-generator/adr/0001-embedded-pest-carve-out/content.md
Source SHA-256: 1d71ab5bf67f59370d5deafddfcb4a43afefd59a4701844c8e519576c03246b8

Scenario: Conformance exercises Decision

  • GIVEN an implementation claims conformance with this capability
  • WHEN behavior governed by this contract section is exercised
  • THEN every MUST, SHALL, REQUIRED, prohibition, and accepted decision in the section is satisfied

Informative Source Provenance

The records below preserve migration history and are not normative except where text was extracted into a requirement above.

Source Record: Pest parser generator

Authority: informative provenance
Legacy path: /platform-spec/compiler/compiler-mods/pest-parser-generator/
Source: site/spec-content/platform-spec/compiler/compiler-mods/pest-parser-generator/content.md
SHA-256: e32070e1c9a08da5ac428d522b26865df6c09e161fd66cf9b10d1f94d23e1cd4

Migrated source text
<SpecSection title="Contract statement" id="contract-statement">
Projects with `type = Mod` **may** register a **`GrammarGenerator`** contract. At `beskid mod rebuild`, the host schedules **`corelib_pest_gen`** (or another grammar mod), which uses **`Core.Text.Pest`** to read `grammars/**/*.pest` and emit Beskid combinator parsers via [Core.Text.Parser](/platform-spec/core-library/text-and-parsing/text-parser-combinators/) (**PARSER-005** naming). Legacy Rust **`beskid_pest_gen`** remains a temporary golden bridge until Beskid emit is AOT-ready. Embedded grammars **must not** modify host [`beskid.pest`](/platform-spec/compiler/front-end/parser-and-ast-contracts/) (see ADR carve-out).
</SpecSection>

<SpecSection title="Implementation anchors" id="implementation-anchors">
- `compiler/corelib/mods/corelib_pest_gen/`
- `compiler/corelib/packages/foundation/src/Core/Text/Pest/`
- `compiler/crates/beskid_pest_gen/` (temporary bridge; retire per design model)
- `compiler/crates/beskid_tests/fixtures/mods/pest_gen_mod/`
</SpecSection>

## Decisions
<!-- spec:generate:adr-index -->
No open decisions. Closed choices are normative ADRs under **`adr/`** (`D-COMP-MODS-0020`); use the reader **ADRs** tab for expandable detail.
<!-- /spec:generate:adr-index -->
## Articles
<!-- spec:generate:article-index -->
- [Contracts and edge cases](./articles/contracts-and-edge-cases/)
- [Design model](./articles/design-model/)
- [Examples](./articles/examples/)
- [FAQ](./articles/faq-and-troubleshooting/)
- [Flow and algorithm](./articles/flow-and-algorithm/)
- [Verification](./articles/verification-and-traceability/)
<!-- /spec:generate:article-index -->

Source Record: Embedded Pest grammars carve-out

Authority: informative provenance
Legacy path: /platform-spec/compiler/compiler-mods/pest-parser-generator/adr/0001-embedded-pest-carve-out/
Source: site/spec-content/platform-spec/compiler/compiler-mods/pest-parser-generator/adr/0001-embedded-pest-carve-out/content.md
SHA-256: 1d71ab5bf67f59370d5deafddfcb4a43afefd59a4701844c8e519576c03246b8

Migrated source text
## Context

D-COMP-FRONT-0012 requires a single host `beskid.pest` surface. Corelib needs embedded DSL parsers (markup, regex).

## Decision

Project-local `.pest` files compiled by **GrammarGenerator** parse **text DSLs only**. They **must not** replace or extend `beskid.pest`. Conformance fixtures **must** prove host surface unchanged.

## Consequences

- Clear boundary: host parser vs embedded grammars.
- `beskid_pest_gen` is not an alternate Beskid front-end.

## Verification anchors

- `compiler/crates/beskid_analysis/src/beskid.pest` unchanged by mod rebuild
- `compiler/crates/beskid_tests/fixtures/mods/pest_gen_mod/`

Source Record: Contracts and edge cases

Authority: informative provenance
Legacy path: /platform-spec/compiler/compiler-mods/pest-parser-generator/articles/contracts-and-edge-cases/
Source: site/spec-content/platform-spec/compiler/compiler-mods/pest-parser-generator/articles/contracts-and-edge-cases/content.md
SHA-256: 4f1d39a28ffead3a1ae3a6f1617d2394cde4fb8fc4b5d1eec11a4c66902e4641

Migrated source text
## Normative requirements

| ID | Requirement |
| --- | --- |
| **GRAM-001** | Output **must** use `Core.Text.Parser` combinators only. |
| **GRAM-002** | Generated modules **must** land under `.beskid/obj/mods/<id>/generated/`. |
| **GRAM-003** | Host `beskid.pest` **must not** be modified by mod grammars. |
| **GRAM-004** | Unsupported Pest constructs **must** emit **E18xx** with actionable messages. |
| **GRAM-005** | Canonical grammars `regex.pest` and `console_markup.pest` **must** ship with corelib/console packages. |

Source Record: Design model

Authority: informative provenance
Legacy path: /platform-spec/compiler/compiler-mods/pest-parser-generator/articles/design-model/
Source: site/spec-content/platform-spec/compiler/compiler-mods/pest-parser-generator/articles/design-model/content.md
SHA-256: bc29cb34f0c6ace004fd86df82c544c2655ab66c39d3781e82d93d52996388b9

Migrated source text
## Components

| Component | Role |
| --- | --- |
| **`Beskid.Compiler.Collect.GrammarGenerator`** | Mod contract: registers `.pest` roots via `Collector`; emits combinator modules on fingerprint miss |
| **`Core.Text.Pest`** | Beskid library: parse Pest grammar text, lower to combinator AST, emit `Core.Text.Parser` call sites |
| **`corelib_pest_gen`** | Reference `type = Mod` package (`PestCollector`, `PestGen`); emits `CodeContribution` outputs materialized to `.generated/{modulePath}/{fileName}.g.bd` |
| **`grammar { }` block** | Consumer `.bproj` declares Pest roots and output module paths (camelCase keys; see [project manifest contract](/platform-spec/tooling/manifests-and-lockfiles/project-manifest-contract/)) |
| **`mod.descriptor.json`** | Adds `generatedModules` entries with output paths |

The host **must not** embed Pest-specific emission logic in Rust. Parse and emit authority lives in **`Core.Text.Pest`**; scheduling and disk materialization live in the **`corelib_pest_gen`** mod.

## Core.Text.Pest layout

| Module | Role |
| --- | --- |
| **`Core.Text.Pest.Grammar`** | `GrammarRule { string name, string expression }` |
| **`Core.Text.Pest.Expr`** | `PestExpr` variant tree (`Literal`, `RuleRef`, `Seq`, `Choice`, …) |
| **`Core.Text.Pest.Names`** | `RuleNameToCallable`, `RuleNameToPascal` |
| **`Core.Text.Pest.Emit`** | `EmitCombinatorModule`, `EmitRuleBody`, `EmitRuleCallable` |
| **`Core.Text.Pest`** | Hub re-export |

| Legacy Rust (`beskid_pest_gen`) | Normative Beskid |
| --- | --- |
| `parse_grammar_rules` | `ParseGrammarRules` |
| `emit_combinator_module` | `EmitCombinatorModule` |
| `Parse_{rule}` | `Parse{PascalRule}` (for example `ParsePatBranch`) |

## RuleNameToCallable mapping

Pest rule names remain **snake_case** on disk (Pest idiom). Emit time **must** map them to PascalCase Beskid callables via `RuleNameToCallable`:

| Pest rule | Callable |
| --- | --- |
| `digit` | `ParseDigit` |
| `pat_branch` | `ParsePatBranch` |
| `lower_run` | `ParseLowerRun` |

Mapping rules:

1. Split snake_case segments; capitalize each segment and concatenate (`lower_run` → `LowerRun`).
2. Prefix with configurable callable prefix (default `Parse` from `pestGenConfig.ruleCallablePrefix`).
3. Emit locals as **camelCase** (`choice0`, `seqCur`) per **PARSER-005**.

Diagnostic `rule` strings in `ParseResult::Err` **may** keep the original Pest spelling.

## Rust `beskid_pest_gen` retirement

`compiler/crates/beskid_pest_gen/` is a **temporary golden bridge** only—it is **not** naming authority. The crate **must** be removed from the workspace after:

1. `Core.Text.Pest.EmitCombinatorModule` output for canonical fixtures (for example `regex.pest`) matches current `Generated.bd` semantics.
2. Emitted symbols satisfy **PARSER-005** (no `Parse_[a-z]` callables, no `_[a-z0-9]+` locals).
3. `corelib_pest_gen` mod rebuild replaces `regen_grammar_parsers.sh` in CI.

Until removal, conformance tests **may** diff Rust bridge output against Beskid emit as a migration gate.

## Supported Pest subset

Rules, `~` sequence, `|` choice, `*`/`+`/`?` repeat, string/char literals, rule references. Modifiers `_`, `@`, `$`, `!` mapped per Pest docs. Left recursion **must** be rejected (**E18xx**).

## Implementation anchors

- Beskid emit: `compiler/corelib/packages/foundation/src/Core/Text/Pest/`
- Mod package: `compiler/corelib/mods/corelib_pest_gen/`
- Legacy bridge (retire): `compiler/crates/beskid_pest_gen/`

Source Record: Examples

Authority: informative provenance
Legacy path: /platform-spec/compiler/compiler-mods/pest-parser-generator/articles/examples/
Source: site/spec-content/platform-spec/compiler/compiler-mods/pest-parser-generator/articles/examples/content.md
SHA-256: 4d702f523ef619874145d07d794024b389c67f46f00af2bb2d05d0e370384150

Migrated source text
## Example manifest

```
mod {
  capabilities = [read_project_sources, emit_syntax]
  grammarRoots = ["grammars"]
}
```

Source Record: FAQ

Authority: informative provenance
Legacy path: /platform-spec/compiler/compiler-mods/pest-parser-generator/articles/faq-and-troubleshooting/
Source: site/spec-content/platform-spec/compiler/compiler-mods/pest-parser-generator/articles/faq-and-troubleshooting/content.md
SHA-256: 947d988b49b1611cb4e5148abf37a2199158862ddb22039b0b12fef87f913dbb

Migrated source text
## FAQ

### Why host Rust for codegen?

`pest_derive` is compile-time only; mods orchestrate, host emits.

Source Record: Flow and algorithm

Authority: informative provenance
Legacy path: /platform-spec/compiler/compiler-mods/pest-parser-generator/articles/flow-and-algorithm/
Source: site/spec-content/platform-spec/compiler/compiler-mods/pest-parser-generator/articles/flow-and-algorithm/content.md
SHA-256: b1dac7cf5f52a110a6d74630ec2759c502921d0adabb13f16798a03c17a18ffe

Migrated source text
## Flow

1. Consumer project declares `grammar { roots, grammarOutput }` in `.bproj`.
2. `beskid mod rebuild` builds the grammar mod AOT artifact (`corelib_pest_gen`).
3. On fingerprint miss, the mod's `GrammarGenerator` uses `Core.Text.Pest` to parse `.pest` sources and emit combinator `.bd` files (PascalCase callables, camelCase locals).
4. Host materializes typed output under the mod's `generatedOutput` root.
5. Consumer imports generated parsers from the declared `module` path.

Source Record: Verification

Authority: informative provenance
Legacy path: /platform-spec/compiler/compiler-mods/pest-parser-generator/articles/verification-and-traceability/
Source: site/spec-content/platform-spec/compiler/compiler-mods/pest-parser-generator/articles/verification-and-traceability/content.md
SHA-256: 90287b7e8827daf226447c8eb34e2c4ead2884075a495bf4965d63e03bbbcbd0

Migrated source text
## Anchors

- `compiler/crates/beskid_tests/fixtures/mods/pest_gen_mod/`
- `compiler/crates/beskid_pest_gen/tests/`