review-learning-content
willbooster/exercode-problem-utils · skills.sh
Open source Repository Open in the app JSON README (API)
About
Skill publicada por willbooster/exercode-problem-utils no skills.sh. Instale com: npx skills add willbooster/exercode-problem-utils@review-learning-content
Details
- Kind
- Agent skills
- Publisher
- willbooster
- Origin
- skillssh
- Category
- ferramentas
- Open pull requests
- 1
- Last push
- 2026-10-05T12:02:10Z
- Repository state
- ativo
- Language
- TypeScript
- License
- Apache-2.0
- Added
- 2026-10-07 05:26:01
- Updated
- 2026-10-07 05:26:01
- Origin id
willbooster/exercode-problem-utils/review-learning-content
README
# @exercode/problem-utils
[](https://github.com/WillBooster/exercode-problem-utils/actions/workflows/test.yml)
[](https://github.com/semantic-release/semantic-release)
[](https://github.com/WillBooster/shared/tree/main/packages/wbfy)
:100: A set of utilities for judging programs on Exercode (https://exercode.willbooster.com/).
## Packages
| Package | Purpose |
| --------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `@exercode/problem-utils` | CLI, validation, result types, and stdio/command/GUI/evaluation presets; no browser or AI SDK dependency |
| `@exercode/problem-utils-browser` | Puppeteer Chromium lifecycle, browser judging, and screenshots |
| `@exercode/problem-utils-llm` | LLM judging and AI SDK providers |
Install only the packages the problem uses. Import `llmJudgePreset` from
`@exercode/problem-utils-llm`. Browser judges receive Puppeteer's native `Page`:
```ts
import { DecisionCode } from '@exercode/problem-utils';
import { browserJudgePreset } from '@exercode/problem-utils-browser';
await browserJudgePreset({
testCases: [
[
'heading',
async (page) => ({
decisionCode:
(await page.$$eval('h1', (elements) => elements.length)) === 1
? DecisionCode.ACCEPTED
: DecisionCode.WRONG_ANSWER,
}),
],
],
});
```
The browser preset serves the submitted directory, runs checks in order on one page,
prints each result, stops on the first non-accepted result, and closes the browser even
when a check throws. Checks return learner-facing verdicts; uncaught harness errors
propagate to the caller. Use `timeoutMs`, `viewport`, `contextOptions`, and `launchOptions` to set
problem-specific requirements. `screenshotOnFailure` attaches a full-page image to a
non-accepted result; capture failures are recorded in `stderr` without replacing the verdict.
`launchBrowser`, `createBrowserPage`, and `captureScreenshot` are also exported for harnesses
that manage their own HTTP server or test loop. Use `await createBrowserPage(browser)`
(or pass a browser context) to create a native Puppeteer page with the same dialog handling
as the presets. These pages dismiss dialogs by default. Registering a `dialog` listener takes
over handling: the listener must call `dialog.accept()` or `dialog.dismiss()`, even when it
only inspects the message. Asynchronous handlers retain control until they handle the dialog.
The browser package depends on `puppeteer`. Before running browser judges or PDF export,
install its version-matched Chrome headless shell explicitly when package installation scripts are disabled:
```sh
bun run exercode-browser browsers install chrome-headless-shell
```
`exercode-browser` runs this package's Puppeteer CLI, independently of an application's
E2E test version. Docker/CI should install `unzip` before this command and Chrome's OS
libraries at image build/setup time. Install the fonts required by the course content in that environment.
## HTML comparison
`htmlJudgePreset({ solutionDirectoryPath, requiredFiles?, compareDom?, textNormalizationPattern? })` compares the submitted
page with the specified model-answer directory. It reports `snapshot_body` first,
then `screenshot`, stopping on the first difference. Set `compareDom: false` for exercises that grade only the rendered appearance. The DOM comparison ignores
comments and normalizes text whitespace and attribute order. `textNormalizationPattern` overrides
the regular-expression source used to replace text-node matches with spaces before trimming;
it defaults to `"\\s+"`. Set it when a course requires a different text comparison rule. Screenshots render
formatted HTML at 800×600, with CSS animations disabled and fonts loaded; a difference
includes both PNG files. HTML decoding honors a BOM or declared HTTP/meta charset, defaulting to UTF-8 when
none is declared; formatted responses explicitly use UTF-8. If either document cannot
be decoded or formatted, both are rendered raw.
Each check uses fresh, separate browser contexts for both answers. Pixel comparison requires deterministic
page content; JavaScript timers, random content, and animated images are not frozen. Missing required files are reported before starting Chromium.
For isolated judging, keep shared assets inside the problem directory.
Both directories can use the nearest ancestor's `assets` directory, through `assets/`
or directly from the served root. Local assets are merged with that shared directory;
submission files and links take precedence, followed by local assets. A directory
symlink overrides that entire directory rather than merging shared files into its target. Source files and
linked directories are left unchanged. The temporary
served directories and browser are closed after judging. The directory helpers leave
process signals to the host. Hosts must provide and remove a per-run `TMPDIR` when a
harness terminates before disposal; isolated CLI checks do this automatically for
SIGINT, SIGTERM, and SIGKILL. Long-lived hosts can await disposal in their own
shutdown handlers. `captureHtmlBodySnapshot`,
`captureHtmlScreenshotPair`, and `createHtmlServedDirectory` expose the same operations
for custom checks. The screenshot pair takes two `{ page, url }` targets and returns
PNGs in that order, formatting both documents or neither. Give those pages matching
viewport options and separate fresh browser contexts. See the [HTML example](https://github.com/willbooster/exercode-problem-utils/blob/HEAD/example/web_page_comparison/judge.ts).
## Custom browser pages and lifecycle hooks
`browserJudgePreset` accepts `directoryPath` to serve an assembled exercise directory,
`entryPath` to select its initial page, and `navigationOptions` for native Puppeteer
navigation settings. `initializePage` runs before navigation, so it can register console
and page-error listeners. `afterTests` runs after the checks, including a failing verdict,
while the page remains open. An exception in initialization, navigation, or a check
propagates to the caller and skips `afterTests`; browser cleanup still runs.
## Spring Boot courses
Import `springBootJudgePreset` from `@exercode/problem-utils/presets/springBoot` and pass
`problemDirectoryPath` plus an async `evaluate` callback. The preset builds the problem's
Maven project offline, starts its Spring Boot JAR on port 59000, and invokes the problem's
`judge.ts --evaluate`. It preserves the course build, startup, and evaluation limits
(90, 60, and 60 seconds) and emitted screenshot metadata. The host provides Java, Maven,
Bun, cached dependencies, an exclusive execution slot, and interrupted-run cleanup.
Evaluators can build their request URLs with `buildSpringBootUrl` from the same core subpath.
Browser evaluation can use `launchBrowser` and `captureTomcatScreenshots` from the browser
package; the Spring Boot preset itself adds no browser dependency to core.
## JavaScript and Tomcat courses
`javascriptJudgePreset(problemDirectoryPath, options)` from `@exercode/problem-utils-browser`
runs `main.mjs` or `main.js` in Chromium and compares console output with `test_cases/*.out`.
The `.in` files contain browser setup JavaScript. Set `initializeAndVerifyDom: true` to
call the setup's `initializeTest` and `verifyDom` hooks, and `waitForConsoleIdle: true`
for exercises whose asynchronous console output must settle before comparison.
`javascriptDomJudgePreset(problemDirectoryPath)` supports DOM exercises whose `.in`
files define `initializeTest`, `verifyDom`, or `window.test`. Setup scripts retain global
declarations for submitted code and verification hooks. The preset exposes top-level
function declarations to those callbacks, captures console output, and stops at the
first failing case. Both JavaScript presets require a host-enforced overall timeout:
the console-idle heuristics do not bound programs that keep emitting output.
`tomcatJudgePreset` and `buildTomcatUrl` are available from
`@exercode/problem-utils/presets/tomcat`, without browser dependencies. The preset
accepts `problemDirectoryPath`, `jspDirectory` (`''` or `'WEB-INF/jsp'`), optional
`forbiddenTexts`, and an asynchronous `evaluate` callback. It builds the problem's
`pom.xml` with Maven offline, serves the `judge` application on port 59000, and invokes
the same `judge.ts` with `--evaluate`. The host must provide Maven, GNU time, GNU
timeout, Bun, `CATALINA_HOME`, and exclusive access to that port. Build and evaluation
limits are 60 and 30 seconds respectively.
Browser evaluations can use native Puppeteer pages together with
`captureTomcatScreenshots` or `verifyTomcatHtml` from the browser package. The latter
compares document markup with whitespace removed and records screenshots for the
verdict. `verifyTomcatPath(page, endpoint)` waits up to five seconds for the endpoint's
path and parsed document, reporting the expected and current paths in Japanese if navigation fails;
query strings do not affect the check. All interrupted-run cleanup remains the host's
responsibility. `requirePageElement(page, selector)` immediately returns a native
Puppeteer element handle or reports the missing selector in Japanese; callers use
the handle's native actions for course-specific interactions.
## GUI programs
`guiCommandJudgePreset` from `@exercode/problem-utils/presets/guiCommand` runs a program on Xvfb,
captures every top-level window, waits `screenshotWaitSeconds` (0.3 by default) before the next
capture (so captures are that wait plus the time a capture takes apart), and stops the
program once `stopDetectionThreshold` (5 by default) consecutive captures are identical. The
problem's `test` receives the last capture of each window as PNG files in `runResult.screenshots`.
The host must provide `Xvfb`, `maim`, `xdotool`, and `xwininfo`.
Set `recordsAnimation: true` for programs a grader has to watch moving, such as animations:
```ts
import { DecisionCode } from '@exercode/problem-utils';
import { guiCommandJudgePreset } from '@exercode/problem-utils/presets/guiCommand';
await guiCommandJudgePreset(import.meta.dirname, {
recordsAnimation: true,
test: ({ runResult }) => ({
// A program that is still animating at the time limit is fine; one that shows nothing is not.
decisionCode:
runResult.stopReason === 'timeout' && runResult.screenshots.length === 0
? DecisionCode.TIME_LIMIT_EXCEEDED
: DecisionCode.ACCEPTED,
outputFiles: [...runResult.screenshots, ...runResult.recordings],
}),
});
```
- `runResult.recordings` holds an animated PNG (`<window name>_<window id>_recording.png`) of each
window that kept changing during the run, assembled from the captures above and looping forever.
Browsers play it wherever they show a PNG, so Exercode displays it like a screenshot. A window that
only appeared and was painted has no recording, only its screenshot.
- A run that is still changing at the time limit reaches `test` with `stopReason: 'timeout'` instead of
being reported as `TIME_LIMIT_EXCEEDED`, so `test` decides the verdict (e.g. a time limit exceeded
when nothing was captured). Without `recordsAnimation`, such a run never reaches `test`.
- The recordings of one run take at most `MAX_GUI_RECORDING_BYTES` (2 MiB) in total: the largest one
loses every other frame until they fit, and one that does not fit even with two frames is left out, so
decide a timed-out run by `runResult.screenshots` rather than by the presence of a recording. A lower
`screenshotWaitSeconds` records more smoothly; raise `stopDetectionThreshold` with it, and for programs
that pause longer than the two multiplied.
## PDF export
```typescript
import { markdownToPdf } from '@exercode/problem-utils-browser/pdf';
const pdf = await markdownToPdf(markdown, {
assetDirectoryPath: import.meta.dirname,
pdfOptions: { format: 'A4' },
});
await Bun.write('material.pdf', pdf);
```
PDF assets are served on IPv4 loopback, and encoded paths cannot escape the asset directory. Intentional asset symlinks remain usable. Core also exports `startLocalHttpServer` for callers that need the same local-only server; await its startup before using its address.
The PDF entry point removes YAML mapping frontmatter, renders Markdown with syntax highlighting and CJK-friendly emphasis, and resolves relative
images against `assetDirectoryPath`, and waits for fonts and images before printing. Missing or invalid images leave
browser placeholders without preventing the document from exporting.
Pass `mermaidScriptPath` pointing to a Mermaid browser bundle to render diagrams,
`css` to customize styling, and `pdfOptions` for native Puppeteer PDF settings.
The defaults use screen media, A4 paper when no custom dimensions are supplied, printed backgrounds, and margins of
30 mm top/bottom, 40 mm right, and 20 mm left. PDF rendering dependencies are loaded
through this subpath; importing the browser judging entry point does not load them.
## CLI
The package ships an `exercode-problem` command for problem authors (run it with `bun x` in a repository that depends on `@exercode/problem-utils`):
```bash
# Judge all model answers of all problems (directories containing problem.md or <id>.problem.md) under a directory.
# model_answers/* must be fully accepted; model_answers.fails/* must fail at least one test case.
bun x exercode-problem # everything under the current directory
bun x exercode-problem courses/foo # everything under courses/foo
bun x exercode-problem --only a_plus --skip gui_ --concurrency 2
# Judge one answer directory of the problem in the current directory.
bun x exercode-problem judge model_answers/python
bun x exercode-problem judge model_answers/python '{ "language": "python" }'
# Debug one answer directory of the problem in the current directory.
bun x exercode-problem debug model_answers/python '{ "stdin": "1 2" }'
```
`judge` and `debug` run a custom `judge.ts` / `debug.ts` when the problem has one, and apply `stdioJudgePreset` / `stdioDebugPreset` otherwise, mirroring the Exercode server. The debug fallback applies only to standard problems: a problem with a custom `judge.ts` needs its own `debug.ts`, and `exercode-problem debug` fails with a message otherwise (the server likewise reports debug as unsupported there).
The all-problem check judges serially by default because time limits are measured in wall-clock time; pass `--concurrency <n>` to parallelize when the checked problems are not timing-sensitive.
A standard stdin/stdout problem must NOT commit a `judge.ts` or `debug.ts` that is identical to the default stdio harness: the absence of `judge.ts` marks the problem as standard, and committed copies would drift from the server's defaults. The CLI rejects such files; a file kept intentionally (e.g. to demonstrate the default harness) can add an explanatory comment to be treated as custom.
## Validators
The CLI also validates learning-material files without running any program, mirroring the checks the Exercode importer applies:
```bash
# Validate problem directories (problem.md frontmatter, test_cases, model_answers, templates, judge.ts / debug.ts).
bun x exercode-problem validate-problem <problemDir>...
# Validate a course directory (course.yaml, lecture materials with embedded questions, problem references).
bun x exercode-problem validate-course <courseDir> [--problems-dir <dir>]
# Validate a contest (*.contest.yaml) file.
bun x exercode-problem validate-contest <contestYamlPath> [--problems-dir <dir>]
```
Each target prints `OK` or `NG` followed by its errors and warnings; the command exits 1 when any target has an error. `--problems-dir` points to the directory holding the referenced problems; a course is always searched at any depth, since Exercode links a material only to the problems inside its course, so for a course the option must name a directory inside it. The validators are also exported (`validateProblemDirectory`, `validateCourseDirectory`, `validateContestFile`, `validateMaterialFile`).
## Agent skills
The [`skills/`](https://github.com/willbooster/exercode-problem-utils/blob/HEAD/skills/) directory holds skills for AI coding agents that author and review Exercode learning content: `generate-learning-content` (entry point), `generate-course-materials`, `generate-judge-problems`, `generate-judge-contest`, `review-learning-content`, and `setup-exercode-course-repository`. Agents working in this repository load them through the symlinks under `.claude/skills/`; install them elsewhere with the [skills](https://github.com/vercel-labs/skills) CLI:
```bash
bun x skills add WillBooster/exercode-problem-utils --agent claude-code --agent codex
```
## Test cases
A problem keeps its test cases under `test_cases/`. A test case id is the shared name of the following entries, and each entry is optional:
| Entry | Meaning |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `<id>.in` | Standard input. Omit it (or leave it empty) when the program reads nothing. |
| `<id>.out` | Expected standard output. |
| `<id>.fin/` | Files copied into the working directory before the run (input files). |
| `<id>.fout/` | Expected output files, compared with the files of the same relative paths in the working directory. |
| `_shared.fin/` | Files copied into the working directory before every test case. |
| `<id>.json` | Configuration for a custom `judge.ts` that reads it itself; the presets ignore it (the Judge server lists it as a test case of the custom judge). |
A test case whose id contains `example` (the judge server's rule, e.g. `example_1` or `01_example_small`) is an example shown to learners; every other case is hidden.
Standard output and text files are compared as space-separated tokens: consecutive white spaces count as one separator, and an expected token that contains a decimal point and parses as a finite number (e.g. `3.14`, but not `1`, `1e-3` or `1.0e309`) accepts a value within an absolute or relative error of `1e-6`. A file is text when it is valid UTF-8 without NUL bytes; other files (e.g. images) must match byte for byte. A received file larger than 8 MiB counts as not produced, an expected file larger than 8 MiB is an authoring error (the case is reported as a runtime error), and a file larger than 1 MiB is left out of the reported pair. When a file differs, the result carries `<name>_expected.<ext>` and `<name>_received.<ext>` so Exercode can show both (Exercode decides per test case whether a learner may see them, as it does for expected stdout).
How a missing expectation is treated depends on the harness:
- `stdioJudgePreset` (the default for problems without `judge.ts`) requires `<id>.out` or a non-empty `<id>.fout/` for every test case, so a standard problem cannot accept a run without checking it. A problem whose `problem.md` declares `requiredOutputFilePaths` or `isManualScoringRequired` is exempt, because every test case is judged by those instead; code rules (`requiredRegExpsInCode` etc.) and `requiredSubmissionFilePaths` check the submission once, in addition to the output comparison, and do not exempt.
- `commandJudgePreset` without a `test` option checks whatever expectations exist, and a test case with neither only has to run within the limits. A `test` option replaces that comparison; it receives `testCase.output`, `testCase.fileOutputPath` and `cwd` and can call the exported `judgeAgainstExpectations` (or `compareStdoutAsSpaceSeparatedTokens` and `compareExpectedOutputFiles`). A custom `readTestCases` may return any test case type with `id` (plus optional `input` and `fileInputPath`); the default verdict judges any case that exposes a string `output` or `fileOutputPath`, which the default reader's `CommandTestCase` does.
- `guiCommandJudgePreset` passes the expectations to the problem's `test`, which decides everything.
- `llmJudgePreset` runs no program, so it copies no `.fin/`; it hands `<id>.in` as the prompt input and the whole entry (including `fileOutputPath`) to the problem's `test`.
- `stdioDebugPreset` by default copies the answer directory to a temporary directory and builds and runs it there together with `_shared.fin/` and the first example case's `.fin/` (the case's files win; hidden cases' inputs are never handed to learner code), so the answer directory is left untouched (its `node_modules`, and those of its ancestors, are linked into the copy rather than duplicated); files the program writes are reported only through `requiredOutputFilePaths`. Callers that already own a disposable answer directory can pass `{ disposableWorkingDirectory: true }` as the second argument to build and run directly there; they must remove it themselves, including when the harness is terminated.
- `evaluationJudgePreset` does not use `test_cases/`.
`readTestCases` is exported for harnesses that enumerate `test_cases/` themselves.
## Measurements
`stdioJudgePreset`, `stdioDebugPreset`, and `commandJudgePreset` (with its default runner; a custom `runCommand` decides whether its results carry `cpuTimeSeconds`) run a program under GNU time (`/usr/bin/time` on Linux, `gtime` on macOS) and report its wall time (`timeSeconds`), user plus system CPU time (`cpuTimeSeconds`), and peak resident set size (`memoryBytes`, at least the footprint of the GNU `timeout` wrapper the program runs under, about 1 MiB) in every test case result; `guiCommandJudgePreset` reports the wall time and the peak resident set size, and `llmJudgePreset` the wall time only. Time limits are judged by wall time. The CPU time is recorded even for a run that exceeded its limit, so a judge server sharing CPUs between programs can tell a program that used up its limit (its CPU time, summed over its threads, reaches the limit, so it would exceed it on any single CPU) from one that may only have waited for a CPU (its CPU time stays below the limit) and re-run just the latter alone.
`browserJudgePreset` forwards measurements returned by each check; it does not measure elapsed time, CPU time, or memory automatically. Its `timeoutMs` sets the page default timeout for navigation, waiting and shared screenshot capture. Other native operations follow Puppeteer’s own timeout behavior; this option does not bound the complete check. Checks that require their own measurements or time-limit verdicts must provide them explicitly. The hosting judge can independently limit the overall harness process.