> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fastino.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Use https://docs.fastino.ai/llms.txt to discover and navigate pages. Use https://docs.fastino.ai/llms-full.txt when you need the complete documentation corpus. Use https://docs.fastino.ai/openapi.json as the source of truth for customer-facing routes. For GLiDE decision inference, call POST https://api.fastino.ai/v1/systemone with model fastino/GLiDE. Start managed training (config job) with POST /v1/training/jobs using base_model. For interactive training, native JSON clients use the public operations under /v1/training/jobs and /v1/training/usage. Existing Tinker clients use tinker==0.32.0 with TINKER_BASE_URL=https://api.fastino.ai/v1/training/compat/tinker and TINKER_CREDENTIAL_CMD. Compatibility child routes are private and absent from OpenAPI. Auto Train is private access and has no public route; direct interested users to sales@fastino.ai. Do not infer undocumented routes. Read API keys from FASTINO_API_KEY and never embed credentials in code, logs, or reports.

# Train with verifiable rewards

> Turn a deterministic grader into a learning signal for one interactive Fastino job.

Use a verifiable reward when code can check a response, such as a math answer,
test result, schema constraint, exact extraction, or tool outcome.

```text theme={null}
prompt -> sample a group -> grade -> center rewards -> build datums -> update
```

Sampling, forward-backward passes, optimizer steps, and checkpoints all belong
to the same interactive Fastino job. Reward execution and application state
remain in your code.

## Prerequisites

Start with a working interactive training loop from the
[Decoder Quickstart](https://docs.fastino.ai/training/decoder-quickstart.md) and confirm
that a baseline datum trains successfully. This guide reuses its `call` and
`wait` helpers, `tokenizer`, and `job_id`.

Your application samples responses, grades them, and converts rewards into
per-token advantages. Fastino applies those advantages with a supported loss:
`importance_sampling`, `ppo`, `cispo`, or `dro`. Custom loss functions do not
run inside the training job.

## 1. Define a narrow grader

Separate parsing from scoring so malformed output is observable:

> [!WARNING]
>
> Run generated code or tools only in an isolated, resource-limited
> environment. Fastino doesn't execute your reward function or manage its side
> effects.

```python theme={null}
def parse_answer(response: str) -> str | None:
    answer = response.strip()
    return answer if answer else None


def get_reward(response: str, expected: str) -> float:
    actual = parse_answer(response)
    if actual is None:
        return 0.0
    return 1.0 if actual == expected else 0.0
```

Keep a frozen fixture set that includes correct, incorrect, malformed, and
equivalently formatted answers. Log parse failures separately from valid but
incorrect responses.

## 2. Snapshot the job for sampling

Save a sampler checkpoint from the current weights, then sample a group from
it:

```python theme={null}
def sample_group(question: str, step: int, group_size: int = 4):
    name = f"rollout-{step:06d}"
    saved = call(
        "POST",
        f"/v1/training/jobs/{job_id}/save-weights-for-sampler",
        {"name": name, "ttl_seconds": 24 * 60 * 60},
    )
    wait(job_id, saved["future_id"])

    prompt_tokens = tokenizer.apply_chat_template(
        [{"role": "user", "content": question}],
        add_generation_prompt=True,
        enable_thinking=False,
        return_dict=False,
    )
    sampled = call(
        "POST",
        f"/v1/training/jobs/{job_id}/sample",
        {
            "checkpoint": name,
            "prompt_tokens": prompt_tokens,
            "num_samples": group_size,
            "sampling_params": {"max_tokens": 512, "temperature": 1.0},
        },
    )
    result = wait(job_id, sampled["future_id"])
    return prompt_tokens, result["sequences"]
```

This doesn't create another job. The checkpoint and the samples belong to the
same interactive job.

## 3. Convert rewards to advantages

```python theme={null}
def center_rewards(sequences, expected: str):
    rewards = [
        get_reward(tokenizer.decode(sequence["tokens"]), expected)
        for sequence in sequences
    ]
    mean_reward = sum(rewards) / len(rewards)
    advantages = [reward - mean_reward for reward in rewards]
    return None if all(value == 0.0 for value in advantages) else advantages
```

Track groups with no reward variance. They may indicate that the task is too
easy, too hard, or parsed incorrectly.

## 4. Build aligned datums

The prompt is context, so prompt positions receive zero advantage. Completion
positions receive the response's group-relative advantage and the sampler's log
probabilities. Every tensor has one value per input position:

```python theme={null}
def rl_datum(prompt_tokens, sequence, advantage):
    completion = sequence["tokens"]
    tokens = prompt_tokens + completion
    padding = [0.0] * (len(prompt_tokens) - 1)
    logprobs = padding + sequence["logprobs"]
    advantages = padding + [advantage] * len(completion)
    if not (len(tokens) - 1 == len(logprobs) == len(advantages)):
        raise ValueError("training datum arrays are not aligned")
    return {
        "model_input": {"chunks": [{"type": "encoded_text", "tokens": tokens[:-1]}]},
        "loss_fn_inputs": {
            "target_tokens": {"data": tokens[1:], "dtype": "int64"},
            "logprobs": {"data": logprobs, "dtype": "float32"},
            "advantages": {"data": advantages, "dtype": "float32"},
        },
    }


def build_datums(prompt_tokens, sequences, advantages):
    return [
        rl_datum(prompt_tokens, sequence, advantage)
        for sequence, advantage in zip(sequences, advantages)
    ]
```

Use the tokenizer of the job's `base_model` for both prompts and decoding.

## 5. Update the interactive job

```python theme={null}
def update_job(datums):
    backward = call(
        "POST",
        f"/v1/training/jobs/{job_id}/forward-backward",
        {"datums": datums, "loss_fn": "importance_sampling"},
    )
    train_result = wait(job_id, backward["future_id"])
    update = call(
        "POST",
        f"/v1/training/jobs/{job_id}/optim-step",
        {"adam_params": {"learning_rate": 1e-5}},
    )
    optim_result = wait(job_id, update["future_id"])
    return train_result, optim_result
```

Wait for forward-backward to succeed before submitting the optimizer step.
Wait for and record the optimizer result before advancing the application
step.

Supported `loss_fn_config` fields and defaults:

| Applies to | Fields and defaults |
| - | - |
| `ppo` | `clip_low_threshold: 0.8`, `clip_high_threshold: 1.2` |
| `cispo` | `clip_low_threshold: 0.0`, `clip_high_threshold: 4.0` |
| `dro` | `beta: 0.05` |
| Reward losses | `tis_cap: 2`, `icepop_low: 0.5`, `icepop_high: 5` |

The reward-loss safeguards in the last row are enabled by default. Unknown
configuration fields are rejected.

## 6. Save a recovery boundary

```python theme={null}
def save_recovery_boundary(step: int) -> str:
    state = call(
        "POST",
        f"/v1/training/jobs/{job_id}/save-weights",
        {"name": f"step-{step:06d}"},
    )
    return wait(job_id, state["future_id"])["path"]
```

Persist the returned `path` with:

* Job ID
* Application step and dataset cursor
* Dataset and reward implementation revisions
* Base model and training configuration
* Evaluation metrics

This metadata lets a controller choose a checkpoint without guessing from its
name.

## 7. Run the reward loop

Define the training examples in your application, then compose the helpers
inside the loop. `continue` now skips only the current group when all rewards
are equal:

```python theme={null}
training_examples = [
    ("What is 2 + 2? Reply with the number only.", "4"),
]

for step, (question, expected) in enumerate(training_examples):
    prompt_tokens, sequences = sample_group(question, step, group_size=4)
    advantages = center_rewards(sequences, expected)
    if advantages is None:
        continue

    datums = build_datums(prompt_tokens, sequences, advantages)
    train_result, optim_result = update_job(datums)
    state_path = save_recovery_boundary(step)
    # Persist state_path with the application metadata listed above.
```

## Verify before scaling

* The grader distinguishes known correct and incorrect fixtures.
* Malformed responses don't crash the application loop.
* At least one sample group has reward variance.
* Every datum's tensors have one value per input position.
* Forward-backward passes and optimizer steps complete on the same job.
* A new job can start from a state checkpoint with its client-owned resume
  metadata.
* Held-out evaluation remains separate from the training reward batch.

For the surrounding lifecycle, read
[Build an interactive training loop](https://docs.fastino.ai/training/training-loop.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.