> ## Documentation Index
> Fetch the complete documentation index at: https://silmaril.security/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# LiteLLM guardrails

> Add Silmaril as a LiteLLM pre-call and post-call guardrail and validate coverage in warn mode.

## Prerequisites

* A LiteLLM proxy that already routes model traffic.
* A Silmaril guardrail endpoint and API key from [deployment access](https://silmaril.security/docs/start/access.md).
* If you [self-host](https://silmaril.security/docs/deployment/self-hosting.md), use the guardrail endpoint from the same deployment as your classify endpoint.

Set `SILMARIL_GUARDRAIL_URL` and `SILMARIL_API_KEY` in the proxy environment.

## Configure the guardrail

Start in warn mode while validating proxy coverage. This configuration omits `review` to inherit your deployment's default review setting.

```yaml theme={"theme":"github-light-default"}
guardrails:
  - guardrail_name: silmaril-firewall
    litellm_params:
      guardrail: generic_guardrail_api
      mode: [pre_call, post_call]
      api_base: os.environ/SILMARIL_GUARDRAIL_URL
      headers:
        x-api-key: os.environ/SILMARIL_API_KEY
      default_on: true
      unreachable_fallback: fail_open
      additional_provider_specific_params:
        on_error: warn
```

This configuration applies the guardrail by default. An enforcement configuration is not included here. Agree on one with Silmaril before enabling blocking.

`additional_provider_specific_params.on_error: warn` controls how Silmaril
represents a valid would-block decision. It does not control LiteLLM's handling
of guardrail endpoint failures. Keep LiteLLM's `fail_on_error` default; setting
`fail_on_error: false` broadly bypasses guardrail errors.

## Choose where the guardrail runs

`default_on: true` attaches this guardrail globally. To scope it instead, set
`default_on: false` and attach its `guardrail_name` in the model configuration
or in one request's `guardrails` list. Do not define duplicate guardrails to
cover the same stage. Post-call attachment and same-content behavior are
version-bound; use the qualification table below. All selected lanes recheck
response content changed by a native hook.

The [repository qualification contract](https://github.com/Silmaril-Security/SilmarilFirewall/blob/main/contracts/litellm-guardrail/v1/README.md#published-proxy-qualification)
contains reproducible commands for three separately hash-locked published CLI
profiles on Python 3.12:

| Exact profile | Qualification use | Version-bound result |
| - | - | - |
| `litellm[proxy]==1.108.3` | Historical reproduction only | Same content is checked once; changed content is rechecked. Streamed text completions are unsupported. |
| `litellm[proxy]==1.108.9` with `prisma==0.15.0` | Selected patched LTS profile | Same content is checked once; changed content is rechecked. Streamed text completions are unsupported. |
| `litellm[proxy]==1.109.1` with `prisma==0.15.0` | Selected patched stable profile | Model-attached equal content is classified twice; changed content is rechecked. Buffered streamed text completions are checked without leaking provider fragments. |

These are exact qualification profiles, not a blanket package recommendation.
The patched LTS and stable permission-response profiles require the complete
hash-locked dependency graph, including explicit `prisma==0.15.0`. Bare
`litellm[proxy]` installs of either version are not qualified for this behavior:
without Prisma they return a generic HTTP `500` while formatting the
authorization error. This qualification prerequisite does not prescribe
database setup or broader production dependency requirements.

With the complete profiles, both patched lanes reject low-privilege Azure
Entra overrides for `tenant_id`, `client_id`, `client_secret`,
`azure_username`, `azure_password`, and `azure_scope` as bounded HTTP `401`
JSON authorization errors before the provider or Silmaril classifier runs.
The response contains neither the supplied value nor the qualification
environment path.

For a Gemini-native `function_declarations` tool without `type`, `1.108.9`
fails closed before provider execution; that is not tool support. `1.109.1`
accepts and preserves the exact type-less shape in the Generic Guardrail API
request.

Verify the installed proxy version and complete profile against the contract.
Read the evidence as four distinct states: source is repository behavior;
published is one exact PyPI package; tested is a completed version-selected
lane; deployed is not established by qualification. This guide does not claim
which version is currently installed or deployed.

## Expected result

LiteLLM calls Silmaril before and after the model call. In this starter
configuration, a valid would-block decision is advisory and the proxy call
continues.

`unreachable_fallback: fail_open` is narrower than `fail_on_error: false`. It
allows unreachable transport failures and HTTP `502`, `503`, and `504`
responses to continue. Malformed or non-object JSON and HTTP `500` remain
guardrail errors under LiteLLM's default `fail_on_error: true`. A valid
Silmaril `BLOCKED` response is still a block when native enforcement is
explicitly configured.

## Qualified endpoint and modality scope

The matrix below is bounded to the exact, separately hash-locked
historical `1.108.3`, patched LTS `1.108.9`, and patched stable `1.109.1`
packages on Python 3.12. Except for the version boundaries above and in the
table, the endpoint checks remain shared across all three lanes. Qualification
exercised the published CLI and deterministic provider and Firewall services
over loopback only. It used no model, cloud service, paid inference endpoint,
or customer account. This establishes tested compatibility, not deployment or
customer activation.

| Client endpoint | Qualified guardrail-visible fields | Qualified output | Explicit limit |
| - | - | - | - |
| `/v1/chat/completions` | Messages, function definitions and tool calls | Assistant text and tool calls | Existing OpenAI chat contract only |
| `/v1/responses` | Instructions/input text, structured messages, function definitions and applicable functionless built-in tool configuration | Output text and official `function_call` items; clean and blocked calls are checked through the Generic Guardrail API; buffered streaming block | No image pixels or other media bytes are inspected |
| `/v1/messages` | System/message/tool-result text, structured messages, regular function definitions and input `tool_use` calls | Anthropic text and `tool_use` client shapes; clean and blocked output calls are checked through the Generic Guardrail API; buffered streaming withholds blocked output | The qualified OpenAI-backed proxy configuration bridges this route through `/v1/responses`; a buffered block is an Anthropic-shaped HTTP error. Anthropic native web-search tools are neither exposed in the Generic Guardrail API `tools` array nor retained in that bridge's provider `tools` |
| `/v1/completions` | Prompt text | Completion text; buffered post-call streaming only on `1.109.1` | `1.108.3` and `1.108.9` do not invoke the post-call guardrail for streams and deliver the provider stream; unbuffered content cannot be retracted |
| `/v1/embeddings` | String input | Vector response shape only | Token-ID input and vectors are not inspected as text |
| `/v1/images/generations` | Prompt text | Image response shape only | Generated pixels are not inspected |
| `/v1/audio/speech` | Input text | Binary response delivery only | Generated waveform bytes are not inspected |
| `/v1/audio/transcriptions` | None before the provider | Provider transcript response shape only | The selected published packages do not invoke the post-call guardrail hook on this route, so transcript enforcement is not qualified. Input audio bytes and waveform content are not inspected |
| `/v1/rerank` | Query and instruction text | Relevance score/index shape only | Corpus documents and numeric scores are not inspected |

Pass-through endpoints are not qualified by this matrix. Their visible fields
depend on endpoint/provider-specific JSONPath configuration, and the selected
LiteLLM packages check extracted pass-through text without applying response
rewrites.

Responses WebSocket follow-up content enforcement is version-bound. On stable
`1.109.1`, two `response.create` frames on the same authenticated connection
receive pre-call checks: the clean frame reaches the provider, while the
malicious follow-up is blocked before provider execution and appears in
neither provider nor client output. The authenticated user ID and key hash are
preserved. Per-frame call and trace IDs are null because the published
upstream logging object is not propagated, so full sequence/request
correlation remains unqualified. LTS `1.108.9` runs no per-frame guardrail;
unsafe content reaches the provider and client, and this route is unsupported.
Historical `1.108.3` remains outside the retained receipt. Complete WebSocket
protection, post-call token blocking, and deployment are not qualified.

The endpoint rows are exercised directly rather than inferred from chat
coverage. Responses assertions preserve `instructions` in both the exact
guardrail payload and provider request. Messages assertions preserve top-level
`system`, `tool_result` text, and client input `tool_use` through the actual
selected package's OpenAI Responses bridge. The provider returns the current
Responses `function_call` fields (`id`, `call_id`, `name`, `arguments`, and `status`);
qualification checks the exact guardrail tool-call payload and native Responses
or Anthropic `tool_use` delivery for clean calls, and verifies malicious
arguments are not delivered.

For buffered blocked streams, provider auditing records every emitted text
fragment and final source consumption. Endpoint-native SSE assertions prove
that no provider text fragment, including the safe prefix emitted before the
malicious fragment, reaches the client. Blocked paths also reject Responses
`response.output_text.delta` and Messages `content_block_delta` delivery.

The qualification also exercises LiteLLM's endpoint-specific
`structured_messages` write-back for Responses and Messages. That case uses a
clearly marked synthetic `GUARDRAIL_INTERVENED` response only at the upstream
compatibility boundary. Production Silmaril responses remain `NONE` or
`BLOCKED`; this does not claim a Silmaril rewrite feature.

## Native blocking and streaming

Native blocking is a separate, approved profile. Only the exact provider mode
`mode: enforce` selects it; do not add it to the starter configuration without
completing the agreed qualification. In that profile, a blocked pre-call can
stop provider execution, and a blocked non-streaming post-call can withhold
the response.

Streaming requires an additional delivery decision. Tokens already delivered
before a post-call verdict cannot be retracted, and
`streaming_end_of_stream_only: true` does not withhold them. If blocking must
prevent output exposure, qualify the complete proxy profile with
`streaming_buffer_until_moderated: true` and keep output buffered until the
verdict.

## Optional review selection

Keep `review` omitted to inherit your deployment's default as shown above. Select `standard` or `deep` with `additional_provider_specific_params.review`.

```yaml theme={"theme":"github-light-default"}
additional_provider_specific_params:
  mode: shadow
  review: deep
  on_error: warn
```

The provider-specific `mode: shadow` explicitly selects Shadow response mode and takes precedence over the `on_error: warn` response-mode fallback. Review selection and response mode are independent.

`standard` selects standard review. `deep` requests a second opinion on eligible potentially harmful events and requires a deployment with deep review available. Otherwise Silmaril returns HTTP `400` before classification. An explicit `deep` selection overrides a standard-review default. Explicit `null`, wrong types, whitespace or case variants, and other strings are invalid.

See [review depth](https://silmaril.security/docs/guides/review-depth.md) for availability and response handling.

A successful explicit selection is acknowledged by the same exact string in top-level `silmaril_review`. This confirms the applied setting. An event may complete without additional review. Omitted review needs no acknowledgement and preserves the legacy response shape.

```json theme={"theme":"github-light-default"}
{
  "action": "NONE",
  "silmaril_mode": "shadow",
  "silmaril_review": "deep"
}
```

Ask Silmaril to disable deep-review availability when callers must be restricted to standard review.

## References

* [LiteLLM AI Gateway](https://docs.litellm.ai/docs/proxy/quick_start)
* [LiteLLM Guardrails Quick Start](https://docs.litellm.ai/docs/proxy/guardrails/quick_start)
* [LiteLLM Generic Guardrail API](https://docs.litellm.ai/docs/adding_provider/generic_guardrail_api)

## Next steps

[Verify your integration](https://silmaril.security/docs/start/verify.md), then review [modes and enforcement](https://silmaril.security/docs/guides/enforcement.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.