Skip to main content

Prerequisites

  • A LiteLLM proxy that already routes model traffic.
  • A Silmaril guardrail endpoint and API key from deployment access.
  • If you self-host, use the guardrail endpoint from the same deployment as your classify endpoint.
Set SILMARIL_GUARDRAIL_URL and SILMARIL_API_KEY in the proxy environment.

Configure the guardrail

Start in warn mode while validating proxy coverage. This configuration omits review to inherit your deployment’s default review setting.
This configuration applies the guardrail by default. An enforcement configuration is not included here. Agree on one with Silmaril before enabling blocking. additional_provider_specific_params.on_error: warn controls how Silmaril represents a valid would-block decision. It does not control LiteLLM’s handling of guardrail endpoint failures. Keep LiteLLM’s fail_on_error default; setting fail_on_error: false broadly bypasses guardrail errors.

Choose where the guardrail runs

default_on: true attaches this guardrail globally. To scope it instead, set default_on: false and attach its guardrail_name in the model configuration or in one request’s guardrails list. Do not define duplicate guardrails to cover the same stage. Post-call attachment and same-content behavior are version-bound; use the qualification table below. All selected lanes recheck response content changed by a native hook. The repository qualification contract contains reproducible commands for three separately hash-locked published CLI profiles on Python 3.12: These are exact qualification profiles, not a blanket package recommendation. The patched LTS and stable permission-response profiles require the complete hash-locked dependency graph, including explicit prisma==0.15.0. Bare litellm[proxy] installs of either version are not qualified for this behavior: without Prisma they return a generic HTTP 500 while formatting the authorization error. This qualification prerequisite does not prescribe database setup or broader production dependency requirements. With the complete profiles, both patched lanes reject low-privilege Azure Entra overrides for tenant_id, client_id, client_secret, azure_username, azure_password, and azure_scope as bounded HTTP 401 JSON authorization errors before the provider or Silmaril classifier runs. The response contains neither the supplied value nor the qualification environment path. For a Gemini-native function_declarations tool without type, 1.108.9 fails closed before provider execution; that is not tool support. 1.109.1 accepts and preserves the exact type-less shape in the Generic Guardrail API request. Verify the installed proxy version and complete profile against the contract. Read the evidence as four distinct states: source is repository behavior; published is one exact PyPI package; tested is a completed version-selected lane; deployed is not established by qualification. This guide does not claim which version is currently installed or deployed.

Expected result

LiteLLM calls Silmaril before and after the model call. In this starter configuration, a valid would-block decision is advisory and the proxy call continues. unreachable_fallback: fail_open is narrower than fail_on_error: false. It allows unreachable transport failures and HTTP 502, 503, and 504 responses to continue. Malformed or non-object JSON and HTTP 500 remain guardrail errors under LiteLLM’s default fail_on_error: true. A valid Silmaril BLOCKED response is still a block when native enforcement is explicitly configured.

Qualified endpoint and modality scope

The matrix below is bounded to the exact, separately hash-locked historical 1.108.3, patched LTS 1.108.9, and patched stable 1.109.1 packages on Python 3.12. Except for the version boundaries above and in the table, the endpoint checks remain shared across all three lanes. Qualification exercised the published CLI and deterministic provider and Firewall services over loopback only. It used no model, cloud service, paid inference endpoint, or customer account. This establishes tested compatibility, not deployment or customer activation. Pass-through endpoints are not qualified by this matrix. Their visible fields depend on endpoint/provider-specific JSONPath configuration, and the selected LiteLLM packages check extracted pass-through text without applying response rewrites. Responses WebSocket follow-up content enforcement is version-bound. On stable 1.109.1, two response.create frames on the same authenticated connection receive pre-call checks: the clean frame reaches the provider, while the malicious follow-up is blocked before provider execution and appears in neither provider nor client output. The authenticated user ID and key hash are preserved. Per-frame call and trace IDs are null because the published upstream logging object is not propagated, so full sequence/request correlation remains unqualified. LTS 1.108.9 runs no per-frame guardrail; unsafe content reaches the provider and client, and this route is unsupported. Historical 1.108.3 remains outside the retained receipt. Complete WebSocket protection, post-call token blocking, and deployment are not qualified. The endpoint rows are exercised directly rather than inferred from chat coverage. Responses assertions preserve instructions in both the exact guardrail payload and provider request. Messages assertions preserve top-level system, tool_result text, and client input tool_use through the actual selected package’s OpenAI Responses bridge. The provider returns the current Responses function_call fields (id, call_id, name, arguments, and status); qualification checks the exact guardrail tool-call payload and native Responses or Anthropic tool_use delivery for clean calls, and verifies malicious arguments are not delivered. For buffered blocked streams, provider auditing records every emitted text fragment and final source consumption. Endpoint-native SSE assertions prove that no provider text fragment, including the safe prefix emitted before the malicious fragment, reaches the client. Blocked paths also reject Responses response.output_text.delta and Messages content_block_delta delivery. The qualification also exercises LiteLLM’s endpoint-specific structured_messages write-back for Responses and Messages. That case uses a clearly marked synthetic GUARDRAIL_INTERVENED response only at the upstream compatibility boundary. Production Silmaril responses remain NONE or BLOCKED; this does not claim a Silmaril rewrite feature.

Native blocking and streaming

Native blocking is a separate, approved profile. Only the exact provider mode mode: enforce selects it; do not add it to the starter configuration without completing the agreed qualification. In that profile, a blocked pre-call can stop provider execution, and a blocked non-streaming post-call can withhold the response. Streaming requires an additional delivery decision. Tokens already delivered before a post-call verdict cannot be retracted, and streaming_end_of_stream_only: true does not withhold them. If blocking must prevent output exposure, qualify the complete proxy profile with streaming_buffer_until_moderated: true and keep output buffered until the verdict.

Optional review selection

Keep review omitted to inherit your deployment’s default as shown above. Select standard or deep with additional_provider_specific_params.review.
The provider-specific mode: shadow explicitly selects Shadow response mode and takes precedence over the on_error: warn response-mode fallback. Review selection and response mode are independent. standard selects standard review. deep requests a second opinion on eligible potentially harmful events and requires a deployment with deep review available. Otherwise Silmaril returns HTTP 400 before classification. An explicit deep selection overrides a standard-review default. Explicit null, wrong types, whitespace or case variants, and other strings are invalid. See review depth for availability and response handling. A successful explicit selection is acknowledged by the same exact string in top-level silmaril_review. This confirms the applied setting. An event may complete without additional review. Omitted review needs no acknowledgement and preserves the legacy response shape.
Ask Silmaril to disable deep-review availability when callers must be restricted to standard review.

References

Next steps

Verify your integration, then review modes and enforcement.