Response validation plugin

[morgendave]

---

name: Adding response quality validation for retry
about: response quality validation for retryable error
title: "Adding response quality validation for retry"
labels: ''
assignees: ''

---

Description

For bad rollouts that makes the final output bad which has quality issues, eg, gibberish, repetitions, etc, fail it with this plugin with customizations. Default no-op

Copied from #344 as forked repro has issue of secrets

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration.

Test with current tests and added decorator tests

Checklist:

• My code follows the style guidelines of this project (ran black ., isort ., flake8 .)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

Screenshots (if applicable)

If applicable, add screenshots to help showcase your changes.

Additional context

Add any other context about the PR here.

---

Note

Adds ResponseQualityError (code 103) and a pluggable rollout result post-processor that can trigger backoff retries; updates status/exception mappings and tests.

• Exceptions/Status:
  • Add ResponseQualityError with code 103; map in STATUS_CODE_TO_EXCEPTION.
  • Extend Status.Code and add Status.response_quality_error() factory.
• Pytest framework:
  • Introduce RolloutResultPostProcessor and NoOpRolloutResultPostProcessor; export via pytest.__init__.
  • Extend RolloutProcessorConfig with optional post_processor.
  • Apply post-processing in evaluation_test_utils.rollout_processor_with_retry; re-raise ResponseQualityError to activate retry logic.
  • Include ResponseQualityError in DEFAULT_RETRYABLE_EXCEPTIONS; minor backoff config/type tweaks.
• Tests:
  • Add/extend tests for exception mapping and status code consistency.
  • Add tests for backoff decorator and inclusion of ResponseQualityError in retryable set.
  • Add retry mechanism tests covering ResponseQualityError behavior.

^{Written by Cursor Bugbot for commit 873828e. This will update automatically on new commits. Configure here.}