lazy loading of all modules in init

[shreymodi1]

Note

Improves import/startup performance via lazy loading and adds scheduled benchmarks to catch regressions.

• Introduces lazy exports in eval_protocol/__init__.py using __getattr__, optional placeholders, and a lazily computed __version__; only eagerly imports conflicting symbols under pytest
• Refactors eval_protocol/pytest/__init__.py to lazy-load rollout processors, adapters, and flags; adds TYPE_CHECKING hints
• Updates evaluation_test.py and evaluation_test_utils.py to defer imports of SingleTurnRolloutProcessor, MCPGymRolloutProcessor, and litellm.cost_calculator until needed
• Reworks exception_config.py to lazily build DEFAULT_RETRYABLE_EXCEPTIONS (avoids importing litellm/requests/httpx at module load) and tightens types
• Adds tests/test_cli_startup_benchmark.py with thresholds for CLI and import times; registers benchmark marker in pytest.ini
• Adds .github/workflows/benchmark.yml to run daily multi-Python benchmarks, upload artifacts, and notify Slack on failure

^{Written by Cursor Bugbot for commit ce6742f. This will update automatically on new commits. Configure here.}

[cursor]

Optional adapter imports raise AttributeError instead of returning None

Medium Severity

The __getattr__ handler for optional imports only catches ImportError, but when the .adapters module is importable (it always is) but an individual adapter class like WeaveAdapter isn't available (due to missing dependencies), getattr(module, attr_name) raises AttributeError. This exception is not caught, so users get an AttributeError instead of None. The old code's from .adapters import WeaveAdapter would raise ImportError: cannot import name 'WeaveAdapter', which was caught. The fix requires catching both ImportError and AttributeError.

 

[dphuang2]

@shreymodi1 how do we make sure this isn't a flaky test.

I see at https://github.com/eval-protocol/python-sdk/actions/runs/20938395911/job/60298640888:

FAILED tests/test_cli_startup_benchmark.py::test_evaluation_test_import_time - AssertionError: evaluation_test import time (8.826s) exceeds target (5.0s).
assert 8.825818 < 5.0

1. Either keep this as a smoke test that runs on schedule so it doesn't block PRs
2. Create a more robust assertion formula

[cursor]

WeaveAdapter missing from module __all__ export list

Low Severity

WeaveAdapter was accidentally omitted from __all__ during the lazy loading refactor. It's correctly added to _OPTIONAL_IMPORTS (line 115) and the TYPE_CHECKING imports (line 299), but unlike all other adapters (OpenAIResponsesAdapter, LangfuseAdapter, BraintrustAdapter, LangSmithAdapter), it's not included in the __all__ list. This breaks from eval_protocol import * for users who rely on WeaveAdapter and causes inconsistency with how other adapters are exported.

 

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

[cursor]

Import error silently caught by broad exception handler

Medium Severity

Moving the from litellm.cost_calculator import cost_per_token import inside the try block causes ImportError to be caught by the except Exception handler. Previously, if litellm wasn't installed, the module would fail to load with a clear error. Now, missing litellm silently sets costs to 0.0 with a misleading debug message about the model, masking a dependency issue.