Support `__test__` attribute in PyTest test discovery

[James-Ansley]

A recent release included support for alternative (i.e. non-"test") test prefixes.

Pytest, also uses a custom __test__ dunder when discovering tests: python.py#L349. I.e. any function with the attribute __test__ = True is considered a test regardless of its prefix. Currently, this library does not support this dunder.

Would it be possible to support the __test__ dunder in the recently added is_pytest_test method?

I am happy to work on this unless someone else sees a quick fix.

Suggested Fix

An additional test in the StackFrameNamer class for the __test__ dunder could be something like the following:

@staticmethod
def _is_marked_with_test_dunder(
      method_name: str,
      frame_globals: dict[str, "Any"],
) -> bool:
    function = frame_globals.get(method_name)
    return (
          function is not None
          and hasattr(function, "__test__")
          and getattr(function, "__test__") is True  # i.e. True not truthy
    )

@staticmethod
def is_pytest_test(frame: FrameInfo) -> bool:
    method_name = frame[3]
    frame_globals = frame.frame.f_globals
    patterns = PytestConfig.test_naming_patterns
    return (
          StackFrameNamer._is_match_for_pytest(method_name, patterns)
          or StackFrameNamer._is_marked_with_test_dunder(
                method_name,
                frame_globals,
          )
    )

Here, the function name is looked up in the frame globals, and if found, it checks for the __test__ dunder. I would need to refresh my memory on Python's FrameInfo to check whether this is the best way of doing this.

[nitsanavni]

Looks good!
Usually I would invite you to the weekly ensemble, but it's not NZ timezone friendly unfortunately. Anyway, know that you're always welcome there - https://github.com/approvals/ApprovalTests.Python/blob/main/docs/Contribute.md.

[JayBazuzi]

Bumping this to the top.

[JayBazuzi]

Can we use PyTestCollector.istestfunction()?

[JayBazuzi]

Here's an example test case that demonstrates the issue:

def non_compliant_test_name() -> None:
    verify("foo")


non_compliant_test_name.__test__ = True

[James-Ansley]

Just context for anyone else reading this, this is useful when using decorators to mark test cases. For example:

# tests/utils.py
def test(f):
    f.__test__ = True
    return f

# tests/my_cool_test.py
@test
def one_plus_one_is_eleven():
    verify("1" + "1")

Using slightly more zany decorators that take parameters, the behaviour of tests can be changed without all test cases needing to directly rely on PyTest-specific methods/decorators.