Feat: replaced hardcoded test server admin key with env variable #1568

rohansen856 · 2025-12-29T09:37:02Z

Metadata

Reference Issue: Hide test server admin key #1529
New Tests Added: Yes
Documentation Updated: No
Change Log Entry:

Details

this PR is made to remove the hardcoded test server admin key from the codebase and replace it with environment variable-based authentication.

Summary

in openml/config.py Added a new environment variable constant for the test server admin key:

OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR = "OPENML_TEST_SERVER_ADMIN_KEY"

Testing Base Class Updated in openml/testing.py. Modified the TestBase class to read the admin key from an environment variable instead of using a hardcoded value:

Before:

admin_key = "abc"

After:

admin_key = os.environ.get(openml.config.OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR)

Note:

The admin key is now None by default when the environment variable is not set
Tests requiring the admin key will fail gracefully if the key is not available

Also in the tests, Added pytest.skipif decorators to tests that require admin privileges in the following test files:

`tests/test_openml/test_config.py`

Test: test_switch_to_example_configuration

Added decorator:

@pytest.mark.skipif(
    not os.environ.get(openml.config.OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR),
    reason="Test requires admin key. Set OPENML_TEST_SERVER_ADMIN_KEY environment variable.",
)

`tests/test_datasets/test_dataset_functions.py`

Test: test_data_status

Added decorator:

@pytest.mark.skipif(
    not os.environ.get(openml.config.OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR),
    reason="Test requires admin key. Set OPENML_TEST_SERVER_ADMIN_KEY environment variable.",
)

Note:

These tests will be automatically skipped if the admin key is not provided
Clear skip reason is displayed when tests are skipped
No failures or errors when running tests locally without the admin key

4. CI Configuration Update (`.github/workflows/test.yml`)

Added the environment variable to all test execution steps in the GitHub Actions workflow:

Steps updated:

Run tests on Ubuntu Test
Run tests on Ubuntu Production
Run tests on Windows

Added to each step:

env:
  OPENML_TEST_SERVER_ADMIN_KEY: ${{ secrets.OPENML_TEST_SERVER_ADMIN_KEY }}

Impact:

CI will use the secret stored in GitHub repository settings
Tests requiring admin privileges will run in CI
The actual key value is never exposed in logs or code

@PGijsbers this requires someone to put the admin key in the github secrets which would be a critical step.

Signed-off-by: rohansen856 <rohansen856@gmail.com>

fkiraly

This successfully removes the secret from the code base.

My only question, is there a documentation location (e.g., developer documentation) that also needs to get updated about this? Since developers wishing to use the test server now have to:

obtain credentials (from whom or where?)
set the environment variable in python

codecov-commenter · 2025-12-29T15:51:44Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.76%. Comparing base (99928f8) to head (455dad2).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1568      +/-   ##
==========================================
+ Coverage   52.75%   52.76%   +0.01%     
==========================================
  Files          36       36              
  Lines        4333     4334       +1     
==========================================
+ Hits         2286     2287       +1     
  Misses       2047     2047

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rohansen856 · 2025-12-29T17:03:57Z

This successfully removes the secret from the code base.

My only question, is there a documentation location (e.g., developer documentation) that also needs to get updated about this? Since developers wishing to use the test server now have to:

obtain credentials (from whom or where?)

set the environment variable in python

I think if any documentation update has to be done it should be in Advanced user guide (https://openml.github.io/openml-python/latest/details/). As for local testing, there is no .env file in the codebase so it needs to be provided by the developer before testing (ex.: export OPENML_TEST_SERVER_ADMIN_KEY="your-admin-key" && pytest tests/) But even if a new developer does not have the context about this, The tests would automatically skip if the developer does not provide the variable so tests wont fail because of this.

also this is the admin API key for the OpenML test server (https://test.openml.org`) (even though we used very weak credentials for testing purposes: abc). so it should be behind a env var. So instead of having it exposed, using it as a secret in github actions would be better and i can update the documentation for any developer who want to actually run the two tests that would be skipped any way in case of local testing if they dont provide the value.

Please do let me know if you have other suggestions on this @fkiraly

fkiraly · 2025-12-29T17:14:29Z

@jgyasu, does this need to be added to the developer guide you are writing?

geetu040

Looks fine, just need to add code for loading .env if that is required. And we should also mention this in the internal developer setup.

geetu040 · 2026-01-12T14:46:17Z

.github/workflows/test.yml

    - name: Run tests on Ubuntu Test
      if: matrix.os == 'ubuntu-latest'
+      env:
+        OPENML_TEST_SERVER_ADMIN_KEY: ${{ secrets.OPENML_TEST_SERVER_ADMIN_KEY }}


we should make sure these are added in the github secrets before merging this

geetu040 · 2026-01-12T14:46:28Z

openml/config.py


 OPENML_CACHE_DIR_ENV_VAR = "OPENML_CACHE_DIR"
 OPENML_SKIP_PARQUET_ENV_VAR = "OPENML_SKIP_PARQUET"
+OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR = "OPENML_TEST_SERVER_ADMIN_KEY"


in case of using a .env file, some code is required to load it and may add the extra dependency python-dotenv

yes definitely! Will go ahead with the new dep if approved.

geetu040 · 2026-01-12T14:46:47Z

tests/test_openml/test_config.py

+    )
    @pytest.mark.production()
    def test_switch_to_example_configuration(self):
        """Verifies the test configuration is loaded properly."""


btw this test passes anyways even if you use any random admin_key, right?

Yes, test_switch_to_example_configuration doesn't make any API calls, it only tests that start_using_configuration_for_example() correctly swaps the apikey and server values. So it passes with any value (even None is ok)

But test_data_status (tests\test_datasets\test_dataset_functions.py) does require a valid admin key because it makes actual API calls to openml.datasets.status_update() which the server validates, so if a user passes invalid value it would return authentication error:

FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_data_status - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/status/update ...

Yes, I think the more reasonable approach here is to update this test to use some other sentinel value.

PGijsbers

Mostly LGTM, but I am not yet sold on introducing the .env file.

Regular user settings are currently configured through a dedicated configuration file already. I'm not yet convinced that adding a .env file for (developer) configuration is more logical than extending the existing configuration file. A decision should probably be made in the larger context on how to handle configurations?

Something that seems reasonable is to support openml config files in ./.openml/config for directory level configurations.

In my opinion we can merge this as is (except for the note on test_switch_to_example_configuration), and then discuss ideas on how we should have user and developer configurations after.

PGijsbers · 2026-01-16T13:23:20Z

.github/workflows/test.yml

    - name: Run tests on Ubuntu Test
      if: matrix.os == 'ubuntu-latest'
+      env:
+        OPENML_TEST_SERVER_ADMIN_KEY: ${{ secrets.OPENML_TEST_SERVER_ADMIN_KEY }}


PGijsbers · 2026-01-16T13:28:04Z

tests/test_openml/test_config.py

+    )
    @pytest.mark.production()
    def test_switch_to_example_configuration(self):
        """Verifies the test configuration is loaded properly."""


Yes, I think the more reasonable approach here is to update this test to use some other sentinel value.

rohansen856 added 3 commits December 27, 2025 16:23

feat: implemented env var based admin key loading

808ab63

Signed-off-by: rohansen856 <rohansen856@gmail.com>

tests: updated tests to use the env var for key instead of hardcoding it

f183d11

Signed-off-by: rohansen856 <rohansen856@gmail.com>

refactor: updated github action to use secrets

7a94293

Signed-off-by: rohansen856 <rohansen856@gmail.com>

fkiraly requested changes Dec 29, 2025

View reviewed changes

geetu040 suggested changes Jan 12, 2026

View reviewed changes

rohansen856 added 3 commits January 13, 2026 12:06

Merge branch 'main' into admin_hey_hide

325c328

Merge branch 'main' into admin_hey_hide

4251321

Merge branch 'main' into admin_hey_hide

455dad2

PGijsbers requested changes Jan 16, 2026

View reviewed changes

Uh oh!

Feat: replaced hardcoded test server admin key with env variable #1568

Are you sure you want to change the base?

Feat: replaced hardcoded test server admin key with env variable #1568

Conversation

rohansen856 commented Dec 29, 2025

Metadata

Details

Summary

tests/test_openml/test_config.py

tests/test_datasets/test_dataset_functions.py

4. CI Configuration Update (.github/workflows/test.yml)

Uh oh!

fkiraly left a comment

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rohansen856 commented Dec 29, 2025

Uh oh!

fkiraly commented Dec 29, 2025

Uh oh!

geetu040 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohansen856 Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PGijsbers left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

`tests/test_openml/test_config.py`

`tests/test_datasets/test_dataset_functions.py`

4. CI Configuration Update (`.github/workflows/test.yml`)

codecov-commenter commented Dec 29, 2025 •

edited

Loading

rohansen856 Jan 15, 2026 •

edited

Loading

PGijsbers left a comment •

edited

Loading