Skip to content

Conversation

@a10y
Copy link
Contributor

@a10y a10y commented Jan 19, 2026

This adds the ability to write in PyVortex directly to object storage, along with more ergonomic configuration of object store credentials for the major PyVortex read/write paths: vx.open, vx.io.read, and vx.io.write, as well as VortexWriteOptions.write.

This is using the https://docs.rs/pyo3-object_store/latest/pyo3_object_store/ crate, which is what runs the obstore Python library.

Notably, we can't reuse obstore and all of its types directly. This is b/c of the way it works, you're required to actually link in pyo3-object_store and then expose it as a new module within your Python bundle.

There are some other examples of this pattern:

Sadly there's no nicely distributed types package, so I need to copy all of the pyi files here with their original MIT license.

I've added a Python unit test to demonstrate using the new object store builders with simple local file system object store

Resolves #5673

@a10y a10y added the changelog/feature A new feature label Jan 19, 2026
@a10y a10y requested review from danking and gatesn January 19, 2026 02:48
@codecov
Copy link

codecov bot commented Jan 19, 2026

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 82.93%. Comparing base (12ba988) to head (cc7a06b).
⚠️ Report is 11 commits behind head on develop.

Files with missing lines Patch % Lines
vortex-io/src/write.rs 0.00% 1 Missing ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@a10y a10y force-pushed the aduffy/pyvortex-obstore branch from 64382ba to 77ac62a Compare January 22, 2026 15:53
@a10y a10y added the feature A feature request label Jan 22, 2026
@codspeed-hq
Copy link

codspeed-hq bot commented Jan 22, 2026

CodSpeed Performance Report

Merging this PR will degrade performance by 39.87%

Comparing aduffy/pyvortex-obstore (2d94025) with develop (1ddae04)

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

Summary

⚡ 5 improved benchmarks
❌ 15 regressed benchmarks
✅ 1254 untouched benchmarks
⏩ 1254 skipped benchmarks1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime u8_FoR[1K] 10.2 µs 8.8 µs +16.94%
WallTime u8_FoR[10M] 7 µs 7.8 µs -10.05%
WallTime u16_FoR[1M] 6.2 µs 7.5 µs -17.07%
WallTime u64_FoR[10K] 8 µs 13.4 µs -39.87%
WallTime u8_FoR[10K] 7 µs 6 µs +18.02%
WallTime u8_FoR[1M] 5.7 µs 7.7 µs -25.4%
WallTime u16_FoR[10K] 6.4 µs 7.8 µs -17.76%
Simulation canonical_into_non_nullable[(10000, 1, 0.01)] 36 µs 44.2 µs -18.45%
Simulation canonical_into_non_nullable[(10000, 1, 0.0)] 30.9 µs 39 µs -20.79%
Simulation canonical_into_non_nullable[(10000, 1, 0.1)] 52 µs 60.2 µs -13.71%
Simulation canonical_into_non_nullable[(10000, 100, 0.0)] 1.9 ms 2.7 ms -29.5%
Simulation canonical_into_non_nullable[(10000, 100, 0.01)] 2.1 ms 2.9 ms -27.4%
Simulation canonical_into_non_nullable[(10000, 100, 0.1)] 3.7 ms 4.5 ms -18.03%
Simulation canonical_into_nullable[(10000, 10, 0.0)] 528.8 µs 444.6 µs +18.93%
Simulation into_canonical_non_nullable[(10000, 100, 0.0)] 1.9 ms 2.7 ms -29.35%
Simulation into_canonical_non_nullable[(10000, 100, 0.1)] 3.8 ms 4.6 ms -17.82%
Simulation into_canonical_non_nullable[(10000, 100, 0.01)] 2.2 ms 3 ms -26.73%
Simulation into_canonical_nullable[(10000, 10, 0.1)] 710.5 µs 632.3 µs +12.37%
Simulation into_canonical_nullable[(10000, 10, 0.0)] 537.4 µs 452.3 µs +18.82%
Simulation into_canonical_nullable[(10000, 100, 0.0)] 4.4 ms 5.2 ms -15.77%

Footnotes

  1. 1254 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Signed-off-by: Andrew Duffy <andrew@a10y.dev>
@a10y a10y force-pushed the aduffy/pyvortex-obstore branch from 35ed5b7 to 190d169 Compare January 22, 2026 17:22
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
@a10y a10y force-pushed the aduffy/pyvortex-obstore branch from b4e95cb to ac3aff7 Compare January 22, 2026 18:31
a10y added 2 commits January 22, 2026 14:15
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature feature A feature request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants