-
Notifications
You must be signed in to change notification settings - Fork 118
feat[python]: support full read/write from object storage #6022
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Conversation
Codecov Report❌ Patch coverage is
☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
64382ba to
77ac62a
Compare
CodSpeed Performance ReportMerging this PR will degrade performance by 39.87%Comparing
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | u8_FoR[1K] |
10.2 µs | 8.8 µs | +16.94% |
| ❌ | WallTime | u8_FoR[10M] |
7 µs | 7.8 µs | -10.05% |
| ❌ | WallTime | u16_FoR[1M] |
6.2 µs | 7.5 µs | -17.07% |
| ❌ | WallTime | u64_FoR[10K] |
8 µs | 13.4 µs | -39.87% |
| ⚡ | WallTime | u8_FoR[10K] |
7 µs | 6 µs | +18.02% |
| ❌ | WallTime | u8_FoR[1M] |
5.7 µs | 7.7 µs | -25.4% |
| ❌ | WallTime | u16_FoR[10K] |
6.4 µs | 7.8 µs | -17.76% |
| ❌ | Simulation | canonical_into_non_nullable[(10000, 1, 0.01)] |
36 µs | 44.2 µs | -18.45% |
| ❌ | Simulation | canonical_into_non_nullable[(10000, 1, 0.0)] |
30.9 µs | 39 µs | -20.79% |
| ❌ | Simulation | canonical_into_non_nullable[(10000, 1, 0.1)] |
52 µs | 60.2 µs | -13.71% |
| ❌ | Simulation | canonical_into_non_nullable[(10000, 100, 0.0)] |
1.9 ms | 2.7 ms | -29.5% |
| ❌ | Simulation | canonical_into_non_nullable[(10000, 100, 0.01)] |
2.1 ms | 2.9 ms | -27.4% |
| ❌ | Simulation | canonical_into_non_nullable[(10000, 100, 0.1)] |
3.7 ms | 4.5 ms | -18.03% |
| ⚡ | Simulation | canonical_into_nullable[(10000, 10, 0.0)] |
528.8 µs | 444.6 µs | +18.93% |
| ❌ | Simulation | into_canonical_non_nullable[(10000, 100, 0.0)] |
1.9 ms | 2.7 ms | -29.35% |
| ❌ | Simulation | into_canonical_non_nullable[(10000, 100, 0.1)] |
3.8 ms | 4.6 ms | -17.82% |
| ❌ | Simulation | into_canonical_non_nullable[(10000, 100, 0.01)] |
2.2 ms | 3 ms | -26.73% |
| ⚡ | Simulation | into_canonical_nullable[(10000, 10, 0.1)] |
710.5 µs | 632.3 µs | +12.37% |
| ⚡ | Simulation | into_canonical_nullable[(10000, 10, 0.0)] |
537.4 µs | 452.3 µs | +18.82% |
| ❌ | Simulation | into_canonical_nullable[(10000, 100, 0.0)] |
4.4 ms | 5.2 ms | -15.77% |
Footnotes
-
1254 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
35ed5b7 to
190d169
Compare
b4e95cb to
ac3aff7
Compare
This adds the ability to write in PyVortex directly to object storage, along with more ergonomic configuration of object store credentials for the major PyVortex read/write paths:
vx.open,vx.io.read, andvx.io.write, as well asVortexWriteOptions.write.This is using the https://docs.rs/pyo3-object_store/latest/pyo3_object_store/ crate, which is what runs the obstore Python library.
Notably, we can't reuse
obstoreand all of its types directly. This is b/c of the way it works, you're required to actually link in pyo3-object_store and then expose it as a new module within your Python bundle.There are some other examples of this pattern:
Sadly there's no nicely distributed types package, so I need to copy all of the pyi files here with their original MIT license.
I've added a Python unit test to demonstrate using the new object store builders with simple local file system object store
Resolves #5673