feat: federated multi-package template with crane and nix2container #44
+8,295
−2,920
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Cameron Smith <cameron.ray.smith@gmail.com>
Add 3 epics and 11 child issues for python-nix-template evolution: - pnt-dre: Infrastructure alignment (5 tasks) - pnt-4jg: Dependency model migration (3 tasks) - pnt-btz: pyo3/Rust extension integration (3 tasks) pnt-btz depends on pnt-dre and pnt-4jg completion.
Signed-off-by: Cameron Smith <cameron.ray.smith@gmail.com>
Document the target architecture for integrating Rust extension modules into the federated Python monorepo pattern. Key patterns covered: - Per-package Rust workspace (crates/ inside Python package) - crane buildDepsOnly for cargoArtifacts caching - uv2nix overlay injection for maturin builds - CI matrix with rust-deps/rust-checks/python-wheel categories References local exemplars: - ~/projects/rust-workspace/ironstar (crane patterns) - ~/projects/planning-workspace/langchain (federated Python) - ~/projects/nix-workspace/pyproject.nix (maturin support)
Align pnt-btz epic and tasks with the documented crane + uv2nix integration architecture in docs/notes/architecture/. Updates: - pnt-btz: reference architecture doc, list key patterns - pnt-btz.1: detail federated crates/ structure, maturin binding - pnt-btz.2: specify rust.nix and default.nix overlay patterns - pnt-btz.3: add validation checklist and CI matrix requirements
Root pyproject.toml no longer declares [tool.uv.workspace] with members. Each package resolves independently with its own lock, following the LangChain federated model. Shared tool config (pytest, ruff, coverage) retained for IDE config discovery.
Replace single root workspace with per-package loadWorkspace calls, following Pattern 3 from crane-uv2nix-integration architecture doc. Each package directory is loaded independently and overlays are composed via composeManyExtensions. Exposes packageWorkspaces instead of baseWorkspace for downstream module consumption.
Adapt packages.nix to consume packageWorkspaces instead of baseWorkspace. Merges deps.default from all independent package workspaces to construct the combined virtualenv.
Adapt devshell.nix to consume packageWorkspaces instead of baseWorkspace. Merges deps.all from all independent package workspaces for the development virtualenv.
Adapt containers.nix to consume packageWorkspaces instead of baseWorkspace. Merges deps from all independent package workspaces for container image virtualenvs.
Update cache-dependency-glob from root uv.lock to packages/*/uv.lock in python-test and package-release workflows. Move uv sync, lint, and test steps into per-package working directories since the root uv workspace no longer exists. Update ci.yaml path filters to match per-package lock file paths.
Replace // (shallow merge) with lib.zipAttrsWith for dep-spec merging across federated package workspaces. The // operator silently drops extras from the first package when two packages share a dependency name with different extras lists. zipAttrsWith unions the lists instead. Applied consistently across packages.nix, devshell.nix, and containers.nix.
…kspaces Document the version-conflict invariant for federated overlay composition in python.nix and crane-uv2nix-integration.md. Per-package uv2nix overlays composed via composeManyExtensions silently override shared dependencies — packages must resolve compatible versions.
…kspaces Document version-conflict invariant in architecture doc Pattern 3.
Aligns with pnt-functional by defining test, lint, lint-check, and types tasks under their respective pixi feature sections.
Replaces raw tool invocations with pixi task delegation so that working directory resolution happens inside the package directory where pixi sets it relative to the manifest path.
Resolves deprecation warning for top-level channels in [package.build].
…late Resolves deprecation warning for top-level channels in [package.build].
…ntainer production containers Add pnt-cli production container via nix2container.buildImage with 2-layer strategy. Replace flocken manifest tooling with crane-based mkMultiArchManifest for all containers (nix: transport for production, docker-archive: transport for dev). Add containerMatrix flake output for CI matrix generation via pure nix evaluation.
Move container recipes from 'nix' group to dedicated 'containers' section. Add production container build/load/push recipes for nix2container pattern. Update dev container recipes to use --no-link --print-out-paths pattern. Add container-matrix recipe for CI integration. Fix image tags to match actual image names (python-nix-template-dev).
Replace monolithic container build workflow with discover/build/manifest pattern driven by nix eval .#containerMatrix. Production containers skip QEMU (nix2container), dev containers retain QEMU for cross-compilation. Manifest push uses crane-based tooling with nix: and docker-archive: transport respectively.
The transitive call chain test-release-packages -> package-release.yaml -> build-nix-images.yaml requires packages:write, which GitHub validates at parse time even for conditionally-skipped jobs.
The manifest job references needs.discover.outputs.manifest-matrix but only listed build in its needs array.
The federation migration removed [tool.uv.workspace] and never had a [project] table at root. This causes omnix template init to generate projects where uv lock at root fails with no project to resolve.
Remove GitGuardian scan and set-variables jobs from template workflow. Use setup-nix composite action instead of inline nix installer and cachix setup. Reduce permissions to contents:read only.
Port category-based build script from typescript-nix-template. Supports packages, checks, and devshells categories with per-system discovery via nix eval and individual attribute builds.
Add ci-build-category, scan-secrets, scan-staged recipes to CI/CD group. Add preview-version and release-package recipes to release group for semantic-release via nix develop.
Replace monolithic nixci/om-ci-run job with 3-entry category matrix (packages, checks, devshells) using ci-build-category.sh. Replace GitGuardian with gitleaks via nix run. Add flake-validation and bootstrap-verification jobs. Add preview-release-version job for PRs. Reduce top-level permissions to contents:read with per-job overrides.
…ckage-release Remove astral-sh/setup-uv and threeal/setup-yarn-action, use setup-nix composite action instead. Wrap yarn and uv commands with nix develop. Add cached-ci-job support with dry-run-only cache persistence per ADR-0016. Fix artifact upload path for per-package working directory.
Gitleaks is now in the devshell, so use nix develop -c just scan-secrets instead of nix run nixpkgs#gitleaks directly. This ensures the same command works locally and in CI.
The build-pr-images job calls build-nix-images.yaml which declares packages:write at the workflow level. With ci.yaml top-level permissions reduced to contents:read, the caller must explicitly provide packages:write.
The full nix flake check evaluates all systems including aarch64-linux, which fails on x86_64-linux CI runners due to IFD in nixpod container dependencies (catppuccin-starship). System-specific evaluation and build of checks avoids this cross-system issue.
Manifest packages reference packages from other systems (e.g. aarch64-linux on x86_64-linux runners) and cannot be built in single-system CI jobs. The dedicated build-nix-images workflow handles manifest creation with proper multi-arch infrastructure.
The root pyproject.toml and packages/python-nix-template both used the name python-nix-template, causing uv to error with "Two workspace members are both named" during ci-sync in test-python jobs.
The preview-release-version CI job discovers all packages and runs yarn workspace commands, which requires a package.json in each package directory.
The checks flake output is an attribute set, not a single derivation. Build each check individually to avoid nix build type mismatch.
The dry-run was using GITHUB_REF_NAME which resolves to the PR branch, not a configured release branch. Semantic-release rejects branches not in its branches config, causing ERELEASEBRANCHES errors on PR branches.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.