feat: implement GitLab Terraform state scanner command (gl tf) #480
Conversation
Add new 'gl tf' subcommand to discover, download, and scan native Terraform/OpenTofu state files stored in GitLab for secrets using TruffleHog integration. Changes: - internal/cmd/gitlab/tf/tf.go: CLI command with flag binding and config integration - pkg/gitlab/tf/tf.go: Business logic with project iteration, state detection, and scanning - tests/e2e/gitlab/tf/tf_test.go: Comprehensive E2E test coverage (5 test cases) - internal/cmd/gitlab/gitlab.go: Register tf command - docs/introduction/configuration.md: Add tf configuration documentation - pipeleek.example.yaml: Add tf configuration example Features: - Discovers all projects with Maintainer access - Checks for native Terraform state via GitLab API - Downloads state files concurrently with configurable threads - Scans state files for secrets with TruffleHog - Saves state files to configurable output directory - Full config file/env/flag integration - Structured logging with secret detection events Test Results: - All E2E tests passing (5/5) - All unit tests passing (no regressions) - Verified: TestTFBasic, TestTFNoState, TestTFInvalidURL, TestTFMissingToken, TestTFOutputDir
There was a problem hiding this comment.
Pull request overview
This PR implements a new GitLab Terraform state scanner command (gl tf) that discovers, downloads, and scans Terraform/OpenTofu state files stored in GitLab for secrets using TruffleHog integration. The implementation follows project patterns for command structure, configuration integration, and testing.
Changes:
- Added new
gl tfsubcommand with full CLI interface and configuration support - Implemented business logic for discovering and scanning Terraform state files in projects with Maintainer access
- Added comprehensive E2E test coverage with 5 test cases
- Updated configuration documentation and example files
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/cmd/gitlab/tf/tf.go | CLI command implementation with flag binding, validation, and config integration following project patterns |
| pkg/gitlab/tf/tf.go | Core business logic for iterating projects, detecting states, downloading files, and scanning with TruffleHog |
| tests/e2e/gitlab/tf/tf_test.go | Comprehensive E2E tests covering basic functionality, error cases, and edge cases |
| internal/cmd/gitlab/gitlab.go | Registration of new tf subcommand in GitLab command tree |
| docs/introduction/configuration.md | Documentation of tf command configuration options |
| pipeleek.example.yaml | Example configuration showing tf command settings |
| } | ||
|
|
||
| var options = TFCommandOptions{CommonScanOptions: config.DefaultCommonScanOptions()} | ||
| var maxArtifactSize string |
There was a problem hiding this comment.
The variable maxArtifactSize is declared but never used in this command. It's only passed to AddCommonScanFlags but the value is never read or utilized later in the tfRun function. This variable should be removed if it's not needed for the Terraform state scanning functionality.
| tfCmd.Flags().StringVar(&options.OutputDir, "output-dir", "./terraform-states", "Directory to save downloaded state files") | ||
|
|
||
| // Common scan flags (threads, verification, confidence, hit-timeout, etc.) | ||
| flags.AddCommonScanFlags(tfCmd, &options.CommonScanOptions, &maxArtifactSize) |
There was a problem hiding this comment.
The AddCommonScanFlags function adds several flags that are not relevant to the Terraform state scanning functionality:
--artifacts(line 16 in flags/common.go): TF state scanning doesn't scan artifacts--max-artifact-size(line 17 in flags/common.go): Not applicable to TF state files--owned(line 21 in flags/common.go): TF command already filters by Maintainer access, making this flag redundant
These flags will appear in pipeleek gl tf --help but won't have any effect, which could confuse users. Consider either:
- Creating a more tailored flag addition function for commands that don't scan artifacts, or
- Documenting which flags are not applicable to this command
| and scans them for secrets using TruffleHog. | ||
|
|
||
| GitLab stores Terraform state natively when using the Terraform HTTP backend. | ||
| Each project can have multiple named state files.`, |
There was a problem hiding this comment.
The documentation states "Each project can have multiple named state files" but the implementation only checks for and downloads the "default" state (hardcoded at line 93 in pkg/gitlab/tf/tf.go). This is misleading. Either:
- Update the documentation to accurately reflect that only the "default" state is scanned, or
- Extend the implementation to support discovering and scanning all named state files in a project
GitLab's API does support multiple named Terraform states per project, so this documentation suggests a feature that isn't fully implemented.
| Each project can have multiple named state files.`, | |
| While GitLab supports multiple named Terraform state files per project, this command currently scans only the default state for each project.`, |
| } | ||
|
|
||
| // sanitizeFilename removes invalid characters from filenames | ||
| func sanitizeFilename(name string) string { |
There was a problem hiding this comment.
if really needed use the golangs stdlib functions
| err := util.IterateProjects(git, projectOpts, func(project *gitlab.Project) error { | ||
| log.Debug().Str("project", project.PathWithNamespace).Int64("id", project.ID).Msg("Checking project for Terraform state") | ||
|
|
||
| // Check for Terraform state using HTTP API |
There was a problem hiding this comment.
check all created comments in all files. Only keep the ones providing real additional context. Remove all others.
Add new 'gl tf' subcommand to discover, download, and scan native Terraform/OpenTofu state files stored in GitLab for secrets using TruffleHog integration.
Changes:
Features:
Test Results: