fix: hottier in datasets

[nikhilsinhaparseable]

add hot_tier_enabled = true/false in StreamInfo that fetches the data from stream.json
remove redundant key hottier from DatasetsResponse as this info is available inside StreamInfo

Summary by CodeRabbit

Release Notes

• Refactor
  • Introduced a new helper method for stream information construction from metadata
  • Removed hot tier manager integration from stream processing operations
  • Added hot tier enabled flag to stream metadata

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

This PR refactors StreamInfo construction by introducing a factory method from_metadata() that consolidates field mapping from LogStreamMetadata. Simultaneously, hot tier integration is removed from the Prism logstream module while hot_tier_enabled remains as a metadata field in StreamInfo.

Changes

Cohort / File(s)	Summary
StreamInfo factory method src/storage/mod.rs	Added hot_tier_enabled: bool field to StreamInfo struct and introduced StreamInfo::from_metadata() associated function to consolidate field mapping from LogStreamMetadata (stream_type, created_at, time_partition, log_source, telemetry_type, etc.).
Hot tier removal & refactoring src/prism/logstream/mod.rs	Removed HotTierManager and StreamHotTier imports; deleted get_hot_tier_info() method and hottier field from PrismDatasetResponse. Replaced manual StreamInfo construction with factory method call.
Handler refactoring src/handlers/http/logstream.rs	Replaced manual StreamInfo field assembly with StreamInfo::from_metadata() factory method call in get_stream_info().

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• chore: persist hottier config in object storage #1453 — Shares the same StreamInfo surface area modifications (hot_tier_enabled field and metadata mapping logic).
• add telemetry type to a dataset #1392 — Related through StreamInfo construction/field population changes (telemetry_type propagation and factory method introduction).
• fix: datasets API edgecase where stream is not hottiered #1252 — Both modify hot tier logic in src/prism/logstream/mod.rs (removed here vs. error handling refinement).

Suggested reviewers

• praveen5959
• parmesant

Poem

🐰 A factory method hops into sight,
StreamInfo now builds with elegant might,
Hot tiers fade as metadata stays bright,
Refactoring dances—what a delightful flight! ✨

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title 'fix: hottier in datasets' is vague and does not clearly describe the main changes; it lacks specificity about what is being fixed or the actual modifications.	Provide a more specific title that describes the actual changes, such as 'refactor: move hot_tier_enabled to StreamInfo and remove redundant hottier field' or similar.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description check	✅ Passed	The pull request description provides the core changes but lacks adherence to the template structure, missing the 'Fixes #XXXX' section, detailed rationale, and the testing/documentation checklist items.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

🧹 Recent nitpick comments

src/prism/logstream/mod.rs (1)

339-340: Remove the unreachable Hottier error variant and its import.

The Hottier variant cannot be reached in this module since no function returns or propagates a HotTierError. The import at line 38 and the variant definition at lines 339-340 are unused, as is the corresponding match case in the ResponseError impl at line 361. Remove them in a cleanup pass.

---

📜 Recent review details

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f73fd87 and 2fc8aed.

📒 Files selected for processing (3)

• src/handlers/http/logstream.rs
• src/prism/logstream/mod.rs
• src/storage/mod.rs

🧰 Additional context used

🧠 Learnings (7)

📚 Learning: 2025-08-25T01:31:41.786Z

Learnt from: nikhilsinhaparseable
Repo: parseablehq/parseable PR: 1415
File: src/metadata.rs:63-68
Timestamp: 2025-08-25T01:31:41.786Z
Learning: The TOTAL_EVENTS_INGESTED_DATE, TOTAL_EVENTS_INGESTED_SIZE_DATE, and TOTAL_EVENTS_STORAGE_SIZE_DATE metrics in src/metadata.rs and src/storage/object_storage.rs are designed to track total events across all streams, not per-stream. They use labels [origin, parsed_date] to aggregate by format and date, while per-stream metrics use [stream_name, origin, parsed_date] labels.

Applied to files:

• src/handlers/http/logstream.rs
• src/prism/logstream/mod.rs
• src/storage/mod.rs

📚 Learning: 2025-08-25T01:32:25.980Z

Learnt from: nikhilsinhaparseable
Repo: parseablehq/parseable PR: 1415
File: src/metrics/mod.rs:163-173
Timestamp: 2025-08-25T01:32:25.980Z
Learning: The TOTAL_EVENTS_INGESTED_DATE, TOTAL_EVENTS_INGESTED_SIZE_DATE, and TOTAL_EVENTS_STORAGE_SIZE_DATE metrics in src/metrics/mod.rs are intentionally designed to track global totals across all streams for a given date, using labels ["format", "date"] rather than per-stream labels. This is the correct design for global aggregation purposes.

Applied to files:

• src/handlers/http/logstream.rs
• src/storage/mod.rs

📚 Learning: 2025-09-18T09:59:20.177Z

Learnt from: nikhilsinhaparseable
Repo: parseablehq/parseable PR: 1415
File: src/metrics/mod.rs:700-756
Timestamp: 2025-09-18T09:59:20.177Z
Learning: In src/event/mod.rs, the parsed_timestamp used in increment_events_ingested_by_date() is correctly UTC-normalized: for dynamic streams it remains Utc::now(), and for streams with time partition enabled it uses the time partition value. Both cases result in proper UTC date strings for metrics labeling, preventing double-counting issues.

Applied to files:

• src/handlers/http/logstream.rs
• src/storage/mod.rs

📚 Learning: 2025-10-28T02:10:41.140Z

Learnt from: nikhilsinhaparseable
Repo: parseablehq/parseable PR: 1453
File: src/parseable/mod.rs:397-400
Timestamp: 2025-10-28T02:10:41.140Z
Learning: In Parseable enterprise deployments with multiple query nodes, hot tier configuration must be persisted in object storage so that newly started query nodes can fetch and synchronize the hot tier settings at startup (file: src/parseable/mod.rs, function: create_stream_and_schema_from_storage).

Applied to files:

• src/prism/logstream/mod.rs
• src/storage/mod.rs

📚 Learning: 2025-02-14T09:49:25.818Z

Learnt from: de-sh
Repo: parseablehq/parseable PR: 1185
File: src/handlers/http/logstream.rs:255-261
Timestamp: 2025-02-14T09:49:25.818Z
Learning: In Parseable's logstream handlers, stream existence checks must be performed for both query and standalone modes. The pattern `!PARSEABLE.streams.contains(&stream_name) && (PARSEABLE.options.mode != Mode::Query || !PARSEABLE.create_stream_and_schema_from_storage(&stream_name).await?)` ensures proper error handling in both modes.

Applied to files:

• src/prism/logstream/mod.rs

📚 Learning: 2025-08-18T19:10:11.941Z

Learnt from: nikhilsinhaparseable
Repo: parseablehq/parseable PR: 1405
File: src/handlers/http/ingest.rs:163-164
Timestamp: 2025-08-18T19:10:11.941Z
Learning: Field statistics calculation in src/storage/field_stats.rs uses None for the time_partition parameter when calling flatten_and_push_logs(), as field stats generation does not require time partition functionality.

Applied to files:

• src/storage/mod.rs

📚 Learning: 2025-07-28T17:10:39.448Z

Learnt from: nikhilsinhaparseable
Repo: parseablehq/parseable PR: 1392
File: src/migration/stream_metadata_migration.rs:303-322
Timestamp: 2025-07-28T17:10:39.448Z
Learning: In Parseable's migration system (src/migration/stream_metadata_migration.rs), each migration function updates the metadata to the current latest format using CURRENT_OBJECT_STORE_VERSION and CURRENT_SCHEMA_VERSION constants, rather than producing incremental versions. For example, v5_v6 function produces v7 format output when these constants are set to "v7", not v6 format.

Applied to files:

• src/storage/mod.rs

🧬 Code graph analysis (2)

src/handlers/http/logstream.rs (1)

src/storage/mod.rs (1)

• from_metadata (179-197)

src/prism/logstream/mod.rs (1)

src/storage/mod.rs (1)

• from_metadata (179-197)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)

• GitHub Check: Agent
• GitHub Check: Build Default x86_64-unknown-linux-gnu
• GitHub Check: Build Default aarch64-unknown-linux-gnu
• GitHub Check: Build Default x86_64-pc-windows-msvc
• GitHub Check: Quest Smoke and Load Tests for Standalone deployments
• GitHub Check: Build Kafka x86_64-unknown-linux-gnu
• GitHub Check: Quest Smoke and Load Tests for Distributed deployments
• GitHub Check: Build Default aarch64-apple-darwin
• GitHub Check: Build Kafka aarch64-apple-darwin
• GitHub Check: coverage

🔇 Additional comments (5)

src/storage/mod.rs (2)

172-174: LGTM!

The hot_tier_enabled field is correctly added with #[serde(default)] for backward compatibility, ensuring it defaults to false when deserializing older data that lacks this field.

---

176-197: Clean factory method implementation.

The from_metadata factory method correctly consolidates StreamInfo construction, properly handling:

• Reference-based input to avoid ownership issues
• String cloning for reference-derived fields
• Type conversion for time_partition_limit via .map(|limit| limit.to_string())
• All StreamInfo fields including the new hot_tier_enabled

This centralizes the mapping logic and reduces code duplication across call sites.

src/handlers/http/logstream.rs (1)

359-360: LGTM!

The refactoring to use StreamInfo::from_metadata reduces code duplication and ensures consistent StreamInfo construction across the codebase. The factory method now handles the hot_tier_enabled field automatically from metadata.

src/prism/logstream/mod.rs (2)

38-38: LGTM on import cleanup.

The import is reduced to only HotTierError since hot tier retrieval logic is removed. The error type is still needed for the PrismLogstreamError enum variant.

---

179-180: LGTM!

Consistent with the refactoring in src/handlers/http/logstream.rs, this correctly delegates to StreamInfo::from_metadata. The hot_tier_enabled field is now automatically included from the metadata.

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

This pull request refactors hot tier information handling in datasets by consolidating it into the StreamInfo structure and removing redundant data from the Prism dataset response.

Changes:

• Added hot_tier_enabled boolean field to StreamInfo struct with data sourced from stream.json metadata
• Implemented from_metadata helper method on StreamInfo to construct instances from LogStreamMetadata
• Removed redundant hottier field from PrismDatasetResponse and associated logic

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
src/storage/mod.rs	Added hot_tier_enabled field to StreamInfo and implemented from_metadata factory method
src/prism/logstream/mod.rs	Removed unused hot tier imports, deleted get_hot_tier_info method, removed hottier field from PrismDatasetResponse, and replaced manual StreamInfo construction with from_metadata
src/handlers/http/logstream.rs	Replaced manual StreamInfo construction with from_metadata method call

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.