feat: highly available clickhouse deployment #27
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The previous clickhouse deployment was setup to run as a single node cluster preventing us from having a highly available deployment. The database schema has been adjusted to create a replicated database so that Clickhouse will automatically replicate schema changes and data between replicas. This schema change is a breaking change so the environments will be re-created. I've also adjusted the ordering of audit logs to use the timestamp as a secondary sort column so we can maintain strict ordering of audit logs. There's still benefits in maintaining the hourly bucketing so Clickhouse can skip over entire hours of data through indexes. The apiserver has been updated to use the new ordering. The migration script will now wait for all replicas in the cluster to come up before executing migrations.
scotwells
requested review from
JoseSzycho,
cc-datum,
drewr,
ecv and
zachsmith1
Need operational visibility into Clickhouse Keeper so we can monitor the component and understand replication delay between cluster replicas.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The previous clickhouse deployment was setup to run as a single node cluster preventing us from having a highly available deployment.
The database schema has been adjusted to create a replicated database so that Clickhouse will automatically replicate schema changes and data between replicas.
Important
This schema change is a breaking change so the environments will be re-created.
I've also adjusted the ordering of audit logs to use the timestamp as a secondary sort column so we can maintain strict ordering of audit logs. There's still benefits in maintaining the hourly bucketing so Clickhouse can skip over entire hours of data through indexes. The apiserver has been updated to use the new ordering.
The migration script will now wait for all replicas in the cluster to come up before executing migrations.
Relates to datum-cloud/enhancements#536