Skip to content

add pipe permission control specifications #981

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
leto-b wants to merge 1 commit into
base: main
Choose a base branch
from
+150 −17
Open

add pipe permission control specifications #981

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 19 additions & 4 deletions src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ By declaratively configuring the specific content of the three parts through SQL

### 1.2 Functional limitations and instructions

The schema and auth synchronization functions have the following limitations:
1. The schema and auth synchronization functions have the following limitations:

- When using schema synchronization, it is required that the consensus protocol for `Schema region` and `ConfigNode` must be the default ratis protocol. This means that the `iotdb-system.properties` configuration file should contain the settings `config_node_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus` and `schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus`. If these are not specified, the default ratis protocol is used.

Expand All @@ -89,6 +89,22 @@ The schema and auth synchronization functions have the following limitations:

- During data synchronization tasks, please avoid performing any deletion operations to prevent inconsistent states between the two ends.

2. Pipe Permission Control Specifications

- When creating a pipe, a username and password can be specified for the extraction/write‑back plugins. If the password is incorrect, creation is prohibited. If not specified, the current user is used for synchronization by default.

- During data/metadata synchronization, filtering is first performed according to the path pattern (pattern/path) configured in the Pipe, followed by authentication based on the user’s read permissions:
- If the permission scope is greater than or equal to the write path: full synchronization.
- If the permission scope has no intersection with the write path: no synchronization.
- If the permission scope is smaller than the write path or overlaps partially: synchronize only the intersecting part.

- When encountering data for which the user lacks permission:
- If the sender’s skipIf=no‑privileges, the unauthorized data is skipped.
- If skipIf is left empty (unconfigured), the task reports an error (Error 803).
- Note: This skipIf configuration is independent of the receiver’s skipIf setting (which defaults to empty).

- Data under root.__system and root.__audit will not be synchronized.

## 2. Usage Instructions

Data synchronization tasks have three states: RUNNING, STOPPED, and DROPPED. The task state transitions are shown in the following diagram:
Expand Down Expand Up @@ -514,7 +530,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
| password | Password for the username. | String | No | root |
| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true |
| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 |
| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 |
| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 |
| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 |
| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" |
| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 |
Expand All @@ -524,7 +540,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1

#### iotdb-thrift-ssl-sink


| **Parameter** | **Description** | Value Range | Required | Default Value |
|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ |
| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - |
Expand All @@ -533,7 +548,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
| password | Password for the username. | String | No | root |
| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true |
| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 |
| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 |
| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 |
| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 |
| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" |
| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 |
Expand Down
19 changes: 17 additions & 2 deletions src/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ By declaratively configuring the specific content of the three parts through SQL

### 1.2 Functional limitations and instructions

The schema and auth synchronization functions have the following limitations:
1. The schema and auth synchronization functions have the following limitations:

- When using schema synchronization, it is required that the consensus protocol for `Schema region` and `ConfigNode` must be the default ratis protocol. This means that the `iotdb-system.properties` configuration file should contain the settings `config_node_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus` and `schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus`. If these are not specified, the default ratis protocol is used.

Expand All @@ -91,6 +91,22 @@ The schema and auth synchronization functions have the following limitations:

- During data synchronization tasks, please avoid performing any deletion operations to prevent inconsistent states between the two ends.

2. Pipe Permission Control Specifications

- When creating a pipe, a username and password can be specified for the extraction/write‑back plugins. If the password is incorrect, creation is prohibited. If not specified, the current user is used for synchronization by default.

- During data/metadata synchronization, filtering is first performed according to the path pattern (pattern/path) configured in the Pipe, followed by authentication based on the user’s read permissions:
- If the permission scope is greater than or equal to the write path: full synchronization.
- If the permission scope has no intersection with the write path: no synchronization.
- If the permission scope is smaller than the write path or overlaps partially: synchronize only the intersecting part.

- When encountering data for which the user lacks permission:
- If the sender’s skipIf=no‑privileges, the unauthorized data is skipped.
- If skipIf is left empty (unconfigured), the task reports an error (Error 803).
- Note: This skipIf configuration is independent of the receiver’s skipIf setting (which defaults to empty).

- Data under root.__system and root.__audit will not be synchronized.

## 2. Usage Instructions

Data synchronization tasks have three states: RUNNING, STOPPED, and DROPPED. The task state transitions are shown in the following diagram:
Expand Down Expand Up @@ -607,7 +623,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:<br>​​sync​​: Wait for the local load tsfile operation to complete before returning the response.<br>​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync |
| format | The payload formats for data transmission include the following options:<br> - hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.<br> - tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.<br> - tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid |


#### iotdb-air-gap-sink

| key | value | value Range | required or not | Default Value |
Expand Down
Loading