DOC-13648 document magma flusher threads 7.6

modules/learn/pages/buckets-memory-and-storage/storage-engines.adoc

@@ -18,17 +18,20 @@ If you have a small data set that can fit in memory, then you should consider us
 [.edition]#{enterprise}#
 
 Magma is designed for high performance with very large datasets that do not fit in memory.
-It is ideal for use cases that rely primarily on disk access.
+It's ideal for use cases that rely primarily on disk access.
 The performance of disk access will be as good as the underlying disk sub-systems --
 for example, using NVMe SSDs will give higher performance.
 
-In order to get maximum performance from Magma for disk-oriented workloads, it is recommended to set the Writer Threads to `Disk i/o optimized`. This setting will ensure there are enough threads to sustain high write rates.
-
-To learn more about Writer Thread settings, see xref:manage:manage-settings/general-settings.adoc#data-settings[Data Settings]
-
 Magma can work with very low amounts of memory for large datasets: a minimum memory-to-data ratio of 1% is required.
 For example, if a node is holding 5{nbsp}TB of data, Magma can be used with only 64{nbsp}GB RAM.
 
+Magma has several thread settings to tune performance for different workloads.
+To get maximum performance from Magma for disk-oriented workloads, set the Writer Threads to `Disk i/o optimized`. This setting allocates more threads to sustain high write rates.
+See xref:manage:manage-settings/general-settings.adoc#data-settings[Data Settings] for more information about this setting.
+
+You can also tune the number and allocation of threads that Magma uses to compact and flush data to disk. 
+You may choose to change the ratio of threads that compact data to the threads that flush data if you notice that Magma's data compaction spikes CPU use.
+
 
 .Magma Supported Services
 |===

modules/learn/pages/buckets-memory-and-storage/storage-settings.adoc

@@ -31,6 +31,11 @@ For illustrations of how Couchbase Server saves new and updates existing Couchba
 [#threading]
 == Threading
 
+Couchbase Server uses multiple threads when reading an writing data.
+It offers several configuration settings you can change to optimize performance for your workload and hardware.
+
+=== Reader and Writer Threads
+
 Synchronized, multithreaded _readers_ and _writers_ provide simultaneous, high-performance operations for data on disk.
 Conflicts are avoided by assigning each thread (reader or writer) a specific subset of the 1024 vBuckets for each Couchbase bucket.
 
@@ -59,7 +64,35 @@ Again, the maximum thread-allocation that can be specified for each is _64_, the
 Thread-status can be viewed, by means of the [.cmd]`cbstats` command, specified with the [.param]`raw workload` option.
 See xref:cli:cbstats-intro.adoc[cbstats] for information.
 
-For information on using the REST API to manage thread counts, see xref:rest-api:rest-reader-writer-thread-config.adoc[Setting Thread Allocations].
+For information about using the REST API to manage thread counts, see xref:rest-api:rest-reader-writer-thread-config.adoc[Setting Thread Allocations].
+
+[#magma-flushing-and-compaction-threads]
+=== Magma Flushing and Compaction Threads
+
+Couchbase Server compacts the data it writes to disk for xref:learn:buckets-memory-and-storage/storage-engines.adoc#storage-engine-magma[Magma] buckets.
+It allocates a thread pool (containing 20 threads by default) for background compaction and flushing operations for these buckets.
+You can change the number of threads in this pool using the `num_storage_threads` setting of the xref:rest-api:rest-reader-writer-thread-config.adoc[thread allocation REST API].
+
+Two types of threads share the Magma thread pool: compactor threads that compact data and flusher threads that write data to disk.
+By default, Couchbase Server allocates 20% of the threads to flushing data and 80% to compacting data.
+With the default thread pool size and the default flusher allocation, Couchbase Server uses 4 threads to flush data and 16 threads to compact data for Magma buckets.
+You can also change the ratio of flusher to compactor threads using the `magma_flusher_thread_percentage` setting of the xref:rest-api:rest-reader-writer-thread-config.adoc[thread allocation REST API]
+
+For most workloads, the default thread pool size and flusher allocation percentage work well. 
+If you notice CPU use spikes during heavy data mutation workloads, you may want to investigate whether the compactor threads are the cause.
+You can monitor the compaction and flushing activity using the `kv_magma_compactions` metric.
+This metric counts the number of compactions Couchbase Server has performed.
+You can view this metric via the xref:rest-api:rest-statistics.adoc[] REST API or through Prometheus if you have configured it to collect Couchbase Server metrics.
+See xref:manage:monitor/set-up-prometheus-for-monitoring.adoc[] for more information about using Prometheus with Couchbase Server.
+
+If you see the CPU use of the `memcached` Linux process on your nodes spike while the `kv_magma_compactions` count is increasing, the compactor threads may be the cause of the spike. 
+In this case, you may want to reduce the number of compactor threads by increasing the percentage of flusher threads.
+This reduction limits the compactor's ability to spike CPU use.
+However, reducing the number of compactor threads may lead to higher latency before Couchbase Server writes data to disk. 
+
+NOTE: The number of threads in the compactor and flusher pool and the percentages of flusher threads are advanced settings.
+Contact Couchbase Support before making changes to them.
+Support can help you determine the best settings for your workload and hardware.
 
 [#deletion]
 == Deletion

modules/rest-api/pages/rest-reader-writer-thread-config.adoc

@@ -1,82 +1,237 @@
 = Setting Thread Allocations
-:description: The number of threads allocated per node is configurable.
+:description: Couchbase Server has several settings that let you change how it allocates and uses threads for the entire cluster.
 :page-topic-type: reference
 
 [abstract]
 {description}
 
 == HTTP method and URI
 
+.Get Global Thread Settings
+[source,url]
+----
+GET /pools/default/settings/memcached/global
+----
+
+.Set Global Thread Allocations
+[source]
 ----
 POST /pools/default/settings/memcached/global
 ----
 
 [#description]
 == Description
 
-Couchbase Server allows the number of threads allocated per node for _reading_, _writing_, and for _NonIO_, and _AuxIO_ thread-pools to be configured by the administrator.
+Couchbase Server lets you set the number of threads per node for reading, writing, NonIO, and AuxIO thread pools.
+You can also change the number of threads that Magma buckets use for writing data to disk.
+Increasing thread allocation can improve performance on systems with a large number of CPU cores.
+For example, more writer threads can optimize durable write performance.
+See xref:learn:data/durability.adoc[Durability] for details.
+However, allocating too many threads can reduce performance on nodes with limited resources.
+Test any changes to thread allocation before applying them to production systems.
+
+
+== Getting Current Thread Settings
+
+To get the current global thread settings for Couchbase Server, use the `GET /pools/default/settings/memcached/global` endpoint.
+
+[#get-curl-syntax]
+=== curl Syntax
+
+[source,bash]
+----
+curl -X GET http[s]://{HOST}:{PORT}/pools/default/settings/memcached/global
+  -u $USER:$PASSWORD
+----
+
+
+==== Path Parameters
+:required-privileges: required-privileges-get
+include::partial$user_pwd_host_port_params.adoc[]
+
+[#required-privileges-get]
+=== Required Privileges
+
+Your account must have at least 1 of the following roles to make a GET request from this endpoint:
+
+* xref:learn:security/roles.adoc#full-admin[Full Admin]
+* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin]
+* xref:learn:security/roles.adoc#external-user-security-admin[External User Security Admin]
+* xref:learn:security/roles.adoc#local-user-security-admin[Local User Security Admin]
+
+=== Responses
+
+200 OK::
+The request was successful.
+Returns a JSON object containing the current global thread settings that have been set explicitly.
+
+400 Bad Request::
+The request could not be understood.
+Also returns a JSON object containing an error message.
+
+=== Example
+
+The following example shows how to get the current global thread settings for Couchbase Server:
+
+[source,bash]
+----
+curl -X GET -u Administrator:password \
+http://localhost:8091/pools/default/settings/memcached/global | jq
+----
+
+If successful, the call returns a JSON object containing the any overrides to the default global thread settings:
+
+[source,json]
+----
+{
+  "magma_flusher_thread_percentage": 30,
+  "num_storage_threads": 25
+}
+----
 
-A high thread-allocation may improve performance on systems whose hardware-resources are commensurately supportive (for example, where the number of CPU cores is high).
-In particular, a high number of _writer_ threads on such systems may significantly optimize the performance of _durable writes_: see xref:learn:data/durability.adoc[Durability], for information.
+== Setting Global Thread Allocations
 
-Note, however, that a high thread-allocation might _impair_ some aspects of performance on less appropriately resourced nodes.
-Consequently, changes to the default thread-allocation should not be made to production systems without prior testing.
+To set the global thread allocations for Couchbase Server, perform a `POST` ro the `/pools/default/settings/memcached/global` endpoint.
 
 [#curl-syntax]
-== Curl Syntax
+=== curl Syntax
 
+[source,bash]
 ----
-curl -X POST -d hostname=<ip-address-or-hostname>:8091
-  -d num_reader_threads=<int>
-  -d num_writer_threads=<int>
-  -d num_nonio_threads=<int>
-  -d num_auxio_threads=<int>
-  -d password=<password>
-  -u <administrator>:<password>
-  http://<host>:<port>/pools/default/settings/memcached/global
+curl -X POST http[s]://{HOST}:{PORT}/pools/default/settings/memcached/global
+  [-d num_reader_threads=<integer>]
+  [-d num_writer_threads=<integer>]
+  [-d num_nonio_threads=<integer>]
+  [-d num_auxio_threads=<integer>]
+  [-d num_storage_threads=<integer>]
+  [-d magma_flusher_thread_percentage=<integer>]
+  -u $USER:$PASSWORD
 ----
 
-Each key is specified optionally.
+
+
+==== Path Parameters
+:required-privileges: required-privileges-put
+include::partial$user_pwd_host_port_params.adoc[]
+
+==== Parameters
+
+`num_reader_threads`:: 
+(optional) Sets the number threads Couchbase Server uses to read data.
+
+`num_writer_threads`:: 
+(optional) Sets the number threads Couchbase Server uses to write data.
+
+`num_nonio_threads`:: 
+(optional) Sets the number of NonIO threads to be used by Couchbase Server.
+The NonIO thread pool runs in-memory tasks.
+For example, the durability timeout task uses these threads.
+Valid values are:
 
-The keys `num_reader_threads` and `num_writer_threads` specify the number of threads that are to be used for _reading_ and _writing_, respectively.
++
+* An integer from `1` to `64`: explicitly sets the number of threads.
+* `default`: Couchbase Server calculates and applies an appropriate number of threads.
 
-The keys `num_nonio_threads` and `num_auxio_threads` specify the number of threads that are to be used for _NonIO_ and _AuxIO_ thread pools, respectively.
-Note that:
+`num_auxio_threads`:: 
+(optional) Sets the number of AuxIO threads to be used by Couchbase Server.
+The AuxIO thread pool runs auxiliary I/O tasks -- for example, the access log task.
+Valid values are:
 
-* The _NonIO_ thread pool is used to run _in memory_ tasks -- for example, the _durability timeout_ task.
++
+* An integer from `1` to `64`: explicitly sets the number of threads.
+* `default`: Couchbase Server determines the number of threads.
 
-* The _AuxIO_ thread pool is used to run _auxiliary I/O_ tasks -- for example, the _access log_ task.
+`num_storage_threads`::
+(optional) Sets the number of threads the Magma storage engine can use when compacting and writing data to disk.
+This thread pool is divided into 2 types of threads: flusher threads and compactor threads.
+See xref:learn:buckets-memory-and-storage/storage-settings.adoc#magma-flushing-and-compaction-threads[Magma Flushing and Compaction Threads] for more information.
+The default value is `20`.
+
+`magma_flusher_thread_percentage`::
+(optional) Sets the percentage of Magma storage engine threads to allocate to flushing compacted data to disk.
+The remaining threads in the thread pool are allocated to compacting data.
+For example, if you set `num_storage_threads` to `20` and set `magma_flusher_thread_percentage` to `50`, Couchbase Server uses `10` threads to flush data and `10` threads to compact data.
+Valid values for this setting are integers from `0` to `100`.
+The default setting is `20` which allocates 20% of threads to flushing data.
+
+NOTE: `num_storage_threads` and `magma_flusher_thread_percentage` are advanced settings.
+Contact Couchbase Support before making changes to them.
+Support can help you determine the best settings for your workload and hardware.
+
+[#required-privileges-put]
+=== Required Privileges
+
+Your account must have at least 1 of the following roles to make a POST request to this endpoint:
+
+* xref:learn:security/roles.adoc#full-admin[Full Admin]
+* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin]
 
-In all cases, the value of `int` should be an integer between `1` and `64`, inclusive.
-If in any case the value `default` is specified (for example, `-d num_reader_threads=default`), Couchbase Server itself calculates and applies an appropriate number of threads.
 
 [#responses]
-== Responses
+=== Responses
+
+202 Accepted::
+The request was successful.
+This result also returns a JSON object containing all non-default thread settings.
+
+400 Bad Request::
+The request had errors or invalid values.
+This result also returns a JSON object containing an error message.
+
 Success returns an object whose values confirm the settings that have been made.
 
 [#examples]
-== Examples
+=== Examples
 
 To set the numbers of reader, writer, NonIO, and AuxIO threads for Couchbase Server, use the `POST /pools/default/settings/memcached/global` http method and endpoint as follows:
 
+[source,bash]
 ----
-curl -v -X POST -u Administrator:password \
-http://10.144.220.101:8091/pools/default/settings/memcached/global \
+curl -X POST -u Administrator:password \
+http://localhost:8091/pools/default/settings/memcached/global \
 -d num_reader_threads=12 \
 -d num_writer_threads=8 \
 -d num_nonio_threads=6 \
--d num_auxio_threads=6
+-d num_auxio_threads=6 | jq
+----
+
+This sets the number of `reader` threads to `12`, the number of `writer` threads to `8`, and the numbers of `NonIO` and `AuxIO` threads each to `6`.
+If successful, the call returns an object that confirms the settings you specified:
+
+[source,json]
+----
+{
+  "num_reader_threads": 12,
+  "num_writer_threads": 8,
+  "num_auxio_threads": 6,
+  "num_nonio_threads": 6
+}
 ----
 
-This sets the number of _reader_ threads to `12`, the number of _writer_ threads to `8`, and the numbers of _NonIO_ and _AuxIO_ threads each to `6`.
-If successful, the call returns an object whose values confirm the settings that have been made:
+The following example increases the Magma storage engine's thread pool to 30 threads and allocates 25% of the threads to flushing data to disk:
 
+[source,bash]
 ----
-{"num_reader_threads":12,"num_writer_threads":8,"num_auxio_threads":6,"num_nonio_threads":6}
+curl -X POST -u Administrator:password \
+     http://localhost:8091/pools/default/settings/memcached/global \
+     -d num_storage_threads=30 \
+     -d magma_flusher_thread_percentage=25 | jq
 ----
 
+If successful, the call returns an object confirming your new settings:
+
+[source,json]
+----
+{
+  "magma_flusher_thread_percentage": 25,
+  "num_storage_threads": 30
+}
+----
+
+
+
 [#see-also]
 == See Also
 
-See xref:learn:data/durability.adoc[Durability], for information on durable writes.
-See xref:learn:buckets-memory-and-storage/storage-settings.adoc#threading[Threading] for an overview of threads.
+* See xref:learn:data/durability.adoc[Durability], for information about durable writes.
+* See xref:learn:buckets-memory-and-storage/storage-settings.adoc#threading[Threading] for an overview of threads.

modules/rest-api/partials/user_pwd_host_port_params.adoc

@@ -0,0 +1,15 @@
+// Be sure to set the 'required-priviledges' attribute before including this partial.
+// It must be the anchor to the Required Privileges section for the specific endpoint.
+`HOST`::
+Host name or IP address of a Couchbase Server.
+
+`PORT`::
+Port number for the REST API. 
+Defaults are 8091 for unencrypted and 18901 for encrypted connections.
+
+`USER`::
+
+The name of a user who has at least 1 of the roles listed in xref:{required-priviledges}[Required Privileges].
+
+`PASSWORD`::
+The password for the user.