A witness server is a normal LightDB instance which is not part of the streaming replication cluster; its purpose is, if a failover situation occurs, to provide proof that it is the primary server itself which is unavailable, rather than e.g. a network split between different physical locations.
A typical use case for a witness server is a two-node streaming replication setup, where the primary and standby are in different locations (data centres). By creating a witness server in the same location (data centre) as the primary, if the primary becomes unavailable it's possible for the standby to decide whether it can promote itself without risking a "split brain" scenario: if it can't see either the witness or the primary server, it's likely there's a network-level interruption and it should not promote itself. If it can see the witness but not the primary, this proves there is no network interruption and the primary itself is unavailable, and it can therefore promote itself (and ideally take action to fence the former primary).
Never install a witness server on the same physical host as another node in the replication cluster managed by ltcluster - it's essential the witness is not affected in any way by failure of another node.
For more complex replication scenarios, e.g. with multiple datacentres, it may be preferable to use location-based failover, which ensures that only nodes in the same location as the primary will ever be promotion candidates; see Handling network splits with ltclusterd for more details.
A witness server will only be useful if ltclusterd is in use.
To create a witness server, set up a normal LightDB instance on a server in the same physical location as the cluster's primary server.
This instance should not be on the same physical host as the primary server, as otherwise if the primary server fails due to hardware issues, the witness server will be lost too.
ltcluster 3.3 and earlier provided a ltcluster create witness
command, which would automatically create a LightDB instance. However
this often resulted in an unsatisfactory, hard-to-customise instance.
The witness server should be configured in the same way as a normal ltcluster node; see section Configuration.
Register the witness server with ltcluster witness register. This will create the ltcluster extension on the witness server, and make a copy of the ltcluster metadata.
As the witness server is not part of the replication cluster, further changes to the ltcluster metadata will be synchronised by ltclusterd.
Once the witness server has been configured, ltclusterd should be started.
To unregister a witness server, use ltcluster witness unregister.