Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
To replicate data from an InfluxDB server to another server, one of the possibilities (in the open source/community edition) is to configure a "subscription". Another blog article describes how to create an InfluxDB asynchronous replication using subscription service.
The current subscriptions can be shown on the master server:
root@influxmaster:~# influx -username admin -password secret
Connected to http://localhost:8086 version 1.6.4
InfluxDB shell version: 1.6.4
> SHOW SUBSCRIPTIONS
name: icinga
retention_policy name mode destinations
---------------- ---- ---- ------------
autogen icinga-replication ALL [http://icinga:pass@influx02:8086]
name: mtr
retention_policy name mode destinations
---------------- ---- ---- ------------
autogen mtr-replication ALL [http://mtr:pass@influx02:8086]
But what if data needs to be replicated across multiple nodes (subscribers)?
The CREATE SUBSCRIPTION command syntax allows to define multiple subscribers as last input, separated with a comma:
CREATE SUBSCRIPTION "sub0" ON "mydb"."autogen" DESTINATIONS ALL|ANY 'http://influx02:8086', 'http://influx03:8086'
Note: Very important to focus on the keywords "DESTINATIONS ALL" or "DESTINATIONS ANY". The ALL keyword stands for data replication to all defined subscribers. ANY stands for round-robin data replication across all defined subscribers!
The problem is that there is no "edit" function available for the subscriptions. Only the SHOW, CREATE and DROP commands are available. This means, the existing replication needs to be removed and then re-created with the additional subscriber.
From the SHOW SUBSCRIPTIONS command we know the replication name "icinga-replication" and the source database "icinga". This subscription can now be dropped and then re-created with the additional subscriber:
> DROP SUBSCRIPTION "icinga-replication" ON "icinga"."autogen"
> CREATE SUBSCRIPTION "icinga-replication" ON "icinga"."autogen" DESTINATIONS ALL 'http://icinga:pass@influx02:8086', 'http://icinga:pass@influx03:8086'
The new subscriber should show up:
> SHOW SUBSCRIPTIONS
name: icinga
retention_policy name mode destinations
---------------- ---- ---- ------------
autogen icinga-replication ALL [http://icinga:pass@influx02:8086 http://icinga:pass@influx03:8086]
If your primary InfluxDB server continuously receives a lot of data, the data received during the DROP and CREATE SUBSCRIPTION commands will be lost. The subscription replication is a fire-and-forget kind of replication. When a subscriber does not respond, the data is not queued and then resent at a later time. The subscription service doesn't "catch up" to a certain moment in time either, such as a classical Master-Slave replication (e.g. MySQL).
To make sure all the InfluxDB servers hold the same data, stop all InfluxDB servers and manually sync /var/lib/influxdb from the master server to the others, then start InfluxDB and the subscription service replicated new incoming data.
Unfortunately InfluxDB 2.0 does not offer subscription replication anymore. From the InfluxDB 1.x to 2.0 notes:
InfluxDB 2.0 has no subscriptions API and does not support Kapacitor stream tasks. To continue using stream tasks, write data directly to both InfluxDB and Kapacitor. Use Telegraf and its InfluxDB output plugin to write to Kapacitor and the InfluxDB v2 output plugin to write to InfluxDB v2.