Server Hive Synchronization

-----------------------------------

Affected Roles: Administrator, Power Users

Related Digital Watchdog VMS Apps: DW Spectrum

Software Version: 6.0

Last Edit: April 4, 2025

-----------------------------------

Server Synchronization

In DW Spectrum Systems with more than one (1) server, all Server Databases will form a “Server Hive” and will be synchronized as close to real-time as possible with the given networking constraints.

What is Synchronized Between Servers?

Users & Rights
Device Configurations (including archive index locations, but not the archive itself)
Alarm & Event Rules
Server Settings
Site Settings
Layout Settings

What is Not Synchronized Between Servers?

Information which is associated with timeline/archive: media (recorded video/audio), audit trails, event logs.
Video Archives and Motion Indexes which reside on physical HDD where the media is recorded (NAS, DAS, Local HDD or SSD).

What Type of Network Connections Can Servers Synchronize Across?

Synchronization works in ANY network type - LAN, WAN, or Internet. Synchronization will occur if all Servers in a Hive are able to communicate with at least one other Server in the Hive. Even in challenging network environments like those outlined below.

One-Way Connections

If Server-1 can establish a connection to Server-2, but Server-2 cannot establish a connection to Server-1 (e.g. because of a Firewall or NAT), a connection can still be established and synchronization can occur.

Server-1 → Server-2

Indirect Connections

For example, if there are 3 servers (Server-1, Server-2, Server-3) in a System and Server-1 can only establish a connection to Server-2, while Server-2 can see both Server-1 and Server-3, then Server-1 will remain synchronized with both Server-2 and Server-3.

Server-1 ↔ Server-2 ↔ Server-3

Long Daisy Chains

In a Daisy Chain setup, there are no practical limitations to the chain length.

Server-1 ↔ Server-2
| |
Server-3 ↔ Server-4

Understanding the Hive Synchronization Algorithm

Below we outline the DW Spectrum Server Hive Synchronization.

When Does Synchronization Occur?

Any time a change is made by a System User (e.g. a user changes the recording schedule of a single camera using the DW Spectrum Client) that results in a change to synchronized records.
Any time a change in System configurations is made autonomously (e.g. a Server goes offline, and Automatic Camera Failover is enacted).

How Does Synchronization Work?

The Server generates a Transaction(s) and sends it to all Server(s) connected to the originating Server.
Each Transaction contains a record of changes.
Once a server successfully sends the Transaction(s) to other System Server(s) it includes the Transaction IDs of all servers to whom the Transaction was successfully sent (including its own Transaction ID). This guarantees Transaction(s) would not be sent to same Server more than once - which minimizes the amount of time it takes to synchronize all System Servers as well as minimizes the bandwidth needed to synchronize the System.

For Example - in a Daisy Chain network Topology with 4 Servers:

Server-1 ↔ Server-2
| |
Server-3 ↔ Server-4

User is connected to Server-1.
User enacts a System change (e.g. creates and saves a new Layout)
Server-1 generates a Transaction and sends it to Server-2 and Server-3 with a Transaction ID that includes Server-1, Server-2, and Server-3.
Once Server-2 or Server-3 receives the transaction it would not send a Transaction back to Server-1 as the Transaction ID already included Server-1 (the Server where the original change was made).
Server-2 and Server-3 will then send a Transaction to Server-4.

Potential holes in the scenario above and how they are addressed:

In the above scenario, what if the connection between the two Servers is lost/interrupted while a transaction is going from Server-1 to Server-2?

For example:

Transaction is sent from Server-1 to Server-2 and Server-3 (with Transaction IDs for Server-1, Server-2, and Server-3).
Transaction is lost between Server-1 and Server-2 but successfully sent to Server-3.
Transaction is successfully sent from Server-3 to Server-4 (with Transaction IDs for Server-1, Server-2, Server-3 and Server-4).

In this example, Server-2 never gets the Transaction even though the connection between Server-4 and Server-2 is still accepted, because Server 4 receives a Transaction ID that includes Server-2.

For this reason, we have “Keep Alive” messages.

What is a “Keep Alive” message?

A Keep Alive message is sent by each server to each server it's connected to once per timeout period.
A Keep Alive message has Transaction State Log, which contains data about the latest transaction that each Server has received.

In the above "hole" scenario, Server-2 will send a keep alive packet to Server-3. And Server-3 will subsequently discover that Server-2 has a missing transaction from Server-1, triggering Server-3 to resend the original Transaction.

FAQs about Synchronization

What Protocol is Used During Synchronization?

HTTP (chunk mode) with binary data encrypted by base 64 to avoid any firewall issues.

How Much Data Does Synchronization Use?

In general, not much (e.g. bits) but it depends on the network topology the Site resides upon.

If the user does not make changes - no synchronization is needed.
If there is nothing to synchronize (most of the time, once a Site has been configured) - Servers only send Keep Alive packets to each other (a few kilobytes per several seconds).
If a user does make a change or an autonomous change occurs in the Site, Servers will begin synchronizing and Transactions will be sent (a few kilobytes per Transaction) every second until the System has been successfully Synchronized.

How Long Does Synchronization Take?

In most cases, the synchronization is “instant” – meaning it’s imperceptible to human operators. In worst case scenarios, synchronization will only take seconds to occur.

Is There a Synchronization Timeout?

No, Servers will continuously attempt to synchronize until successful.

______________________________________________________________________________

For More Information or Technical Support

DW Technical Support: https://www.digital-watchdog.com/contact-tech-support/

DW Sales: sales@digital-watchdog.com | www.digital-watchdog.com