Server Hive Synchronization

In DW Spectrum Systems with more than one (1) Server all Server Databases will form a "Server Hive" and will be synchronized as close to real-time as possible given networking constraints.

What information is synchronized between servers?

• Users & Rights
• Device Configurations (including archive index locations, but not the archive itself)
• Alarm & Event Rules
• Server Settings
• System Settings
• Layout Settings

What information is not synchronized?

• Information which is associated with timeline/archive: media (recorded video/audio), audit trails, event logs.
• Video Archives and Motion Indexes which reside on physical HDD where the media is recorded (NAS, DAS, Local HDD or SSD).

What type of network connections can Servers synchronize across?

Short answer: Synchronization works in ANY network type - LAN, WAN, or Internet. Synchronization will occur as long as all Servers in a Hive are able to see at least one other Server in the Hive. Even in challenging network environments like those outlined below:

One Way Connections

If Server1 can establish a connection to Server2, but Server2 cannot establish a connection to Server1 (e.g. because of a firewall or NAT) a connection can still be established and synchronization can occur.

Server1 → Server2

Indirect Connections

For example, if there are 3 Servers (Server1, Server2, Server3) in a System and Server1 can only establish a connection to Server2, and Server2 can see both Server1 and Server3, then Server1 will remain synchronized with both Server2 and Server3.

Server1 ↔ Server2 ↔ Server3

Long Daisy Chains

In Daisy Chains there are no practical limitations to the chain length. 

Server1 ↔ Server2
     |                |
Server3 ↔ Server4

 

Understanding the DW Spectrum Server Hive Synchronization Algorithm

Below we outline the DW Spectrum Server Hive Synchronization in a bit more detail.

When does Synchronization occur?

1. Any time a change is made by a System User (e.g. a user changes the recording schedule of a single camera using the DW Spectrum Client Desktop) that results in a change to synchronized records.
2. Any time a change in System configurations is made autonomously (e.g. a Server goes offline and Automatic Camera Failover is enacted)

How does Synchronization work?

• The Server generates a Transaction(s) and sends it to all Server(s) connected to the originating Server.
• Each Transaction contains a record of changes.
• Once a server successfully sends the Transaction(s) to other System Server(s) it includes the Transaction IDs of all servers to whom the Transaction was successfully sent (including its own Transaction ID). This guarantees Transaction(s) would not be sent to same Server more than once - which minimizes the amount of time it takes to synchronize all System Servers as well as minimizes the bandwidth needed to synchronize the System.

For Example - in a Daisy Chain network Topology with 4 Servers:

Server1 ---- Server2
     |                 |
Server3 ---- Server4

1. User is connected to Server 1.
2. User enacts a System change (e.g. creates and saves a new Layout)
3. Server 1 generates a Transaction and sends it to Server2 and Server3 with a Transaction ID that includes Server1, Server2, and Server3.
4. Once Server2 or Server3 receives the transaction it would not send a Transaction back to Server1 as the Transaction ID already included Server1 (the Server where the original change was made).
5. Server2 and Server3 will then send a Transaction to Server4.

Potential holes in the scenario above and how they are addressed:

In above scenario there is a little hole. What if while transaction is going from Server1 to Server2 the connection between the two Servers is lost / interrupted? For example:

• Transaction is sent from Server1 to Server2 and Server3 (with Transaction IDs for Server1, Server2, and Server3).
• Transaction is lost between Server1 and Server2 but successfully sent to Server3.
• Transaction is successfully sent from Server3 to Server4 (with Transaction IDs for Server1, Server2, Server3 and Server4).

In this example Server2 never gets the Transaction even though the connection between Server4 and Server2 is still ok because Server 4 receives a Transaction ID that includes Server2.

For this reason we have Keep Alive messages.

What is a Keep Alive message?

• Keep Alive message is sent by each server to each server it's connected to once per timeout period.
• Keep Alive message has Transaction State Log, which contains data about latest transaction that each Server has received.

So in above "hole" scenario, Server2 will send a keep alive packet to Server3. And Server3 will subsequently discover that Server2 has a missing transaction from Server1, triggering Server3 to resent the original Transaction.

FAQ's about Synchronization

What protocol is used during synchronization?

• HTTP (chunk mode) with binary data encrypted by base64 to avoid any firewall issues.

How much data does synchronization create?

In general not much (e.g. bits) but it depends on the network topology the System resides upon.

• If user does not make changes - no synchronization is needed.
• If there is nothing to synchronize (99.999% of the time once a System has been configured) - servers only send Keep Alive packets to each other (a few KB per several seconds).
• If a user does make a change or an autonomous change occurs in the System Servers will begin synchronizing and Transactions will be sent (a few KB per Transaction) every second until the System has been successfully Synchronized)

How long does synchronization take?

• In most cases it synchronization is "instant" - meaning it's nearly imperceptible to human operators. In worst case scenarios synchronization will take a matter of seconds.

Is there any synchronization timeout? Will servers eventually give-up if a connection could not be established to other System Servers? 

• No. System Servers will continue to try to synchronize until successful.