Restoring databases when you use Horizon

If you use Horizon and need to restore a database, there are steps you should take to avoid causing disruptions in Horizon caused by a data sequence mismatch. These mismatches can prevent data from flowing into the warehouse, and result in service errors.

Horizon uses a producer service to send messages from source databases to a consumer service, which inserts the data into the warehouse. Each message includes a sequence number to ensure proper ordering. 

A mismatch happens when the consumer service receives a message with a lower sequence number than the last one it processed. This causes Horizon to reject the message and stop processing.

Common causes of data sequence mismatch include:

• Database restores on the producer side.
• Data loss or corruption.
• Service interruptions or misconfigurations.

A restore is usually performed as part of disaster recovery, after data corruption, or when rolling back to a previous state due to operational issues.

Before you begin

• Coordinate database restore or backup-related activities with your regional support team. This allows proactive monitoring and faster responses if issues arise.

Steps

If conducting database maintenance:

1. Turn off the producer service before taking a backup. It should remain offline during any restore or related actions.
2. Turn the producer service back on after completing the backup and any restore or related actions.

Resolving a data sequence mismatch

A data sequence mismatch is a possible cause if Horizon stops processing data or appears broken (for example, data doesn't sync or errors appear). 

If you encounter an issue like this, you should contact support. It helps if you can provide us with the following information:

1. Database restore confirmation: was a restore performed on the producer database?
2. Backup age: how old was the backup used for the restore?

This allows us to assess the best way to resolve the issue and assess the risk of data inconsistency.

If we find that a data sequence mismatch is causing an issue, we usually fix it by adjusting the consumer service's message sequence numbers using an internal audit sequence tool to match them to the producer's current sequence.

Important! This method is the fastest way to restore normal operation, but may result in some data loss or inconsistency. We do not manually delete any data, but older data may be overwritten by newer messages. Recovery of overwritten data is not possible.

We will confirm if this resolution is acceptable before we run the tool. If you have any concerns about data integrity or prefer alternative options, please let us know so we can discuss them further. 

See also:

• Horizon overview