Monitoring the Database Backup and Streaming Replication

You can monitor every database backup and replication component running on the server, including:

• Streaming replication
• Base backup
• WAL archive (continuous archive)

Monitoring Database Backup and Replication from the Notifications Pane

To monitor database backup and replication from the NOTIFICATIONS page:

• In the global administration view, select Main > Notifications in the navigation pane.
  The NOTIFICATIONS page is displayed.
  
  Any alerts related to database backup or streaming replication are displayed.

Monitoring Database Backup and Replication from the Server Task Manager

If streaming replication is set up, you can monitor it from the Server Task Manager. The CTERA Portal runs a task every few minutes to verify that replication is working as expected. If any issues are detected, the task fails, and the CTERA Portal displays an appropriate notification in the NOTIFICATIONS page, and also sends an email alert to the portal administrators.

To monitor database backup and replication from the Server Task Manager:

1. In the global administration view, select Main > Servers in the navigation pane.
   The SERVERS page is displayed, listing all the servers for the CTERA Portal.
   
2. Click the replicating secondary server name.
   The server window is displayed with the server name as the window title.
   
3. Click the DB Replication option.
   The portal reports the status of its scheduled base backups and transaction log archiving process, as well as additional metrics to help detect when database replication falls behind due to lags in the process. In the event that replication falls behind, portal administrators are notified via email. The relevant email templates are Replication setup failed and Replication has errors.
   

Monitoring Database Backup and Replication from the Server Console

You can view the status of each database backup and replication component.

To monitor database backup and replication from the server console:

1. Using SSH, log in as root to your CTERA Portal primary database server or secondary database server.

2. Run the following command: portal-storage-util.sh extend_db_archive_pool device

3. Run the following command: portal.sh replication_status | json_reformat
   The output is displayed in JSON format. For example:

   {
       "streaming_replication": {
           "status": "ok",
           "lastSuccess": "256KB"
       },
       "base_backup": {
           "status": "ok",
           "lastSuccess": "none"
       },
       "wal_archive": {
           "status": "failed",
           "lastSuccess": "3 hours"
       }
   }
   

   Both the primary and secondary servers have the same output structure. However, the fields have slightly different meanings.
   
   
   streaming_replication

   Field	Primary Server	Secondary Server
   status	Indicates whether a secondary server is streaming from the primary server: ok – Streaming replication is running. failed – Streaming replication failed. not configured – Streaming replication is not configured on this server.	Indicates whether this server is streaming from the primary server: ok – Streaming replication is running. failed – Streaming replication failed.
   lastSuccess	The difference in size (in KB) between the last sent WAL segment and the secondary server’s last checkpoint, a point in the WAL logs at which all data files have been written to the disk. In other words, the amount of data the primary database sent to the secondary database, which has not been processed by the secondary database.	The difference in size (in KB) between the last received WAL segment and the secondary server’s last checkpoint, a point in the WAL logs at which all data files have been written to the disk. In other words, the amount of data the primary database sent to the secondary database, but which has not been processed by the secondary database. If the value is none then the secondary database is up to date.

   base_backup

   Field	Primary Server	Secondary Server
   status	Indicates whether the last base backup was completed successfully: ok – Last base backup completed successfully. failed – Last base backup failed. not configured – Base backup is not configured for this server. This means that continuous archiving was not configured on the server.	Indicates whether the last base backup was completed successfully: ok – Last base backup completed successfully. failed – Last base backup failed.
   lastSuccess	The last time a successful backup was run (in days). If the value is none, the status field is ok, and the last base backup was completed successfully.	The last time a successful backup was run (in days). If the value is none, the status field is ok, and the last base backup was completed successfully.

   wal_archive

   Field	Primary Server	Secondary Server
   status	Indicates whether a WAL was successfully archived in the past hour: ok – WAL directory was archived successfully in the past hour. failed – WAL directory was not archived in the past hour. This does not indicate a problem. You can change the status to ok, by running a manual command that forces archiving. For more information, contact CTERA support. not configured – WAL archiving, that is continuous archiving, is not configured on this server. Note: Archiving occurs after 16MB of data has been written or during a system restart.	Indicates whether a WAL was successfully synchronized in the past hour: ok – WAL directory was synchronized successfully in the past hour. failed – WAL directory was not synchronized in the past hour.
   lastSuccess	The last time a WAL log was archived successfully (in hours). If the status field's value is ok, then none is output.	The last time a WAL log was archived successfully (in hours). If the status field's value is ok, then none is output.

4. Run the following command on the secondary server to view the secondary archive synchronization log: portal-log.sh replication [-f]
   Include the -f flag to display the log in the command window. Otherwise, the log is displayed in vim.