Upgrading a Portal from Running on CentOS 7 to Run on CentOS 9

CTERA Portal versions that run on CentOS 7 need to be upgraded to run on CentOS 9.

CTERA is releasing portal versions running on CentOS 9 for the following versions:

• 8.1
• 8.2

To upgrade your portal to CentOS 9 you must first upgrade your portal to the latest 8.1.x CentOS 7 release (8.1.1417.36). You need to upgrade both the image and software.

Upgrading a CentOS 7 Portal Cluster to CentOS 9 By Disk Replacement (ESXi and AWS)

The following procedure for ESXi and AWS replaces the operating system disk running on CentOS 7 with the new operating system disk running on CentOS 9.

You can also upgrade the portal cluster using the failover method for all platforms, including both ESXi and AWS.

Preparing the CTERA Portal to Move to CentOS 9

Before moving the CTERA Portal to CentOS 9, you need to create a migration backup on the existing portal. The migration backup script backs up essential configuration files from the CentOS 7 root disk to the data disk. These files will then be restored to the CentOS 9 root disk at the end of the upgrade.

To create a migration backup:

1. Shutdown all the portal servers.
   First stop preview servers and then application servers. Next stop the replication database server and finally stop the main database server.

   1. Using SSH, log in as root to each CTERA Portal server.
   2. Run the following command: portal-manage.sh stop

   When the server has stopped, the Done message is displayed on the screen.

2. Get the latest migration.sh script from CTERA support.

   Note

   The migration.sh script is often modified to be more efficient. Always use the latest script from CTERA.

3. Copy the script, migration.sh, to $datadir on every VM in the portal cluster.

4. Navigate to $datadir: cd $datadir

5. Apply execute permissions on the script: chmod +x migration.sh

6. On every VM in the portal cluster, run the migration.sh script: bash migration.sh backup

7. Shutdown each VM in the portal cluster.

8. Continue with the procedure for your platform for each VM in the portal cluster in the following order:
   First the main database server followed by the replication database server and then the application servers and finally any preview servers.

   • ESXi
   • AWS

ESXi Platform

1. Log in to vSphere Client Console and using the OVA from CTERA for the CentOS 9 server version 8.1.1417.36.1, create a new server, as described in Installing CTERA Portal Instances under ESXi in the Installing a CTERA Portal article.
2. Make copies of the root disk, a copy for each server VM in the portal cluster. Save each copy to the folder that contains the CentOS 7 root disk for that VM.
   1. Expand Hard disk 1 and note the Disk File location.
      
      Note

      The above screenshot shows the default portal settings with a 16GB root disk and two data disks, 50GB and 60GB. Apart from the root disk, your environment might have a different number of disks and with different sizes.
      

      The disk file location is also displayed in the VM Summary tab, in the Related Objects area.

   2. In Storage, , navigate to the vmdk for Hard disk 1.
      Note

      Select the storage volume and then the Files tab. You can then filter the list of folders using the portal name.

   3. Select the 32GB vmdk disk, 33554432KB, and click COPY TO.
      For example, for an 8.1.x CentOS 9 portal:
      
   4. Select the destination folder for the disk and click OK.
      For example, for an 8.1.x CentOS 7 portal VM that you want to upgrade to CentOS 9:
      
3. Under VMs and Templates, , for each of the CentOS 7 VMs in the cluster, click the Snapshots tab and if there are any snapshots, click DELETE ALL.
4. For each of the CentOS 7 VMs in the cluster, right-click the VM and select Edit Settings.
   The Edit Settings window is displayed.
   
5. Expand Hard disk 1 and note the Virtual Device Node settings: SCSI(0:0)
   
6. Remove Hard disk 1, the 16GB root disk.
   Note

   CTERA recommends not checking Delete files from datastore.

7. Click OK.
8. Click Edit Settings.
9. Click ADD NEW DEVICE and select Existing Hard Disk.
   
10. Select a CentOS 9 root disk that you copied and click OK.
    
11. Expand New Hard disk and make sure that the Virtual Device Node settings are the same as they were for Hard Disk 1: SCSI(0:0)
    
12. Click OK.
    
13. Power on the VM and, using the remote console, log in as root to the VM.
14. Change the default password.
15. Navigate to $datadir: cd $datadir, and run the following script: bash migration.sh restore
    The script restores the backed up files that were saved on the data disk to the new root disk.
16. On the primary database server, run the following query to find the size of the largest index: SELECT relname AS index_name, pg_size_pretty(pg_relation_size(oid)) AS size, pg_relation_size(oid) AS size_bytes FROM pg_class WHERE relkind = 'i' AND relnamespace = (SELECT oid FROM pg_namespace WHERE nspname = 'public') ORDER BY size_bytes DESC LIMIT 1;
    Note

    Make sure that $datadir has at least as much free space as the largest index. Use the following command to see how much space is available: df -h $datadir

17. Start a screen session: screen -S C9-migration
18. Run the following command in the screen console: dmesg -D && sh migration.sh restore |& tee C9-migration.log
    Note

    dmesg -D disables console messages from kernel during the restore and reindex and |& tee C9-migration.log pipes stdout/stderr to a log file.

    The restore process starts and does the following:
    • Disables cloud-init
    • Restores the current network connection
    • Restores any root cron entries
    • Restores any custom NTP settings
    • Restores any custom sudoers entries
    • Restores any customer hosts entries
    • Restores and imports previously imported certificates (Active Directory rejoin)
    • Restores and applies any keytab files (for SSO)
    • Checks and if required, reloads docker images
    • Checks the server role and takes the following action:
      • If the server is the primary database server, restarts the portal service and performs reindexing tasks.
      • If the server is an application server, restarts the portal service and then the script finishes.
19. To validate that the VM runs on CentOS 9, run cat /etc/centos-release to show that the portal is running on CentOS 9.
20. Repeat this procedure for all the VMs in the portal cluster but instead of running the command dmesg -D && sh migration.sh restore |& tee C9-migration.log run dmesg -D && sh migration.sh restore
21. Continue with Ensuring the Base Backup is Successful.

AWS Platform

1. In the AWS Management Console select Services.
2. Under the Compute service, select EC2.
3. In the navigation pane, click Images > AMIs.
   The Amazon Machine Images (AMIs) page is displayed.
4. Note the AMI ID of the CentOS 9 CTERA Portal image that CTERA shared with you.
   
5. Click Instances > Instances and select the CentOS 7 instance.
   Notes

   The instance must be running. If it is Stopped, click Instance state > Start instance.

   The CentOS 7 root disk volume type must be GP2. If the volume type is GP3, stop the CentOS 7 instance, change the volume type to GP2 and then restart the CentOS 7 instance. Refer to AWS documentation to convert a GP3 volume type to GP2.

6. Click Actions > Monitor and troubleshoot > Replace root volume.
   
   The Replace root volume page is displayed.
   
7. Select the Image option under Restore and click in the Image box to display the list of AMIs.
   
8. Select the CentOS 9 AMI image using the AMI ID you previously noted and click Create replacement task.
9. Connect to the VM and log in as root to the VM.
10. Change the default password.
11. Navigate to $datadir: cd $datadir and run the following script: bash migration.sh restore
    The script restores the backed up files that were saved on the data disk to the new root disk.
12. On the primary database server, run the following query to find the size of the largest index: SELECT relname AS index_name, pg_size_pretty(pg_relation_size(oid)) AS size, pg_relation_size(oid) AS size_bytes FROM pg_class WHERE relkind = 'i' AND relnamespace = (SELECT oid FROM pg_namespace WHERE nspname = 'public') ORDER BY size_bytes DESC LIMIT 1;
    Note

    Make sure that $datadir has at least as much free space as the largest index. Use the following command to see how much space is available: df -h $datadir

13. Start a screen session: screen -S C9-migration
14. Run the following command in the screen console: dmesg -D && sh migration.sh restore |& tee C9-migration.log
    Note

    dmesg -D disables console messages from kernel during the restore and reindex and |& tee C9-migration.log pipes stdout/stderr to a log file.

    The restore process starts and does the following:
    • Disables cloud-init
    • Restores the current network connection
    • Restores any root cron entries
    • Restores any custom NTP settings
    • Restores any custom sudoers entries
    • Restores any customer hosts entries
    • Restores and imports previously imported certificates (Portal AD rejoin)
    • Restores and applies any keytab files (for an agent SSO)
    • Checks and if required, reloads docker images
    • Checks the server role and takes the following action:
      • If the server is the primary database server, restarts the portal service and performs reindexing tasks.
      • If the server is an application server, restarts the portal service and then the script finishes.
15. Navigate to /etc/ssh/sshd_config.d/00-aws.conf and comment out the line PermitRootLogin and then run systemctl restart ssh
16. To validate that the VM runs on CentOS 9, run cat /etc/centos-release to show that the portal is running on CentOS 9.
17. Repeat this procedure for all the VMs in the portal cluster but instead of running the command dmesg -D && sh migration.sh restore |& tee C9-migration.log run dmesg -D && sh migration.sh restore
18. Continue with Ensuring the Base Backup is Successful.

Ensuring the Base Backup is Successful

After upgrading to CentOS 9, the base backup might fail. If this happens perform the following:

• Using SSH, log in as root to each server and restart the portal using the command portal-manage.sh restart

Failing Over to Upgrade to CentOS 9

1. Install the same number of portal version 8.1.1417.36.1 servers running on CentOS 9 as you have version 8.1.1417.36 servers running on CentOS 7.
2. If CTERA Messaging Service is enabled:
   1. Log in as root over SSH to the primary database server.
   2. Run the following command to deprovision the messaging servers: messaging-servers deprovision --force
3. Modify the CentOS 7 replication server so that it is no longer a replication server.
4. Connect a CentOS 9 server to the CentOS 7 primary database server and define it as the replication server.
5. Log in as root over SSH to the CentOS 9 replication server.
6. Failover to the CentOS 9 server by running the following command: portal-failover.sh become_master
7. Shut down all the servers in the portal cluster running on CentOS 7.
8. On the new primary database server run the following: platform-services removeserversfromuninstalled <server_name> where server_name is the sever name for each CentOS 7 server running on CentOS 7.
9. On the new primary database server run platform-services rerunlazyinstall --force
10. Connect all the servers running on CentOS 9 to the server running on CentOS 9 that is now the master server.

Set up the CTERA Portal services, such as the CTERA messaging service on the new cluster.