Liveliness Check between UPF and RCM using Heartbeat Communication

The application-level heartbeat mechanism between UPF and RCM allows you to monitor the liveliness of the TCP connection. This feature will resolve the half-closed TCP connections between RCM checkpoint managers and UP session managers.

RCM sends a heartbeat message every 3 seconds. It checks if it has received the heartbeat message from UPF in the last 60 seconds. RCM will close the TCP connection if it has not received the message. This behavior is applicable for both Active UP to RCM and RCM to Standby UP communication.

UPF also behaves similarly where it sends a heartbeat message every 3 seconds. UPF checks if it has received the message from RCM in the last 60 seconds. If UPF has not received the message, then it will close the TCP connection.

The heartbeat functionality is configurable on RCM and UPF using the following commands:

• RCM—k8 smf profile rcm-config-ep enable-up-heartbeat { true | false } in Config mode

• UPF—up-sm-heartbeat { enable | disable } in Redundancy Configuration Module mode

The following show commands on RCM and UPF display the total number of heartbeat messages received and sent:

• RCM—rcm show-statistics checkpointmgr-endpointstats

• UPF—show rcm checkpoint statistics sessmgr all

Default Setting: Disabled – Configuration required to enable

IP Pool Audit Timer Behavior

UPF and RCM support a configurable timer to initiate IP pool audit periodically.

Previous Behavior: When the send-ip-pool-audit CLI in Exec mode was configured, UPF sent the IP pool audit messages to RCM only.

New Behavior: If you configure the send-ip-pool-audit command in Exec mode, UPF will also stop and restart any running audit timer in addition to sending the audit message to RCM. This behavior maintains the periodical audit of configured IP pool chunks.

Sending IP Pool Audit Message Periodically

Previous Behavior: RCM did not support the timer to initiate IP pool audit periodically.

New Behavior: RCM supports the functionality to trigger the IP pool audit message manually using the send-ip-pool-audit CLI in Exec mode and to audit the IP pool information periodically with a timer using the ip-pool-audit interval audit_timer CLI in RCM Service Configuration mode.

• Using the send-ip-pool-audit CLI command in Exec mode, RCM sends the IP pool audit message.

• Using the [ no ] ip-pool-audit interval audit_timer CLI command in the RCM Service Configuration mode, UPF sets a time interval towards RCM controller between successive audits. audit_timer is the timer value ranging from 900 to 43200 seconds.

  The default value of the timer is 1 hour.

  The no ip-pool-audit interval command disables the timer.

Socket Write Timeout Configuration from RCM Controller to UPF

Previous Behavior: RCM did not support the I/O timeout configuration for TCP connection between RCM Controller and UPF.

New Behavior: RCM supports a configurable I/O timeout for TCP socket from RCM controller to UPF.

The following CLI command is introduced in the Configuration mode to configure write timeout:

write_timeout_value must be an integer ranging from 300 to 60000 milliseconds. The default value is 1000 milliseconds.

Customer Impact: The TCP socket from RCM controller towards UPF could get blocked. This CLI can determine the timeout if the TCP socket gets blocked. If an error such as I/O timeout is seen in the controller logs, you can try increasing the timeout value. You can configure the write timeout value based on your tolerance requirements.

Socket Write Timeout Configuration from RCM CheckpointMgr to UPF

Previous Behavior: RCM did not support the I/O timeout configuration for TCP connection between RCM CheckpointMgr and standby UPF SessMgr.

New Behavior: RCM supports a configurable I/O timeout for TCP socket from RCM CheckpointMgr to UPF. The default value of the write timeout between RCM CheckpointMgr and standby UP SessMgr is 10 seconds.

The following CLI command is introduced in the Configuration mode to configure the write deadline timeout during switchover:

The default value is 10000 milliseconds.

Customer Impact: The TCP socket from RCM CheckpointMgr towards UPF SessMgr could get blocked. This CLI can determine the timeout if the TCP socket gets blocked. If an error such as I/O timeout is seen in the CheckpointMgr logs, you can try increasing the timeout value. You can configure the write timeout value based on your tolerance requirements.

TCP Connectivity between Backup RCM and UPF

The RCM CheckpointMgr and ConfigMgr pods will be restarted in the following scenarios:

• When RCM is brought up with system mode running

• When RCM keepalived pod restarts due to configuration or operation change

• When one MASTER RCM moves to BACKUP if dual MASTER resolution is done by VRRP

• In CNDP deployment, the pods will be restarted twice in the new BACKUP RCM after RCM HA migration—in the FAULT state first and then in BACKUP state

• In VM deployment, the pods will be restarted after booting the RCM VM

Previous Behavior: When RCM moved to the BACKUP state directly from the MASTER state, the CheckpointMgrs were not notified and not restarted.

Using the k8 smf profile rcm-keepalived-ep vrrp-config group CLI, RCM supported default values for the following commands:

• tuning-params script-interval —30

• tuning-params fall—5

New Behavior: When the Keepalived pod moves to the BACKUP state from any state or no state (such as RCM startup), it will restart the ConfigMgr pod and all CheckpointMgr pods.

Using the k8 smf profile rcm-keepalived-ep vrrp-config group CLI, RCM supports new default values for the following commands:

• tuning-params script-interval —40

• tuning-params fall—20

Customer Impact: To prevent stray TCP connections between backup RCM and UPFs, some backup RCM pods are restarted when RCM Keepalived moves to the Backup state.