Portable Radios and Vehicle Tracking Unavailable After IMW Server Failover

Description

Portable radios and vehicle tracking become unavailable following a failover of IMW servers. The root cause is a misconfigured config.yaml in the UHF containers, where the failed-over server has insufficient retry settings, preventing connection from being established.

Symptoms

Portable radios not visible in UA.
Vehicle tracking not visible in UA.
Only HF remains visible in UA.
UHF crawler logs show connection errors similar to:

"message": "[Presence endpoint] Exception caught in connect_to_servers for extra data:
HTTPSConnectionPool(host='<IMW_HOST>', port=9031): Max retries exceeded with url:
/as/token.oauth2?grant_type=client_credentials&... (Caused by NewConnectionError(
'Failed to establish a new connection: [Errno 111] Connection refused'))"

Root Cause

When IMW servers fail over (e.g., from Brindisi to Valencia), the config.yaml of the UHF containers may still have:

Higher priority set for the now-unreachable servers (e.g., Brindisi priority: 0, Valencia priority: 1), meaning the active servers are deprioritised.
number_retries: 0 set for the new active servers (e.g., Valencia), causing the connection attempt to fail immediately with no retries.

Both conditions must be corrected — fixing priority alone is not sufficient.

Affected Components

UHF containers (Zone 1, Zone 2, Zone 3 — node 02)
IMW servers (config.yaml settings)

Solution

1. Edit `config.yaml` for Zone 1 and Zone 2 containers

Update the server entries so that the currently active servers (Valencia) have the highest priority and at least one retry:

imw_servers:
  - token_server_host: '10.130.211.134'   # Brindisi
    token_server_port: 9031
    ...
    number_retries: 1
    priority: 1                            # Lower priority (higher number = lower priority)

  - token_server_host: '10.130.81.134'    # Valencia
    token_server_port: 9031
    ...
    number_retries: 1                      # Must be >= 1
    priority: 0                            # Higher priority (lower number = higher priority)

Key changes:

Set Valencia priority: 0 (highest priority).
Set Brindisi priority: 1 (lower priority).
Set number_retries: 1 (or higher) for the Valencia servers.

2. Edit `config.yaml` for Zone 3 — node 02 only

Zone 3 on node 01 was unaffected. Zone 3 on node 02 had correct priority but number_retries: 0.

Set number_retries: 1 for the Valencia servers.

3. Restart the containers

After saving changes, restart the affected UHF containers.

4. Verify

Check the logs to confirm the containers reconnect successfully and that radios and vehicle tracking are visible in UA.

Verification Commands

Use telnet to confirm connectivity to IMW server endpoints before and after the fix:

telnet 10.130.211.134 9031    # Brindisi token server
telnet 10.130.81.134  9031    # Valencia token server
telnet 10.130.211.134 65001   # Brindisi main server

Notes

This issue was first identified on 03-04-2025 and resolved on 04-04-2025.
Zone 3 node 01 was working correctly throughout and required no changes.
For IMW server issues, contact: Milosz (TBC).

SR: IM-1-12380361146
WO: WO-IM-1-12381236334