Edit page

Portable Radios and Vehicle Tracking Unavailable After IMW Server Failover

Title

Slug

Status

Context

Categories

Topics

Markdown

## Description

Portable radios and vehicle tracking become unavailable following a failover of IMW servers. The root cause is a misconfigured `config.yaml` in the UHF containers, where the failed-over server has insufficient retry settings, preventing connection from being established.

***

## Symptoms

* Portable radios not visible in UA.
* Vehicle tracking not visible in UA.
* Only HF remains visible in UA.
* UHF crawler logs show connection errors similar to:

```
"message": "[Presence endpoint] Exception caught in connect_to_servers for extra data:
HTTPSConnectionPool(host='<IMW_HOST>', port=9031): Max retries exceeded with url:
/as/token.oauth2?grant_type=client_credentials&... (Caused by NewConnectionError(
'Failed to establish a new connection: [Errno 111] Connection refused'))"
```

***

## Root Cause

When IMW servers fail over (e.g., from Brindisi to Valencia), the `config.yaml` of the UHF containers may still have:

1. Higher priority set for the now-unreachable servers (e.g., Brindisi `priority: 0`, Valencia `priority: 1`), meaning the active servers are deprioritised.
2. `number_retries: 0` set for the new active servers (e.g., Valencia), causing the connection attempt to fail immediately with no retries.

Both conditions must be corrected — fixing priority alone is not sufficient.

***

## Affected Components

* UHF containers (Zone 1, Zone 2, Zone 3 — node 02)
* IMW servers (`config.yaml` settings)

***

## Solution

### 1\. Edit `config.yaml` for Zone 1 and Zone 2 containers

Update the server entries so that the currently active servers (Valencia) have the highest priority and at least one retry:

```
imw_servers:
  - token_server_host: '10.130.211.134'   # Brindisi
    token_server_port: 9031
    ...
    number_retries: 1
    priority: 1                            # Lower priority (higher number = lower priority)

- token_server_host: '10.130.81.134'    # Valencia
    token_server_port: 9031
    ...
    number_retries: 1                      # Must be >= 1
    priority: 0                            # Higher priority (lower number = higher priority)
```

Key changes:

* Set Valencia `priority: 0` (highest priority).
* Set Brindisi `priority: 1` (lower priority).
* Set `number_retries: 1` (or higher) for the Valencia servers.

### 2\. Edit `config.yaml` for Zone 3 — node 02 only

> Zone 3 on node 01 was unaffected. Zone 3 on node 02 had correct priority but `number_retries: 0`.

* Set `number_retries: 1` for the Valencia servers.

### 3\. Restart the containers

After saving changes, restart the affected UHF containers.

### 4\. Verify

Check the logs to confirm the containers reconnect successfully and that radios and vehicle tracking are visible in UA.

***

## Verification Commands

Use `telnet` to confirm connectivity to IMW server endpoints before and after the fix:

```
telnet 10.130.211.134 9031    # Brindisi token server
telnet 10.130.81.134  9031    # Valencia token server
telnet 10.130.211.134 65001   # Brindisi main server
```

***

## Notes

* This issue was first identified on **03-04-2025** and resolved on **04-04-2025**.
* Zone 3 node 01 was working correctly throughout and required no changes.
* For IMW server issues, contact: **Milosz** (TBC).

***

## Related

* SR: `IM-1-12380361146`
* WO: `WO-IM-1-12381236334`

Cancel