Edit page

OSM — OpenStreetMap Platform

Title

Slug

Status

Context

Categories

Topics

Markdown

# OSM — OpenStreetMap Platform

## iNeed Asset Name
Open Street Map Server (ImpOSM)

## What it is
OpenStreetMap data platform. ImpOSM is the import/processing component.
Managed by Zeeshan's team (GIS PM). Carlos handles DB/infra support.

## Servers (PROD, dpko.un.org, RHEL 7)
- dfs-imposm-01 — ImpOSM import server
- dfs-osmpgsql-01, 02, 03 — PostgreSQL DB servers
- GIS servers: dfs-gisosm-01 (PROD), dfs-gisosm-11 (PRE)
- GIS DB: dfs-gispgsql-02 (PROD), dfs-gispgsql-12 (PRE)

## Active Tasks
- OSM data migration - pending tasks (overdue Mar 19)
- Migration to Proxmox - Prepare OSM data server migration (overdue Mar 20)
- OSM Monthly patches - Consolidate procedure (due Mar 31)
- OSM Maps Batch 5A (due Mar 31)

## Projects
- New OSM updates procedure - Consolidate and execute (no deadline)
- Migration to Proxmox - Phase 2 (overdue Mar 6)

## FW Rules Issue (Current)
FW rules requested for OSM server access got stuck in security approval with Antonio Candelaria.
Zeeshan spoke to Antonio. Carlos to debrief with Zeeshan (call due this week).

## Knowledge Gaps
- What does the monthly patch procedure involve step by step?
- What does OSM Batch 5A mean?
- What is the OSM data migration pending — what data?

---

## Infrastructure Detail — unosm-r-app-01

*Compiled 2026-03-31*

### Server Overview
| Property | Value |
|---|---|
| Hostname | unosm-r-app-01 |
| OS | RHEL (Red Hat Enterprise Linux) |
| Role | OSM data processing |

### Filesystem Layout
| Mount | Size | Notes |
|---|---|---|
| `/` | 53G | Root filesystem |
| `/var` | 15G | Logs and system data |
| `/data` | 1000G | Main data volume |
| `/boot` | 994M | Boot partition |
| `/tmp` | 63G | RAM-backed (tmpfs) |

**Note:** Application data (imposm diffs, service account home dirs) lives under `/home` on root, NOT on `/data`.

### Service Accounts
| Account | Purpose |
|---|---|
| `_osmprod` | Main OSM production — runs imposm |
| `_osmstat` | OSM statistics processing |
| `_osm2pgsql` | osm2pgsql bulk imports |
| `adm.pfrizzer` | Admin user (UN domain) |

### Application Stack

**Imposm**
- Location: `/home/_osmprod/imposm/`
- Versions: 0.11.1, 0.14.0, 0.14.2
- Function: keeps PostGIS DB in sync with OSM planet via daily replication diffs
- Replication source: `https://planet.openstreetmap.org/replication/day/`
- Diff storage: `/home/_osmprod/imposm/diffs/osm_planet/` and `osm_planet_buildings/`
- State file: `/home/_osmprod/imposm/diffs/osm_planet/last.state.txt`

**Imposm downloader behaviour:**
- Uses parallel workers; temp files named `SEQ.osc.gz~ID` during download
- On success: renamed to final filename, temp files deleted automatically
- On interruption: orphaned temp files left behind — must be cleaned up manually
- Also contains LevelDB caches: `nodes/`, `ways/`, `relations/`, `coords_rel_index/`, `ways_index/`

**osm2pgsql**
- Service account: `_osm2pgsql`
- Handles bulk imports into PostGIS

**osmosis 0.49.2** — present under imposm bin dir

### Installed Agents
| Tool | Purpose |
|---|---|
| Elastic Agent (`/opt/Elastic/Agent`) | Monitoring + log/metric forwarding |
| Nessus Agent (`/opt/nessus_agent`) | Vulnerability scanning |
| filebeat | Additional log shipping |
| GCC Toolsets 12/13/14 | Build toolchains (compiled software present) |

### Security & Identity
- Domain-joined to `global.un.org` via SSSD
- auditd running (audit logging active)
- filebeat + Elastic Agent for log shipping

---

## UNOSM Architecture (detailed reference)

### Mental Model

* **Write brain:** unosm-r-db-01
* **Read helpers:** unosm-r-db-02, unosm-r-db-03
* **Importer:** unosm-r-app-01
* **User entry points:** unosm-r-app-02 (Brindisi), unosm-p-app-02 (Valencia)
* **Traffic cop:** NetScaler → osmpgsql.dfs.un.org
* **Exporter:** unosm-r-data-01

***

### Database Layer (PostgreSQL)

#### Master

* **unosm-r-db-01** (Brindisi)
* Handles **all writes**: INSERT, UPDATE, DELETE
* Manages transactions locally
* Generates **WAL logs**

#### Slaves

* **unosm-r-db-02** (Brindisi)
* **unosm-r-db-03** (Valencia)
* **Read-only replicas**
* Receive WAL logs **asynchronously**
* Can be slightly behind the master

#### Replication Model

* Asynchronous replication
* Master commits **without waiting** for slaves
* Faster writes, possible replication lag

#### Failure Behavior

* No automatic failover
* If **unosm-r-db-01** fails:
    * **Manual promotion** of unosm-r-db-02 or unosm-r-db-03 required

***

### Application Layer

#### Data Ingestion

* **unosm-r-app-01**
* Imports OpenStreetMap data using **Imposm**
* Feeds data into unosm-r-db-01

#### Client Access

* **unosm-r-app-02** (Brindisi)
* **unosm-p-app-02** (Valencia)
* Receive user requests
* Distribute read queries to database slaves

***

### Load Balancing & Traffic Flow

#### Global Load Balancer

* **NetScaler**
* Public endpoint: `osmpgsql.dfs.un.org`
* Routes traffic based on:
    * Client location
    * Site load
    * Performance metrics

#### Current Behavior

* Active load balancing mainly **within Brindisi**
* Valencia is available but less actively used

***

### Export & Supporting Services

#### Data Export

* **unosm-r-data-01**
* Dedicated to export jobs
* Prevents heavy exports from impacting core services

### TL;DR Flow

Users → NetScaler → App Servers → Read SlavesImports → unosm-r-app-01 → unosm-r-db-01Writes → unosm-r-db-01 → WAL → Slaves

Cancel