Disk space issue in / partition on unosm-r-app-01.global.un.org

Title

Status

Due

Scheduled

Context

Project

Category

Notes Saved

## 31 Mar 26

Contacted by Zeeshan by Teams.
Root partition at 100% on unosm-r-app-01.global.un.org
Ticket opened to Fulvio on March 13th.

* [x] Check disk space issue

We identified 100% disk usage on the root filesystem by systematically narrowing down space consumption with `du`, tracing it to `/home/_osmprod/imposm/diffs/` where 852 orphaned temp files had accumulated. Cross-referencing imposm's state file confirmed these were all from failed runs since February 27, none of which were ever applied to the database.

* [x] Inform Zeeshan
* [x] Delete files and inform Emmanuel
* [x] Connect OneDrive to have access to disk request templates
* [x] Find request template

Reference ticket: [RFS-1-12444570177]()
Server Modifications with approval

Please increase 100GB of disk space on root partition for the following servers:
unosm-r-app-01

* [x] Request 100GB addition to / partition

Opened RFS-1-12846309167

* [x] Provide RFS
* [x] Check uCMDB technical maintainer
* [x] Generate Claude tshoot and learning report

## Report

# Troubleshooting Report — 100% Disk Usage

**Server:** unosm-r-app-01
**Date:** March 31, 2026
**Filesystem affected:** `/dev/mapper/system-root` (53G) mounted on `/`

***

## 1\. Initial Assessment

Running `df -h` confirmed the root filesystem was completely full, with only 20K available:

```
[root@unosm-r-app-01 ~]# df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/system-root    53G   53G   20K 100% /
/dev/mapper/data-lv_data 1000G  559G  442G  56% /data
/dev/mapper/system-var     15G  2.4G   13G  16% /var
```

All other filesystems (`/data`, `/var`, `/boot`) were healthy. The problem was isolated to `/`.

***

## 2\. Identifying the Space Consumer

A filesystem-bound `du` scan (using `-x` to avoid crossing mount boundaries) was run to find the top consumers:

```
[root@unosm-r-app-01 ~]# du -x --max-depth=3 / 2>/dev/null | sort -rh | head -30
54025092        /
46430112        /home
32181416        /home/_osmprod
24935640        /home/_osmprod/imposm
7241192         /home/_osmprod/.local
5895276         /home/_osmstat
...
```

**Finding:** `/home` was consuming 45G of the 53G root filesystem. The biggest single contributor was `/home/_osmprod/imposm` at \~24G.
A check for deleted-but-open files (`lsof +L1`) showed some entries on `/var` (separate mount) — not relevant to the root filesystem issue.

***

## 3\. Drilling Into `/home/_osmprod/imposm`

```
[root@unosm-r-app-01 ~]# du -x --max-depth=3 /home/_osmprod/imposm 2>/dev/null | sort -rh | head -20
24935640        /home/_osmprod/imposm
24204688        /home/_osmprod/imposm/diffs
12954840        /home/_osmprod/imposm/diffs/osm_planet
12954836        /home/_osmprod/imposm/diffs/osm_planet/000
11249844        /home/_osmprod/imposm/diffs/osm_planet_buildings
11249840        /home/_osmprod/imposm/diffs/osm_planet_buildings/000
```

Nearly all 24G was inside the `diffs/` directory. Initial assumption was `.osc.gz` diff files, but a search returned zero results:

```
[root@unosm-r-app-01 ~]# find /home/_osmprod/imposm/diffs/ -name "*.osc.gz" | wc -l
0
```

***

## 4\. Identifying the Actual Files

Further investigation revealed the files had non-standard names — a `~NUMBER` suffix indicating they were temp files from imposm's parallel downloader:

```
[root@unosm-r-app-01 ~]# find /home/_osmprod/imposm/diffs/ -type f -printf "%s %p\n" | sort -rn | head -5
107556350 /home/_osmprod/imposm/diffs/osm_planet/000/004/_545.osc.gz_
69556355  /home/_osmprod/imposm/diffs/osm_planet_buildings/000/004/919.osc.gz~744778
67094391  /home/_osmprod/imposm/diffs/osm_planet/000/004/919.osc.gz~1141209
66747751  /home/_osmprod/imposm/diffs/osm_planet_buildings/000/004/919.osc.gz~1141210
66700075  /home/_osmprod/imposm/diffs/osm_planet_buildings/000/004/919.osc.gz~768258
```

**852 orphaned temp files, totalling 22.98 GB:**

```
[root@unosm-r-app-01 ~]# find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" | wc -l
852
[root@unosm-r-app-01 ~]# find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" -printf "%s %p\n" | awk '{sum+=$1} END {print sum/1024/1024/1024 " GB"}'
22.9814 GB
```

***

## 5\. Confirming Files Are Orphaned $Not Normal$

Imposm's state file was checked to determine the last successfully processed sequence:

```
[root@unosm-r-app-01 ~]# cat /home/_osmprod/imposm/diffs/osm_planet/last.state.txt
timestamp=2026-02-27T00\:00\:00Z
sequenceNumber=4916
replicationUrl=https://planet.openstreetmap.org/replication/day/
```

**Last successful run: February 27, 2026 — sequence 4916.** All temp files belong to sequences 917, 918, and 919 — every run since Feb 27 has failed mid-download.
Age breakdown confirmed accumulation over 32 days:

```
[root@unosm-r-app-01 ~]# find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" -printf "%TY-%Tm-%Td\n" | sort | uniq -c
     12 2026-02-28
     22 2026-03-01
    291 2026-03-02
    352 2026-03-03
     31 2026-03-04
     ...
      1 2026-03-21
```

The spike on March 2–3 (643 files) indicates a major crash event on those dates.

***

## 6\. Remediation

After confirmation from the application owner, the orphaned temp files were deleted:

```
# Dry run
find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" | wc -l
find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" -printf "%s %p\n" | awk '{sum+=$1} END {print sum/1024/1024/1024 " GB"}'

# Delete
find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" -delete

# Verify
df -h /
```

**Expected result:** \~23GB recovered, root filesystem down to \~57% used.

***

## 7\. Root Cause & Follow\-up

The disk filling was a symptom. The root cause is that **imposm has been failing to process daily OSM diffs since February 27, 2026**, leaving the OSM database 32+ days out of date. Without fixing the underlying imposm issue, temp files will begin accumulating again on the next run.
**Action required by app team:** Investigate why imposm has been crashing since Feb 27 — check service logs, OOM events, and upstream replication URL availability.

Supports Markdown: bold, italics, code, lists, links. Auto-saves.

AI Notes - managed by Claude

No AI notes yet.

📂 Activities for this Item

➕ Add Activity

Open Activities

No open activities.

Closed Activities

Synced from Mission Control

Service Requests / Work Orders

RFS-1-12848747197

Service Request

WO-RFS-1-12848747203

Normalized time tracking

Last 30 days

Logged: 0:00 · Normalized: 0:00

No normalized entries for this item yet.

Meeting Notes (Project / Context)

Select a meeting to preview details.

iNeed Text

Title

Description

OSM - Disk space issue in / partition on unosm-r-app-01.global.un.org
Assign to Carlos Rodrigo - AOM DS

Asset#

Defined per context on the Taxonomy page.

Templates

Change Title

Change Description

Change Steps

Template Notes

Communications

Label

Order

Subject

Email Body

Attachments

No attachments yet.