Disk space issue in / partition on unosm-r-app-01.global.un.org

Title

Status

Due

Scheduled

Context

Project

Category

Notes Saved

## 31 Mar 26

Contacted by Zeeshan by Teams.
Root partition at 100% on unosm-r-app-01.global.un.org
Ticket opened to Fulvio on March 13th.

* [x] Check disk space issue

We identified 100% disk usage on the root filesystem by systematically narrowing down space consumption with `du`, tracing it to `/home/_osmprod/imposm/diffs/` where 852 orphaned temp files had accumulated. Cross-referencing imposm's state file confirmed these were all from failed runs since February 27, none of which were ever applied to the database.

* [x] Inform Zeeshan
* [x] Delete files and inform Emmanuel
* [x] Connect OneDrive to have access to disk request templates
* [x] Find request template

Reference ticket: [RFS-1-12444570177]()
Server Modifications with approval

Please increase 100GB of disk space on root partition for the following servers:
unosm-r-app-01

* [x] Request 100GB addition to / partition

Opened RFS-1-12846309167

* [x] Provide RFS
* [x] Check uCMDB technical maintainer
* [x] Generate Claude tshoot and learning report

## Report

# Troubleshooting Report — 100% Disk Usage

**Server:** unosm-r-app-01
**Date:** March 31, 2026
**Filesystem affected:** `/dev/mapper/system-root` (53G) mounted on `/`

***

## 1\. Initial Assessment

Running `df -h` confirmed the root filesystem was completely full, with only 20K available:

```
[root@unosm-r-app-01 ~]# df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/system-root    53G   53G   20K 100% /
/dev/mapper/data-lv_data 1000G  559G  442G  56% /data
/dev/mapper/system-var     15G  2.4G   13G  16% /var
```

All other filesystems (`/data`, `/var`, `/boot`) were healthy. The problem was isolated to `/`.

***

## 2\. Identifying the Space Consumer

A filesystem-bound `du` scan (using `-x` to avoid crossing mount boundaries) was run to find the top consumers:

```
[root@unosm-r-app-01 ~]# du -x --max-depth=3 / 2>/dev/null | sort -rh | head -30
54025092        /
46430112        /home
32181416        /home/_osmprod
24935640        /home/_osmprod/imposm
7241192         /home/_osmprod/.local
5895276         /home/_osmstat
...
```

**Finding:** `/home` was consuming 45G of the 53G root filesystem. The biggest single contributor was `/home/_osmprod/imposm` at \~24G.
A check for deleted-but-open files (`lsof +L1`) showed some entries on `/var` (separate mount) — not relevant to the root filesystem issue.

***

## 3\. Drilling Into `/home/_osmprod/imposm`

```
[root@unosm-r-app-01 ~]# du -x --max-depth=3 /home/_osmprod/imposm 2>/dev/null | sort -rh | head -20
24935640        /home/_osmprod/imposm
24204688        /home/_osmprod/imposm/diffs
12954840        /home/_osmprod/imposm/diffs/osm_planet
12954836        /home/_osmprod/imposm/diffs/osm_planet/000
11249844        /home/_osmprod/imposm/diffs/osm_planet_buildings
11249840        /home/_osmprod/imposm/diffs/osm_planet_buildings/000
```

Nearly all 24G was inside the `diffs/` directory. Initial assumption was `.osc.gz` diff files, but a search returned zero results:

```
[root@unosm-r-app-01 ~]# find /home/_osmprod/imposm/diffs/ -name "*.osc.gz" | wc -l
0
```

***

## 4\. Identifying the Actual Files

Further investigation revealed the files had non-standard names — a `~NUMBER` suffix indicating they were temp files from imposm's parallel downloader:

```
[root@unosm-r-app-01 ~]# find /home/_osmprod/imposm/diffs/ -type f -printf "%s %p\n" | sort -rn | head -5
107556350 /home/_osmprod/imposm/diffs/osm_planet/000/004/_545.osc.gz_
69556355  /home/_osmprod/imposm/diffs/osm_planet_buildings/000/004/919.osc.gz~744778
67094391  /home/_osmprod/imposm/diffs/osm_planet/000/004/919.osc.gz~1141209
66747751  /home/_osmprod/imposm/diffs/osm_planet_buildings/000/004/919.osc.gz~1141210
66700075  /home/_osmprod/imposm/diffs/osm_planet_buildings/000/004/919.osc.gz~768258
```

**852 orphaned temp files, totalling 22.98 GB:**

```
[root@unosm-r-app-01 ~]# find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" | wc -l
852
[root@unosm-r-app-01 ~]# find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" -printf "%s %p\n" | awk '{sum+=$1} END {print sum/1024/1024/1024 " GB"}'
22.9814 GB
```

***

## 5\. Confirming Files Are Orphaned $Not Normal$

Imposm's state file was checked to determine the last successfully processed sequence:

```
[root@unosm-r-app-01 ~]# cat /home/_osmprod/imposm/diffs/osm_planet/last.state.txt
timestamp=2026-02-27T00\:00\:00Z
sequenceNumber=4916
replicationUrl=https://planet.openstreetmap.org/replication/day/
```

**Last successful run: February 27, 2026 — sequence 4916.** All temp files belong to sequences 917, 918, and 919 — every run since Feb 27 has failed mid-download.
Age breakdown confirmed accumulation over 32 days:

```
[root@unosm-r-app-01 ~]# find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" -printf "%TY-%Tm-%Td\n" | sort | uniq -c
     12 2026-02-28
     22 2026-03-01
    291 2026-03-02
    352 2026-03-03
     31 2026-03-04
     ...
      1 2026-03-21
```

The spike on March 2–3 (643 files) indicates a major crash event on those dates.

***

## 6\. Remediation

After confirmation from the application owner, the orphaned temp files were deleted:

```
# Dry run
find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" | wc -l
find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" -printf "%s %p\n" | awk '{sum+=$1} END {print sum/1024/1024/1024 " GB"}'

# Delete
find /home/_osmprod/imposm/diffs/ -name "*.osc.gz~*" -delete

# Verify
df -h /
```

**Expected result:** \~23GB recovered, root filesystem down to \~57% used.

***

## 7\. Root Cause & Follow\-up

The disk filling was a symptom. The root cause is that **imposm has been failing to process daily OSM diffs since February 27, 2026**, leaving the OSM database 32+ days out of date. Without fixing the underlying imposm issue, temp files will begin accumulating again on the next run.
**Action required by app team:** Investigate why imposm has been crashing since Feb 27 — check service logs, OOM events, and upstream replication URL availability.

Supports Markdown: bold, italics, code, lists, links. Auto-saves.

📂 Activities for this Item

➕ Add Activity

Open Activities

No open activities.

Closed Activities

Synced from Mission Control

Service Requests / Work Orders

RFS-1-12848747197

Service Request

WO-RFS-1-12848747203

Normalized time tracking

Last 30 days

Logged: 2:20 · Normalized: 2:45

2026-03-31

Day total: 2:45

Normalized (1) · 2:45

Disk space issue in / partition on unosm-r-app-01.global.un.org
Logged: 2:20

2:45

Time entries 2:20

Synced from Mission Control - Identified 100% disk usage on the root filesystem by narrowing down space consumption with du, tracing it to /home/_osmprod/imposm/diffs/ where 852 orphaned temp files had accumulated. Cross-referencing imposm's state file confirmed these were all from failed runs since February 27, none of which were ever applied to the database. Deleted files and prepared the request for additional disk space Opene the disk space request and updated the knowledge base
2:45

Synced from Mission Control
Identified 100% disk usage on the root filesystem by narrowing down space consumption with du, tracing it to /home/_osmprod/imposm/diffs/ where 852 orphaned temp files had accumulated. Cross-referencing imposm's state file confirmed these were all from failed runs since February 27, none of which were ever applied to the database.

0:48 · 07:05
Synced from Mission Control
Deleted files and prepared the request for additional disk space

1:08 · 09:52
Synced from Mission Control
Opene the disk space request and updated the knowledge base

0:24 · 12:14

Meeting Notes (Project / Context)

Select a meeting to preview details.

iNeed Text

Title

Description

OSM - Disk space issue in / partition on unosm-r-app-01.global.un.org
Assign to Carlos Rodrigo - AOM DS

Asset#

Defined per context on the Taxonomy page.

Open Prompt

- Connect to https://ineedservice.un.org/erm_enu/start.swe?SWECmd=Login&SWEPL=1&SRN=YBQUIxGq9ccz7LVKJnezxs3Je6BWYmMN1w9dFiAfPc8b&SWETS=1480331200
- From Favorites section, select UNGSC AOM DS Generic SR
- Click on Continue
- Select 'UNLB47 - ICTS/ICC Technical Services/Application Operations Management' in Service/Section
- Go to Fill out Request details
- In Summary, substitute the current text with:
OSM - Disk space issue in / partition on unosm-r-app-01.global.un.org
- In Describe/Justify... put the following:
OSM - Disk space issue in / partition on unosm-r-app-01.global.un.org
Assign to Carlos Rodrigo - AOM DS
- Click on Submit
- Click on the SR reference for the ticket we just created to enter in the ticket
- Save the WO reference that says it was created for UNGSC AOM DS
- Open now https://ineed.un.org/
- Go to Work Order section
- Go to All my group's Work Orders, click on Search icon and in the WO # put the WO reference
- In 'Assigned To' put 'RODRIGO' and hit Tab key to save
- Click on the WO reference
- Go to Assets and click on the + button
- In asset number put 'Open Street Map Server (ImpOSM)' and save
- Click on the Change status button and select Status: Pending and Sub-Status: Waiting on Customer
- Provide here the RFS and WO number
- If there is already an asset # don't add another one

Close Prompt

- Open https://ineed.un.org/ and login
- Go to Work Order section
- Click on Search icon and in the WO # put the following WO reference:
WO-RFS-1-12848747203
- Click on Priority field to deselect the WO # field
- Click on the WO reference link
- Go to Assets and review that in the Asset List, there is at least one item. If not, stop.
- Click on the Change status button, which is a Hierarchy/org-chart icon located just right from the 'Form available' text
- From Status dropdown select: Closed
- From Sub-Status dropdown select: Completed
- In Comments field just below Sub-Status, write 'Completed' and click on Save

Templates

Change Title

Change Description

Change Steps

Template Notes

Communications

Label

Order

Subject

Email Body

Attachments

No attachments yet.