# Recover from AOF corruption

The AOF (append-only file) is Redis's write-ahead log. A truncated or corrupted
AOF stops the server from starting because Redis refuses to load a file it can't
fully parse. This is a deliberate guardrail. The recovery is methodical and you
have time to do it right.

The single most important rule: copy the AOF file before you do anything. Every
fix here is reversible only if you still have the original bytes.

## 0. Stop and back up

The server is already down (this is why you are here). Confirm, then copy:

```sh
systemctl status redis-server                            # confirm it is not running
cp -a /var/lib/redis/appendonly.aof /var/lib/redis/appendonly.aof.bak
ls -l /var/lib/redis/appendonly.aof*
```

If multipart AOF is enabled (Redis 7+, the default), the directory is named
`appendonlydir/` and contains `appendonly.aof.<n>.base.rdb`,
`appendonly.aof.<n>.incr.aof`, and a `manifest`. Copy the whole directory.

```sh
cp -a /var/lib/redis/appendonlydir /var/lib/redis/appendonlydir.bak
```

## 1. Find out what is actually wrong

Read the server's last log lines first. Redis tells you which file failed and at
which offset:

```sh
journalctl -u redis-server -n 100 --no-pager
# or
tail -200 /var/log/redis/redis.log
```

Typical errors:

- `Bad file format reading the append only file: make a backup of your AOF file,
  then use ./redis-check-aof --fix <filename>` — the file is truncated or has
  trailing garbage. This is the common case after a host crash.
- `Reading the AOF file: short read while loading the DB` — same family of
  problem, same fix.
- `Error reading the manifest file ... bad version` — the multipart manifest is
  damaged; the AOF data itself may be intact.

## 2. Run the check (read-only first)

`redis-check-aof` ships with Redis. Without `--fix` it reads the file and reports
the first offset it cannot parse, with no changes:

```sh
redis-check-aof /var/lib/redis/appendonly.aof
# or for multipart:
redis-check-aof /var/lib/redis/appendonlydir/appendonly.aof.manifest
```

The exit code is non-zero on corruption, and the output names the file and offset.
On modern Redis the tool understands the multipart layout when given the manifest.

## 3. Fix only what you have to

If the corruption is at the tail of the file (the host crashed mid-write), the
right answer is to truncate the file at the last valid command. Redis loses the
writes that were in flight at the crash, which is what AOF promises with
`appendfsync everysec`.

```sh
redis-check-aof --fix /var/lib/redis/appendonly.aof
# or for multipart:
redis-check-aof --fix /var/lib/redis/appendonlydir/appendonly.aof.manifest
```

The tool prints how many bytes it truncated. If that number is small (kilobytes
on a healthy disk), you lost at most one second of writes. If it is large
(megabytes or more), do not run `--fix` blindly. The corruption is not just a
truncated tail and you need to make a decision:

- Restore from the most recent RDB or backup if you have one. That moves the
  recovery to a known good point.
- Read the AOF in a text editor (it is RESP, mostly readable) at the offset the
  check tool named, and confirm what is being thrown away.
- If a replica was healthy at the time of the crash, promote the replica and
  rebuild this node from it. The replica's data is the source of truth in that
  case.

## 4. Special case: AOF rewrite was in flight

If the crash happened during `BGREWRITEAOF`, Redis may leave a partial
`temp-rewriteaof-*.aof` next to the live file. The live AOF is still authoritative
in this case. Delete the temp file before restart:

```sh
ls /var/lib/redis/*.aof*
rm /var/lib/redis/temp-rewriteaof-*.aof   # only the *temp* files
```

For multipart, a partial base file shows up as
`appendonlydir/appendonly.aof.<n>.base.rdb.partial` or similar. Again, remove
only the `*.partial` files; the manifest will fall back to the previous base.

## 5. Restart and verify

```sh
systemctl start redis-server
journalctl -u redis-server -n 50 --no-pager
```

The log should show `DB loaded from append only file: <N> seconds`. If it does,
the recovery is finished. Confirm the dataset is intact:

```sh
redis-cli DBSIZE
redis-cli INFO persistence | grep -E 'aof_|loading'
redis-cli INFO replication | grep -E 'role:|connected_slaves|master_link'
```

`loading:0` means the load completed; `aof_last_write_status:ok` means new writes
are landing. On a replica, `master_link_status:up` means the link to the primary
re-established.

## 6. After the recovery

- Bring evidence to a post-mortem. Why did the AOF tear? The usual answer is the
  host lost power or the disk filled. `df -h /var/lib/redis` is a one-line
  check that should be in your monitoring.
- Force one clean rewrite so the file is compact again:

```sh
redis-cli BGREWRITEAOF
redis-cli INFO persistence | grep aof_rewrite_in_progress
```

- If this happened in production, the right follow-up is to schedule a recovery
  drill on a non-prod copy of the same data so you have practiced the steps
  before the next incident.

## What the RKB tool gives you for free here

- The Health card flags `aof_last_write_status` and `rdb_last_bgsave_status`
  the moment they go non-ok, so a degraded persistence subsystem is visible
  before it becomes an outage.
- The Capacity card prints the persistence disk per shard with the AOF 2x and
  RDB 0.7x factors and the formula next to it, and the fork-on-save peak RSS
  headroom you should reserve.
- The Backup runbook RKB generates from a build wraps the same fix path
  (`redis-check-aof --fix` on a backed-up copy) into the verify step so the
  recovery is tested before you need it.