fix(备份): 先轮转再 dump + 空间预检 —— 这个脚本自己把测试库崩过两次

测试机 47.251.104.47 的每日备份(`/root/pac-backup.sh`,cron 0 4 * * *)
连着两次(08-11、08-15)在 04:30 前后失败,backup.err 里是
「server closed the connection unexpectedly」。翻 postgres 容器日志才看清:

  PANIC: could not write to file "pg_logical/replorigin_checkpoint.tmp":
         No space left on device
  checkpointer process was terminated by signal 6: Aborted

⇒ **不是 PG 的毛病,是这个脚本撑爆的盘。** 轮转写在 dump 成功之后,
  于是写第 4 份(~12G)时旧 3 份 33G 全程占着 → 盘满 → PG 当场 PANIC、
  整个实例重启走 WAL 崩溃恢复 → pg_dump 连接跟着断。

⚠️ 最难受的一点:脚本删掉 .partial 之后空间就回来了、库自己恢复完毕,
  **白天查什么都正常**,所以连崩两次都没人发现。
⚠️ 崩溃恢复还清空了 pg_stat_*(统计文件不跨崩溃保留)——
  之后查到的死元组数只是崩溃后攒的,⛔ 别拿它判断表膨胀。

两处结构性修改:
① **先轮转、再 dump**:留 KEEP-1 份进 dump,写完正好 KEEP 份。
   峰值从 (KEEP+1)×份 压到 KEEP×份 —— 不再需要凭空多出一份的余量。
   代价:dump 失败时手上只剩 KEEP-1 份;比起把库撑崩,可接受。
② **动手前先算够不够**,不够就不开工并大声记日志。
   ⛔ 不许为腾地方自动多删旧备份 —— 少留几天恢复窗口是人的决定,不是脚本的。
   预检在**删之前**算(用"轮转能腾出多少"做加数),不够时一份都不动。

另外:KEEP 3→2(盘 197G / PG 卷已 75G / 一份 dump 12G,KEEP=3 两天就回到
5G 余量,正是事故前的水位);加 --dry-run(只算不动手);backup.err 超 5M 自转。

📌 顺带把它收进仓库:此前**只存在于服务器上**,而 scripts/backup-db.sh 是另一个
   本地临时用的脚本,两者早已各走各的。README 写清谁是谁。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

README.md

@@ -250,13 +250,29 @@ ps aux | grep -E "node.*(prisma|nest|next)" | grep -v grep | awk '{for(i=11;i<=N
 ps aux | grep -E "node.*(prisma|nest|next)" | grep -v grep | awk '{print $2}' | xargs kill 2>/dev/null
 ```
 
-### Backups (`scripts/backup-db.sh`)
+### Backups
+
+Two scripts, different jobs — don't confuse them:
+
+| | |
+|---|---|
+| `scripts/backup-db.sh` | **ad-hoc / local.** Dump before a risky migration, etc. |
+| `deploy/pac-backup.sh` | **the nightly cron on the test host** (`/root/pac-backup.sh`, `0 4 * * *`). Edit it here and copy it up — for months it existed only on the server. |
 
 ```bash
 ./scripts/backup-db.sh                                # ./backups, keep 7
 BACKUP_DIR=/var/backups RETENTION=14 ./scripts/backup-db.sh
+
+bash /root/pac-backup.sh --dry-run                    # on the host: check space, delete nothing
 ```
 
+> ⚠️ **The nightly script rotates *before* it dumps, and refuses to start when space is short.**
+> It used to rotate afterwards, so the disk had to hold `KEEP+1` dumps at once — on 2026-08-11
+> and 2026-08-15 it filled the disk and Postgres died mid-dump
+> (`PANIC: could not write to file "pg_logical/replorigin_checkpoint.tmp": No space left on device`).
+> The DB recovered on its own once the partial dump was deleted, so **nothing looked wrong by
+> daylight** and it happened twice before anyone noticed. Don't reorder those two steps back.
+
 Restore (test in a scratch DB regularly — an unverified backup is no backup):
 ```bash
 docker exec -i pac-postgres pg_restore -U pac -d pac \

deploy/pac-backup.sh

+#!/usr/bin/env bash
+# PAC Postgres 每日备份 — pg_dump 自定义压缩格式 + 落地校验 + 保留最近 KEEP 份。
+#
+# 部署:测试机 47.251.104.47 的 /root/pac-backup.sh,cron `0 4 * * *`,
+#      日志 >> /root/pac-backups/backup.log、stderr >> backup.err。
+# 存放:/root/pac-backups/(宿主盘)。⚠️ 同机同盘,防误删/逻辑错误/升级回滚;
+#      ⛔ 不防"整机/磁盘损坏" —— 异地容灾(如阿里云 OSS)仍是下一步。
+#
+# 用法:
+#   bash /root/pac-backup.sh              # 正常跑
+#   bash /root/pac-backup.sh --dry-run    # 只算不动手(预检 + 会删哪几份),用来验空间够不够
+#
+# ═══════════════════════════════════════════════════════════════════════
+# 🔴 2026-08-15 事故:**这个脚本自己把库搞崩了**,而且连崩两次(08-11、08-15)。
+#
+# 经过:04:00 开始写第 4 份 dump(~12G),而**轮转写在 dump 成功之后** ——
+#   旧 3 份 33G 全程占着 → 盘打满 → Postgres 当场:
+#     PANIC: could not write to file "pg_logical/replorigin_checkpoint.tmp":
+#            No space left on device
+#     checkpointer process was terminated by signal 6: Aborted
+#   → 整个实例重启、走 WAL 崩溃恢复;pg_dump 的连接跟着断,
+#     报出来的却是 "server closed the connection unexpectedly"。
+#
+# ⚠️ **它看起来像 PG 的毛病,其实是这个脚本撑爆的盘。**
+#   而且脚本删掉 .partial 之后空间就回来了、库自己恢复完毕,
+#   白天查什么都正常 —— 所以连崩两次都没人发现。
+# ⚠️ 崩溃恢复还顺手清空了 pg_stat_*(统计文件不跨崩溃保留):
+#   那之后查到的死元组数只是崩溃之后攒的,⛔ 别拿它判断表膨胀。
+#
+# ⇒ 两处结构性修改,⛔ 别改回去:
+#   ① **先轮转、再 dump**(留 KEEP-1 份进 dump,写完正好 KEEP 份)。
+#      峰值占用从 (KEEP+1)×份 压到 KEEP×份 —— 不再需要凭空多出一份的余量。
+#      ⚠️ 代价:dump 失败时手上只剩 KEEP-1 份。比起"把库撑崩",这个代价可接受。
+#   ② **动手之前先算够不够**,不够就**不开工**,大声记日志后退出。
+#      ⛔ 不许为了腾地方自动多删旧备份 —— 少留几天恢复窗口是人的决定,不是脚本的。
+# ═══════════════════════════════════════════════════════════════════════
+set -euo pipefail
+
+DIR=/root/pac-backups
+CONTAINER=pac-postgres-1
+DB=pac
+DBUSER=pac
+
+# 🔴 2026-08-15 由 3 降到 2。算式(盘 197G,PG 数据卷已 75G 且在长,一份 dump ~12G):
+#   KEEP=3 静息就占 36G,两天内盘会重新回到 5G 余量 —— 那正是事故前的水位。
+#   KEEP=2 静息 24G、dump 期间峰值同样 24G(先轮转),给 WAL 和别的项目构建留出余地。
+# ⚠️ 这是权宜之计:dump 每天还在长(11G→11G→12G),真正的出路是把它送出本盘。
+KEEP=2
+
+# dump 跑完盘上至少还要剩这么多(GB)。⛔ 别调到 0:
+#   PG 的 WAL、docker 构建、以及这台机器上另外 ~30 个无关容器都还要写盘。
+RESERVE_GB=10
+
+DRY_RUN=0
+[[ "${1:-}" == "--dry-run" ]] && DRY_RUN=1
+
+TS=$(date +%F_%H%M)
+FILE="$DIR/pac-$TS.dump"
+mkdir -p "$DIR"
+log() { printf "%s %s\n" "$(date "+%F %T")" "$*"; }
+gb() { awk -v k="$1" 'BEGIN{printf "%.1fG", k/1024/1024}'; }
+
+# ── backup.err 自身也会把盘写满(每次失败都追加,事故当天涨得最快)────────
+if [[ -f "$DIR/backup.err" ]] && (($(du -k "$DIR/backup.err" | cut -f1) > 5120)); then
+  mv -f "$DIR/backup.err" "$DIR/backup.err.1"
+fi
+
+# ── ① 预检:先算,不够就不开工 ──────────────────────────────────────
+mapfile -t DUMPS < <(ls -1t "$DIR"/pac-*.dump 2>/dev/null || true)
+
+# 这次大概要写多大:按现存最大的一份 ×1.25 估(库在长,⛔ 别按上次的实际值卡死)
+if ((${#DUMPS[@]} > 0)); then
+  NEED_KB=$(($(du -k "${DUMPS[@]}" | sort -nr | head -1 | cut -f1) * 5 / 4))
+else
+  NEED_KB=$((14 * 1024 * 1024)) # 一份都没有(首次/刚清过):按 14G 估
+fi
+
+# 轮转会删掉哪几份、能腾出多少 —— ⚠️ 这一步只算,还没删
+DOOMED=()
+((${#DUMPS[@]} >= KEEP)) && DOOMED=("${DUMPS[@]:$((KEEP - 1))}")
+FREEABLE_KB=0
+((${#DOOMED[@]} > 0)) && FREEABLE_KB=$(du -k "${DOOMED[@]}" | awk '{s+=$1} END{print s+0}')
+
+AVAIL_KB=$(df -Pk "$DIR" | awk 'NR==2{print $4}')
+AFTER_KB=$((AVAIL_KB + FREEABLE_KB - NEED_KB))
+log "预检:可用 $(gb "$AVAIL_KB") + 轮转可腾 $(gb "$FREEABLE_KB") − 本次约需 $(gb "$NEED_KB") = 结束后约 $(gb "$AFTER_KB")(下限 ${RESERVE_GB}G)"
+
+if ((AFTER_KB < RESERVE_GB * 1024 * 1024)); then
+  log "FAIL: 空间不足,本次不备份 —— ⛔ 现存 ${#DUMPS[@]} 份一份没动。"
+  log "      (⛔ 不自动多删旧备份腾地方:少留几天恢复窗口是人的决定,不是脚本的)"
+  log "      处理:清盘 / 调小 KEEP / 把 dump 送出本盘(OSS)。"
+  exit 1
+fi
+
+# ── ② 先轮转,再 dump(峰值就是靠这个顺序压下来的)──────────────────
+if ((${#DOOMED[@]} > 0)); then
+  log "先轮转(⚠️ 在 dump 之前):删 ${#DOOMED[@]} 份,留 $((KEEP - 1)) 份进本次 dump"
+  printf '  %s\n' "${DOOMED[@]}"
+  ((DRY_RUN == 0)) && rm -f "${DOOMED[@]}"
+fi
+
+if ((DRY_RUN == 1)); then
+  log "--dry-run:到此为止,⛔ 没删文件、没跑 pg_dump"
+  exit 0
+fi
+
+log "开始 pg_dump -> $FILE"
+if docker exec "$CONTAINER" pg_dump -U "$DBUSER" -Fc "$DB" >"$FILE.partial" 2>>"$DIR/backup.err"; then
+  mv "$FILE.partial" "$FILE"
+else
+  rm -f "$FILE.partial"
+  log "FAIL: pg_dump 失败(见 backup.err);盘还剩 $(gb "$(df -Pk "$DIR" | awk 'NR==2{print $4}')")"
+  exit 1
+fi
+log "完成 size=$(du -h "$FILE" | cut -f1)"
+
+# 落地校验:能列出归档 TOC = 文件可读(不是坏 dump)
+if docker exec -i "$CONTAINER" pg_restore -l <"$FILE" >/dev/null 2>&1; then
+  log "校验 OK(pg_restore -l 可读)"
+else
+  log "WARN: 校验失败,dump 可能损坏 — 保留文件人工查"
+fi
+
+# ⚠️ 这里**不再**轮转:该删的在 dump 之前就删了,现在正好 KEEP 份。
+log "现有 $(ls -1 "$DIR"/pac-*.dump 2>/dev/null | wc -l) 份(上限 $KEEP),盘还剩 $(gb "$(df -Pk "$DIR" | awk 'NR==2{print $4}')"):"
+ls -lh "$DIR"/pac-*.dump 2>/dev/null | awk '{print "  "$9" "$5}'

scripts/backup-db.sh

@@ -3,6 +3,11 @@
 # Dump the Postgres database to a timestamped pg_dump custom-format file
 # and rotate to the most recent N backups.
 #
+# ⚠️ This is the ad-hoc / local one (dump before a risky migration, etc.).
+#    The nightly cron on the test host is a DIFFERENT script: deploy/pac-backup.sh.
+#    It rotates *before* dumping and refuses to start when space is short — see the
+#    incident note in its header. Fix bugs in both, or in neither.
+#
 # Usage:
 #   ./scripts/backup-db.sh                              # default: ./backups, keep 7
 #   BACKUP_DIR=/var/backups RETENTION=14 ./scripts/backup-db.sh