Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
P
pac
Overview
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
ai-tools
pac
Commits
c594bce4
Commit
c594bce4
authored
Sep 05, 2026
by
luoqi
Browse files
Options
Browse Files
Download
Plain Diff
merge: 兜底注释按实测更正 → main
parents
289a7182
0ead4e0b
Pipeline
#3679
failed in 0 seconds
Changes
2
Pipelines
1
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
17 additions
and
6 deletions
+17
-6
deploy/health-watch.sh
+9
-5
docker-compose.prod.yml
+8
-1
No files found.
deploy/health-watch.sh
View file @
c594bce4
...
...
@@ -155,11 +155,15 @@ fails=$((fails + 1))
[[
-z
"
$since
"
]]
&&
since
=
"
$(
ts
)
"
say
"失败 #
$fails
$diag
"
DIAG_HINT
=
'**先看这几样:**
> docker ps -a --filter name=pac-
> 容器 exit=137 → 撞了 compose 的 mem_limit(8g);restart:always 会自己拉起,回头查那轮 plan 的 heapUsed= 日志
> free -g / dmesg -T | tail -30 → 整机内存(生产无 swap,活锁不会留 OOM 日志)
> docker logs --tail=200 pac-pac-service-1'
DIAG_HINT
=
'**照这个顺序看:**
> docker inspect -f "{{.RestartCount}} {{.State.Status}} {{.State.StartedAt}}" pac-pac-service-1
> ⚠️ 别看 State.ExitCode/OOMKilled —— restart:always 会立刻拉起,这两个字段显示的是
> **当前这条命**,被杀那条已被覆盖(09-05 实测:内核明确杀了 2 次,inspect 仍报 ExitCode=0)。
> 判据是 RestartCount 变大 + StartedAt 很新。
> dmesg -T | grep -i CONSTRAINT_MEMCG | tail -3
> 有 = 撞了 compose 的 mem_limit(8g),这是**预期内的自愈**,回头查那轮 plan 的 heapUsed= 日志
> 没有 + 机器整体很卡 = 09-04 那种无 swap 的 page-cache 活锁,内核不留任何 OOM 记录
> free -g; docker logs --tail=200 pac-pac-service-1'
if
[[
"
$alerted
"
==
"0"
&&
"
$fails
"
-ge
"
$WATCH_FAIL_THRESHOLD
"
]]
;
then
alerted
=
1
...
...
docker-compose.prod.yml
View file @
c594bce4
...
...
@@ -101,8 +101,15 @@ services:
# ② cgroup OOM —— 容器没上限(HostConfig.Memory=0),这条压根不存在 ← 本行修的就是它
# ③ 内核 OOM killer —— 机器**没有 swap**,内核宁可反复丢/重读 page cache 也不判 OOM
# (file page 永远"可回收"),于是没有任何一次 out_of_memory() 被触发 → refault 活锁。
# 所以本行买的不是"不涨",是**把爆炸半径从整机缩回单容器**:越界 → SIGKILL
(exit 137)
# 所以本行买的不是"不涨",是**把爆炸半径从整机缩回单容器**:越界 → SIGKILL
# → restart:always 拉起 → 回到 08-29 那种分钟级自愈。
# ✅ 2026-09-05 在测试机用**同一个 pac-service 镜像**、256MB 上限的探针实证过整条链:
# 内核 `oom-kill:constraint=CONSTRAINT_MEMCG` → docker 自动重启(RestartCount 累加)。
# 同一次实测还记到 rss=241MB 时 heapUsed 只有 4MB —— **堆上限管不到堆外**,
# 这正是 max-old-space-size 当不了兜底、必须有本行的原因。
# ⚠️ 事后查证别看 `docker inspect` 的 State.ExitCode / OOMKilled:restart:always 下
# 它们显示的是**重启后那条命**(实测内核杀了 2 次,inspect 仍报 ExitCode=0 OOMKilled=false)。
# 判据是 RestartCount 变大 + `dmesg -T | grep CONSTRAINT_MEMCG`。
#
# 【8g 这个数怎么来的】(生产 pac-friday 实测)
# 全机 15.36 GB
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment