Compare commits
7
Commits
1b688d128c
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
  BluscreamandClaude Opus 5.5
|
5e6abadb95 | ||
|
BluscreamandClaude Sonnet 5.5
|
a16a879b1e | ||
|
BluscreamandClaude Opus 5.5
|
44c2d1502c | ||
|
BluscreamandClaude Opus 5
|
e385e01993 | ||
|
BluscreamandClaude Opus 5
|
6207de1a97 | ||
|
Bluscream
|
b8505fc623 | ||
|
BluscreamandClaude Opus 5
|
5aef9707d2 |
No files matched your search
@@ -9,8 +9,10 @@ A collection of public, reusable agent skills, cheatsheets, and operational work
|
||||
| Skill File | Scope / Topics Covered |
|
||||
| :--- | :--- |
|
||||
| **[`coding-dev.md`](skills-public/coding-dev.md)** | Standing rules for all code work: maximum linter/compiler strictness, never filtering build output, reuse-before-reinvent dependency ordering, file/function size limits, test & regression discipline, verification gates, scope control, commit hygiene, and honest reporting. |
|
||||
| **[`software-vision.md`](skills-public/software-vision.md)** | Starting from a software idea: turn it into a requirements checklist, search the web first (projects, forks, skeletons, libraries, user reviews), then merge in training-data recall checked against current sources, shallow-clone promising candidates to a temp dir to check them, then present use / extend / fork / start-fresh options before building anything. |
|
||||
| **[`cli-tooling.md`](skills-public/cli-tooling.md)** | Using command-line tools correctly: reading the help tree instead of guessing flags, preferring an existing CLI over an ad-hoc script, non-interactive invocation, machine-readable output formats, orienting on prior work before starting, script placement & symlinking, and care around destructive commands. |
|
||||
| **[`afk-sleep-work.md`](skills-public/afk-sleep-work.md)** | Operating unattended and autonomously: continuous forward momentum, avoiding polling traps and deadlocks, host preservation and resource budgeting (disk, packages, processes), isolated tool provisioning, avoiding "last breath" traps on token/quota exhaustion, bypassing or driving GUI launchers, and wake-up checkpoints. |
|
||||
| **[`android-backup-restore.md`](skills-public/android-backup-restore.md)** | Extracting/backing up/restoring Android data by access tier before a wipe or migration: EDL/firehose & bootloader, custom recovery (TWRP/OrangeFox/CWM), root, device-owner, Shizuku, plain ADB, and on-device apps; FBE decryption limits, per-partition dumps (userdata, EFS/modem/IMEI, persist), SELinux/UID-correct restores, the death of `adb backup`, and app-specific data (WhatsApp, Signal, TOTP/Keystore) that does not survive. |
|
||||
|
||||
### 🔤 Per-Language Rules ([`code-languages/`](skills-public/code-languages/))
|
||||
|
||||
@@ -19,10 +21,13 @@ A collection of public, reusable agent skills, cheatsheets, and operational work
|
||||
| **Rust** | [`rust.md`](skills-public/code-languages/rust.md) | `--workspace` clippy gate, `[workspace.lints]` single source of truth, no `unwrap`/`unsafe`, core-crate split |
|
||||
| **TypeScript / JavaScript** | [`js-typescript.md`](skills-public/code-languages/js-typescript.md) | TS > JSDoc > plain JS, banned constructs table, React & hooks rules, resource lifecycle, sandbox boundaries |
|
||||
| **Bash / Shell** | [`bash.md`](skills-public/code-languages/bash.md) | `set -euo pipefail`, shellcheck, no wait loops, destructive-command guards, build-gate structure |
|
||||
| **Python** | [`python.md`](skills-public/code-languages/python.md) | `mypy --strict`, broad ruff rule set, asyncio task discipline, rate-limit queueing |
|
||||
| **Python** | [`python.md`](skills-public/code-languages/python.md) | `mypy --strict`, broad ruff rule set, asyncio task discipline, rate-limit queueing, MCP/cog/HA framework rules |
|
||||
| **PHP** | [`php.md`](skills-public/code-languages/php.md) | `strict_types` everywhere, PHPStan `level: max`, PER-CS 2.0 risky rules, `final readonly` value objects, PSR interfaces over implementations, escaping & unwritable document root |
|
||||
| **Go** | [`go.md`](skills-public/code-languages/go.md) | strict `golangci-lint` set, concrete types over `interface{}`, context everywhere, CGO-free static builds |
|
||||
| **C# / .NET** | [`csharp-dotnet.md`](skills-public/code-languages/csharp-dotnet.md) | `TreatWarningsAsErrors`, nullable enabled, no `.Result`, trimming/single-file publish caveats, SDK shims |
|
||||
| **C / C++** | [`c-cpp.md`](skills-public/code-languages/c-cpp.md) | full warning set + `-Werror`, sanitizers, RAII/Rule of Zero, target-based CMake, FFI ownership |
|
||||
| **Lua** | [`lua.md`](skills-public/code-languages/lua.md) | pin the dialect first, `luac -p` + luacheck as the only gate, locals over globals, `pcall` at the boundary, `#t`/pattern traps, embedded & HavokScript hosts |
|
||||
| **GSC** | [`gsc.md`](skills-public/code-languages/gsc.md) | compile off-engine with `gsc-tool --mode comp`, `isdefined` on everything external, `endon` before the first wait, per-frame budget, precache at load, level-state side effects |
|
||||
|
||||
Each language file extends `coding-dev.md` rather than repeating it. A project's own
|
||||
`AGENTS.md` / `CLAUDE.md` / `.rules` always overrides both.
|
||||
|
||||
+1
-1
@@ -120,7 +120,7 @@ When remaining budget or context drops below a safe margin:
|
||||
|
||||
## 7. Checkpointing & The "Wake-Up" Audit Trail
|
||||
|
||||
Maintain a persistent status file (`PROGRESS.md` or session artifact) updated at each milestone:
|
||||
Maintain a persistent status file (for the tsw-plugins project `tsw-plugins/tasks/{todo,in-progress,done}.md`; otherwise a session artifact) updated at each milestone:
|
||||
|
||||
| Section | Contents |
|
||||
| :--- | :--- |
|
||||
|
||||
@@ -0,0 +1,562 @@
|
||||
---
|
||||
name: android-backup-restore
|
||||
description: >-
|
||||
Use this skill when extracting, backing up, migrating, or restoring data from an Android
|
||||
device — especially before a wipe, ROM flash, factory reset, or device-to-device move, and
|
||||
when deciding what is salvageable at a given access level. Covers the full ladder of access
|
||||
tiers (EDL/firehose & bootloader, custom recovery CWM/TWRP/OrangeFox, root, device-owner,
|
||||
Shizuku, plain ADB, and on-device apps), file-based-encryption decryption limits, per-partition
|
||||
dumps (userdata, EFS/modem/IMEI, persist), SELinux/UID-correct restores, the death of
|
||||
`adb backup`, and app-specific data (WhatsApp, Signal, TOTP/Keystore) that does not survive.
|
||||
---
|
||||
|
||||
# Android Backup & Restore — Extraction by Access Level
|
||||
|
||||
How to pull as much as possible off an Android device before wiping, flashing, or migrating —
|
||||
and how to put it back. Organised as a **ladder of access tiers**: the more privileged the
|
||||
access, the more you can extract. Work out which tier you have, pull from the highest tier
|
||||
available, and fall back down the ladder for whatever that tier cannot reach.
|
||||
|
||||
This is a *data-recovery and migration* skill for devices you own or are authorised to service.
|
||||
Shell orchestration here follows [`bash.md`](skills-public/code-languages/bash.md) (strict mode,
|
||||
hard `timeout`s, no wait-loops); CLI invocation follows
|
||||
[`cli-tooling.md`](skills-public/cli-tooling.md) (read `<tool> --help` before trusting any flag
|
||||
below — tool surfaces drift, and the exact partition/block paths are **per-device**).
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **The single biggest variable is encryption.** Every modern Android device uses File-Based
|
||||
> Encryption (FBE); the `/data` partition is a ciphertext blob until something that holds the
|
||||
> user credential derives the keys. That "something" is normally the booted OS. A custom
|
||||
> recovery can only read plaintext `/data` if it was *built to decrypt your exact Android
|
||||
> version* and you type the lockscreen PIN/password into it. See §3. This is why "boot it
|
||||
> normally and pull over ADB/root" beats "boot to recovery and dd" whenever the device still
|
||||
> boots.
|
||||
|
||||
---
|
||||
|
||||
## 1. The Access Ladder (decision table)
|
||||
|
||||
Pick the highest row you can reach. Each tier can extract everything below it plus its own column.
|
||||
|
||||
| Tier | Access you need | Reaches raw partitions? | Reaches decrypted `/data/data` app data? | Reaches EFS/IMEI/persist? | Notes |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| **EDL / firehose / BROM** (§8) | Vendor edl/mtk loader, device in 9008/BROM | ✅ full flash, offline | ⚠️ only as ciphertext (still FBE-locked) | ✅ | Last resort for dead/bricked devices; dumps are encrypted blobs unless you also have the keys. |
|
||||
| **Bootloader / fastboot** (§8) | Unlocked bootloader | ⚠️ some (`fastboot fetch` on newer; mostly flash-only) | ❌ | partial | Mostly a *flashing* interface, not a dumping one. |
|
||||
| **Custom recovery** TWRP/OrangeFox/CWM (§3) | Unlocked bootloader + matching recovery | ✅ every partition via `dd`/built-in backup | ⚠️ **only if recovery decrypts your FBE** + you enter the PIN | ✅ | Classic "nandroid". Decryption is the catch on Android 10+. |
|
||||
| **Root (booted OS)** (§4) | Magisk/KernelSU/APatch on a booting device | ✅ (block devices readable) | ✅ **fully decrypted** (OS is running) | ✅ | The best tier for a *working* device — plaintext data + raw partitions together. |
|
||||
| **Device Owner / DPC** (§5) | `dpm set-device-owner` on a fresh/unprovisioned device | ❌ | ❌ (own managed scope only) | ❌ | Niche; grants/denies permissions, reads some providers; not a bulk dumper. |
|
||||
| **Shizuku / ADB-shell privilege** (§6) | Shizuku (wireless-debug or one-time ADB) | ❌ | ⚠️ APKs, `Android/data`, `Android/obb`, media — **not** `/data/data` of release apps | ❌ | `adb shell` UID (2000) reach, on-device, no PC tether. |
|
||||
| **Plain ADB** (§6) | USB debugging authorised | ❌ | ⚠️ same as Shizuku + `run-as` for *debuggable* apps only | ❌ | `adb backup` is effectively dead (§7). Pull storage, providers, bugreport. |
|
||||
| **On-device app, no root** (§6.4) | Just the unlocked phone | ❌ | ❌ (own data only) | ❌ | Vendor cloud/D2D, Seedvault, per-app export. Last-mile for the non-technical case. |
|
||||
|
||||
**Rule of thumb for a pre-wipe sweep:** if it boots and you can unlock it → get **root** (§4) if
|
||||
possible, else **ADB** (§6). If it does *not* boot → **custom recovery** (§3), else **EDL** (§8).
|
||||
Always grab **EFS/IMEI/persist** (§4.3) before any flash regardless of tier.
|
||||
|
||||
---
|
||||
|
||||
## 2. First: inventory what is actually worth saving
|
||||
|
||||
Before dumping 128 GB of userdata, enumerate. Most of the value is small and specific.
|
||||
|
||||
- **Media & documents** — `/sdcard` (a.k.a. `/storage/emulated/0`): DCIM, Pictures, Download,
|
||||
Documents, Music, ringtones. Reachable at *every* tier including plain ADB. Pull this first; it
|
||||
is the easy win and the bulk of what users actually miss.
|
||||
- **App data** — `/data/data/<pkg>` and `/data/user/0/<pkg>` (credential-encrypted, "CE") and
|
||||
`/data/user_de/0/<pkg>` (device-encrypted, "DE"). Needs root or recovery-decrypt. This is where
|
||||
game saves, app logins, notes, 2FA seeds live.
|
||||
- **Scoped external app data** — `Android/data/<pkg>` and `Android/obb/<pkg>` on `/sdcard`.
|
||||
Blocked from ordinary file managers since Android 11/13 but reachable via ADB/Shizuku/root.
|
||||
- **Telephony/PIM** — SMS/MMS, call log, contacts, calendar. See §6.2 (providers) and §9.
|
||||
- **Identity partitions** — EFS/modem (`modemst1`, `modemst2`, `fsg`), `persist`, `efs`. IMEI,
|
||||
Wi-Fi/BT MAC, sensor calibration. §4.3. **Irreplaceable — back up before any flash.**
|
||||
- **The things that will NOT survive** no matter the tier — hardware-backed Keystore keys, and
|
||||
by extension many 2FA/banking apps. §10. Plan migrations for these *before* wiping.
|
||||
|
||||
---
|
||||
|
||||
## 3. Custom Recovery (TWRP / OrangeFox / CWM) — the offline nandroid
|
||||
|
||||
Use when the device will not boot normally, or you want a block-level image independent of the OS.
|
||||
Requires an **unlocked bootloader** (itself a wipe on most devices — so unlock *before* you have
|
||||
data worth keeping, or accept the loss). Boot without flashing when you can:
|
||||
`fastboot boot twrp.img` leaves the installed recovery untouched.
|
||||
|
||||
> [!TIP]
|
||||
> **Try the *installed* recovery before downloading or temp-booting one.** `adb reboot recovery`,
|
||||
> then `adb shell id` — many custom ROMs ship a current OrangeFox/TWRP whose `adbd` already runs as
|
||||
> **root** and, crucially, is new enough to **decrypt FBE on the installed Android version** (so
|
||||
> `/data/data` shows real package names and readable file contents). This beats `fastboot boot`-ing
|
||||
> an old recovery image: a recovery built from a much older branch often won't boot on a new ROM at
|
||||
> all (its kernel panics and the device falls back to the OS — you'll land back in Android as
|
||||
> `uid=2000(shell)`, the tell-tale that it didn't take). Confirm decryption with
|
||||
> `ls /data/data` (real names, not hex) and by reading one `*/shared_prefs/*.xml`.
|
||||
|
||||
### 3.1 The decryption problem (read this before trusting a backup)
|
||||
|
||||
TWRP/OrangeFox can only back up **plaintext** `/data` if its decryption code matches your device's
|
||||
FBE implementation *and* you enter the lockscreen credential in recovery (Keymaster derives the
|
||||
keys from it). Reality on Android 10+:
|
||||
|
||||
- Use a recovery **built from the branch matching your Android version** (e.g. an `android-12.1`+
|
||||
build for Android 12+). A recovery built from an old branch shows `/data` as garbage folder names
|
||||
or fails to mount.
|
||||
- Devices that **shipped on Android ≤11 and were upgraded** to 12+ often cannot be decrypted by any
|
||||
TWRP build — the keystore/Keymaster path changed. OrangeFox sometimes succeeds where TWRP fails;
|
||||
try both. If neither decrypts, you can still image the partition but only as ciphertext (useless
|
||||
without the keys).
|
||||
- If decryption fails, **do not factory-reset to "fix" it** — that destroys the keys and the data.
|
||||
|
||||
Verify decryption worked before relying on it: in the TWRP terminal, `ls -la /data/data` should
|
||||
show real package names (`com.whatsapp`, …), not hex blobs.
|
||||
|
||||
### 3.2 Built-in backup vs. raw `dd`
|
||||
|
||||
- **Built-in backup** (TWRP "Backup" / OrangeFox): tick System, Data, Boot, EFS, etc. Produces a
|
||||
nandroid set under `/sdcard/TWRP/BACKUPS/`. If `/data` decrypted, the Data archive holds plaintext
|
||||
app data; if not, it backs up the raw (encrypted) partition. Pull the whole folder off afterwards.
|
||||
- **Raw partition image** when you want a specific partition or the built-in backup won't cover it.
|
||||
From the recovery `adb shell` or terminal — **block paths are per-device**, find them first:
|
||||
|
||||
```bash
|
||||
# from recovery shell — enumerate by name, DO NOT guess the numbers
|
||||
ls -l /dev/block/by-name/ # symlinks: userdata, boot_a, modemst1, efs, persist, …
|
||||
cat /proc/partitions
|
||||
|
||||
# image a named partition to portable storage (not back onto /data)
|
||||
timeout 1800 dd if=/dev/block/by-name/userdata of=/sdcard/userdata.img bs=8M
|
||||
```
|
||||
|
||||
Pull with `adb pull /sdcard/userdata.img` (or write to an SD card / OTG drive). A raw `userdata`
|
||||
image is only restorable/mountable if you also have the FBE keys — prefer the decrypted built-in
|
||||
Data backup for app data, and reserve raw images for small partitions (§4.3).
|
||||
|
||||
### 3.3 Restore
|
||||
|
||||
Restore the built-in nandroid through the recovery UI, or `dd` a raw image back **only onto the
|
||||
exact same partition of the same device** (never another device's EFS — §4.3). After a decrypted
|
||||
Data restore across Android versions, expect some apps to misbehave; a clean reinstall + selective
|
||||
data restore (§4.2) is often more reliable than a whole-partition nandroid across versions.
|
||||
|
||||
---
|
||||
|
||||
## 4. Root on a booting device — the best tier for a live phone
|
||||
|
||||
If the device boots and you can unlock it, root (Magisk / KernelSU / APatch) is the strongest
|
||||
position: the OS has already derived the FBE keys, so **all app data is plaintext**, and you still
|
||||
have raw block-device access. (For installing root on this workstation's own test devices, Magisk
|
||||
APKs/modules may already be on the NAS — see
|
||||
[check-nas-before-downloading](check-nas-before-downloading.md) before fetching.)
|
||||
|
||||
### 4.1 Bulk app-data pull
|
||||
|
||||
```bash
|
||||
# as root, on device or via `adb shell su -c`
|
||||
# tar preserves owners/modes/symlinks — plain `cp`/`adb pull` loses them (see §4.4)
|
||||
su -c 'tar -cpf /sdcard/appdata-whatsapp.tar \
|
||||
/data/data/com.whatsapp /data/user_de/0/com.whatsapp'
|
||||
# then from the PC:
|
||||
adb pull /sdcard/appdata-whatsapp.tar
|
||||
```
|
||||
|
||||
Pull `/data/data/<pkg>` (CE) **and** `/data/user_de/0/<pkg>` (DE) for each app — some apps split
|
||||
state across both. For a whole-device grab, `tar` `/data/data`, `/data/user/0`, `/data/user_de/0`
|
||||
and `/data/misc` (wifi configs, etc.), but mind disk budget on the pulling host.
|
||||
|
||||
> [!TIP]
|
||||
> **Stream straight off the device with `adb exec-out` — no on-device staging.** When `/data` has
|
||||
> little free space (you can't write a multi-GB tar back onto the partition you're reading), pipe
|
||||
> the archive directly to the host, and `dd` images the same way (`exec-out` is binary-safe; plain
|
||||
> `adb shell` mangles bytes):
|
||||
> ```bash
|
||||
> adb exec-out 'cd /data && tar -cp -f - data user_de system misc vendor adb' > data-full.tar
|
||||
> adb exec-out 'dd if=/dev/block/by-name/boot status=none' > boot.img
|
||||
> ```
|
||||
> Toybox/busybox `tar` can't store SELinux contexts and exits non-zero when it skips sockets
|
||||
> (`unknown file type '140000'` — harmless). Capture contexts separately as a restore sidecar:
|
||||
> `for d in /data/data/* /data/user_de/0/*; do stat -c '%a %U %G %C %n' "$d"; done`. The per-app
|
||||
> SELinux *categories* (`:c123,c456`) encode the app sandbox and are **not** regenerable by
|
||||
> `restorecon` alone — this sidecar is how you put them back.
|
||||
>
|
||||
> **Record restore metadata while you have the device.** A `dumpsys package` capture gives, per
|
||||
> app: `codePath` (→ system/vendor/product/`/data/app`=user), `pkgFlags` (SYSTEM/UPDATED_SYSTEM_APP),
|
||||
> `installerPackageName` (Play `com.android.vending` vs sideloaded `null`), versionCode, and granted
|
||||
> runtime permissions; `cmd appops dump` adds special ops (SYSTEM_ALERT_WINDOW,
|
||||
> MANAGE_EXTERNAL_STORAGE, …). Parse these into a manifest so you can later restore *only* user apps
|
||||
> and re-grant their exact permissions.
|
||||
|
||||
Prefer dedicated tools over hand-rolled tar when restoring to a *different* device or Android
|
||||
version, because they record ownership/SELinux/permissions that a naive tar restore drops (§4.4):
|
||||
|
||||
- **Swift Backup** (root; Shizuku for a weaker subset) — app + data, permissions, SSAID.
|
||||
- **Neo Backup** (FOSS, OAndBackupX fork; root incl. KernelSU/APatch) — app + data, schedules.
|
||||
- **OAndBackupX / Migrate** — FOSS nandroid-style app+data sets.
|
||||
|
||||
### 4.2 Raw partitions while rooted
|
||||
|
||||
Same `dd` from `/dev/block/by-name/` as §3.2, but from the running OS (`su -c dd …`). Good for
|
||||
grabbing `boot`/`vendor_boot` to patch, or EFS/persist (§4.3) without rebooting to recovery.
|
||||
|
||||
### 4.3 EFS / modem / IMEI / persist — back these up before ANY flash
|
||||
|
||||
These hold IMEI, baseband calibration, Wi-Fi/BT MAC, and sensor calibration. Corrupting them bricks
|
||||
the radio permanently and there is no download that restores *your* unique values. **Partition names
|
||||
and block numbers are per-device — look them up, never copy numbers from a guide.**
|
||||
|
||||
```bash
|
||||
su -c 'ls -l /dev/block/by-name/' | grep -Ei 'efs|modemst|fsg|fsc|persist|nvram|nvdata|protect'
|
||||
# Qualcomm: commonly modemst1, modemst2, fsg, fsc, persist
|
||||
# MediaTek: commonly nvram, nvdata, protect_f, protect_s, persist
|
||||
for p in modemst1 modemst2 fsg persist; do
|
||||
su -c "dd if=/dev/block/by-name/$p of=/sdcard/efs-$p.img bs=4M"
|
||||
done
|
||||
```
|
||||
|
||||
Pull the images **off the device and keep copies on a host/NAS** — a backup that lives only on the
|
||||
partition you are about to wipe is no backup. Restore the same way (`dd … of=/dev/block/by-name/$p`)
|
||||
**only to the identical partition on the identical device**. Never flash one phone's EFS onto
|
||||
another — you would overwrite its unique identity with useless values.
|
||||
|
||||
### 4.4 Restoring raw `/data/data` tars correctly (UID + SELinux)
|
||||
|
||||
A tar of `/data/data/<pkg>` restored by hand will *not* launch unless you fix two things Android
|
||||
assigns at install time:
|
||||
|
||||
1. **Install the app first** (same package, ideally same version) so Android creates the data dir
|
||||
with the correct per-app UID and SELinux context.
|
||||
2. Restore the files, then **fix ownership and labels**:
|
||||
|
||||
```bash
|
||||
# find the app's real UID from the freshly-created dir, don't assume 100xx
|
||||
su -c 'ls -ld /data/data/com.example' # e.g. u0_a142 u0_a142
|
||||
su -c 'tar -xpf /sdcard/appdata-example.tar -C /'
|
||||
su -c 'chown -R u0_a142:u0_a142 /data/data/com.example'
|
||||
su -c 'restorecon -RFD /data/data/com.example' # argument order matters; -v to see work
|
||||
```
|
||||
|
||||
Then reboot before launching. Skipping the UID chown or `restorecon` is the usual reason a
|
||||
"successful" restore produces an app that crashes on open. Dedicated backup apps (§4.1) do this
|
||||
bookkeeping for you — hand-tar only when you must.
|
||||
|
||||
### 4.5 Getting root headlessly (no GUI taps) when the device boots
|
||||
|
||||
If the bootloader is unlocked and you need root to reach §4 but can't tap through the Magisk app,
|
||||
patch and flash boot entirely from the shell:
|
||||
|
||||
1. **Get the *exact* stock boot.img.** The authoritative copy is the live partition — dump it from
|
||||
a root recovery (§3) with `dd`/`exec-out`. Don't flash a boot.img of unknown origin: check its
|
||||
cmdline first (`androidboot.hardware=qcom` = Qualcomm vs MediaTek's `bootopt=…` — a wrong-SoC
|
||||
image bricks the device).
|
||||
2. **Patch it** with Magisk's own tooling. The Magisk APK is a zip: its `assets/boot_patch.sh` needs
|
||||
the arm64 libs renamed (`lib/arm64-v8a/libmagiskboot.so`→`magiskboot`, `libmagiskinit.so`→
|
||||
`magiskinit`, `libmagisk.so`→`magisk`, `libinit-ld.so`→`init-ld`) plus `stub.apk` and
|
||||
`util_functions.sh` in the same dir, then `sh boot_patch.sh boot.img` → `new-boot.img`.
|
||||
- Set the flags **explicitly** (standalone `boot_patch.sh` treats unset as false, which can
|
||||
bootloop): for a modern FBE device with a vbmeta partition and dynamic partitions use
|
||||
`KEEPVERITY=true KEEPFORCEENCRYPT=true PATCHVBMETAFLAG=false RECOVERYMODE=false LEGACYSAR=false`.
|
||||
`Failed to patch` on the kernel DTB step is benign (nothing to patch).
|
||||
- The recovery installer (`META-INF/com/google/android/update-binary`, which auto-detects these
|
||||
flags via `get_flags()`) is cleaner **but** fails outside a real sideload: it tries to mount
|
||||
`/system`, which on super/dynamic-partition ROMs is `/dev/mapper/system` and won't mount in a
|
||||
plain `adb shell`. `boot_patch.sh` with explicit flags sidesteps that.
|
||||
3. **Flash** `new-boot.img` (verify it unpacks: `magiskboot unpack new-boot.img`) with
|
||||
`fastboot flash boot` or, from a root recovery, `dd … of=/dev/block/by-name/boot`. Keep the stock
|
||||
image for instant rollback. Reboot: `magiskd` in `ps -A` + `magisk -V` confirms root took and
|
||||
survives reboot.
|
||||
4. **Grant su to the ADB shell** (Magisk denies it by default, silently): Magisk app → Superuser
|
||||
access = "Apps and ADB" + Automatic response = "Grant", or seed a `policies` row
|
||||
(`uid=2000 policy=2`) in `/data/adb/magisk.db`. Until then `adb shell su -c …` returns
|
||||
`Permission denied` even though root is installed and working.
|
||||
|
||||
---
|
||||
|
||||
## 5. Device Owner (DPC) — narrow, not a bulk extractor
|
||||
|
||||
A device-owner app (set with `adb shell dpm set-device-owner <pkg>/<receiver>` on a device with
|
||||
**no accounts added yet**) gets elevated *policy* powers, not filesystem powers. It is relevant to
|
||||
extraction only in that it can silently grant its own runtime permissions and read some content
|
||||
providers that would otherwise prompt — useful for a managed enrolment flow, not for cleaning out a
|
||||
phone before a wipe. Hard limits:
|
||||
|
||||
- `setPermissionGrantState()` for **sensor** permissions (location, camera, mic, body sensors,
|
||||
activity recognition) is restricted/deprecated for *profile* owners on Android 12+ (silently
|
||||
returns false); device owners retain more but the surface is shrinking each release.
|
||||
- It cannot read other apps' `/data/data`, cannot image partitions, and cannot reach EFS.
|
||||
|
||||
If you find yourself reaching for device-owner to extract data, the real answer is almost always
|
||||
root (§4) or ADB providers (§6.2) instead.
|
||||
|
||||
---
|
||||
|
||||
## 6. ADB & Shizuku — no root, device boots
|
||||
|
||||
Plain ADB runs as the `shell` user (UID 2000). Shizuku grants that **same** privilege to on-device
|
||||
apps (started via wireless debugging or a one-time ADB command), so their reach is identical to ADB
|
||||
— handy when you have no PC.
|
||||
|
||||
### 6.1 External storage & scoped app dirs
|
||||
|
||||
```bash
|
||||
adb pull /sdcard/ ./sdcard-backup/ # DCIM, Download, Documents, etc. — works everywhere
|
||||
# Android/data and Android/obb are blocked from on-device file managers (11/13+) but
|
||||
# the adb shell UID can still read them:
|
||||
adb pull /sdcard/Android/data/ ./android-data/
|
||||
adb pull /sdcard/Android/obb/ ./android-obb/
|
||||
```
|
||||
|
||||
### 6.2 Content providers (telephony/PIM without root)
|
||||
|
||||
`adb shell content query` reaches providers the `shell` UID is allowed to read. What actually works
|
||||
on a stock device is uneven:
|
||||
|
||||
```bash
|
||||
adb shell content query --uri content://contacts/phones # contacts — usually readable
|
||||
adb shell content query --uri content://com.android.calendar/events
|
||||
# SMS/MMS and call_log generally FAIL for the shell UID (need READ_SMS / be default app):
|
||||
adb shell content query --uri content://sms # typically permission denied
|
||||
```
|
||||
|
||||
For SMS/MMS, call logs, and contacts as restorable files, use an on-device FOSS app rather than
|
||||
fighting providers over ADB:
|
||||
|
||||
- **SMS Import / Export** (`com.github.tmo1.sms_ie`, F-Droid) — SMS+MMS+call log+contacts to
|
||||
JSON/ZIP, no root, and it *restores* too (set it as default SMS app during restore).
|
||||
- **Open Android Backup** (mrrfv) — PC-driven, rootless, uses a companion app (not `adb backup`):
|
||||
restores apps-without-data, internal storage, and contacts (vCard); exports SMS (CSV) and call
|
||||
log (txt) as **view-only** (no restore, no MMS attachments).
|
||||
- Contacts also export cleanly from the stock Contacts app ("Export to .vcf") or via Google account
|
||||
sync — often the least-effort path for the non-technical case.
|
||||
|
||||
### 6.3 `run-as` — app data for *debuggable* apps only
|
||||
|
||||
```bash
|
||||
adb shell run-as com.example.debug tar -cpf - ./ > example-data.tar
|
||||
```
|
||||
|
||||
`run-as` only works for apps with `android:debuggable="true"` (i.e. your own dev builds, not Play
|
||||
Store releases). Shizuku-based file managers use the same `run-as` path and inherit the same limit.
|
||||
Release-app `/data/data` is **not** reachable at this tier — that needs root (§4) or a
|
||||
recovery-decrypt (§3).
|
||||
|
||||
### 6.4 Bugreport & app lists (cheap, always worth grabbing)
|
||||
|
||||
```bash
|
||||
adb bugreport ./bugreport.zip # logs, settings dump, installed-package metadata
|
||||
adb shell pm list packages -3 -f # third-party packages + APK paths, to reinstall later
|
||||
adb shell cmd package list packages --show-versioncode
|
||||
```
|
||||
|
||||
To re-pull the APKs themselves for reinstall:
|
||||
|
||||
```bash
|
||||
adb shell pm path com.example # -> package:/data/app/.../base.apk + splits
|
||||
adb pull /data/app/~~.../com.example-.../base.apk # repeat for each split; reinstall with
|
||||
adb install-multiple base.apk split_config.*.apk # install-multiple
|
||||
```
|
||||
|
||||
### 6.5 Vendor D2D and cloud (no-tech last mile)
|
||||
|
||||
- **Google** cloud backup + Pixel/Android cable "device-to-device" restore at setup. App data rides
|
||||
along *only* if the Google **account** is transferred (Block Store is account-bound); untick
|
||||
nothing under "Accounts". `allowBackup="false"` stops cloud backup but D2D still transfers app data
|
||||
unless the app sets `dataExtractionRules`/`android:fullBackupOnly` to opt out.
|
||||
- **Samsung Smart Switch** (PC or cable): backs up most app data, but **encrypted app data (e.g.
|
||||
WhatsApp), banking logins, and sideloaded APKs do not transfer** — handle those per §9/§10.
|
||||
- **Seedvault** (CalyxOS/GrapheneOS/LineageOS): the FOSS backup transport. On Android 15 builds its
|
||||
**device-to-device** mode (Expert settings) is default-on and captures far more app data than the
|
||||
old cloud-style transport, because it presents as a D2D migration to each app.
|
||||
|
||||
---
|
||||
|
||||
## 7. The death of `adb backup` (don't build a plan on it)
|
||||
|
||||
`adb backup` was deprecated in Android 10 and gutted in Android 12 (API 31): app data is excluded by
|
||||
default and only included if the app is **debuggable**, so on essentially all release apps across
|
||||
Android 12–16 `adb backup` produces an archive with no app data. Treat it as unavailable:
|
||||
|
||||
- Don't rely on it for a migration. Use root tooling (§4.1), on-device FOSS exporters (§6.2), or
|
||||
vendor/Seedvault D2D (§6.5) instead.
|
||||
- `adb restore` likewise cannot restore encrypted backups and is a dead end for modern data.
|
||||
- The `bmgr`/LocalTransport backup-manager plumbing still exists for *developers testing their own
|
||||
backup agents* (`adb shell bmgr …`), but it is a per-app developer tool, not a device-wide
|
||||
extractor — out of scope for a pre-wipe sweep.
|
||||
|
||||
---
|
||||
|
||||
## 8. EDL / firehose / BROM — dead or bootloader-locked devices
|
||||
|
||||
The lowest level, for devices that will not boot to Android *or* recovery. These talk to the SoC's
|
||||
emergency loader below the OS:
|
||||
|
||||
- **Qualcomm EDL (9008)** — [`bkerler/edl`](https://github.com/bkerler/edl) with a firehose
|
||||
programmer (`.mbn`/`.elf`) matching the SoC. `edl printgpt` lists partitions; `edl rl dumps
|
||||
--skip=userdata --genxml` dumps all partitions (minus the huge userdata) to a folder with a
|
||||
`rawprogram` XML; `edl r <partition> <out.img>` grabs one. Newer SoCs need `--loader=<file>`
|
||||
because loader autodetection no longer works.
|
||||
- **MediaTek BROM** — [`bkerler/mtkclient`](https://github.com/bkerler/mtkclient). `./mtk printgpt`,
|
||||
`./mtk rl out` dumps all partitions to `out/`, `./mtk r <partition> <out.img>` for one; `--preloader`
|
||||
for BROM mode.
|
||||
|
||||
The hard limit: **these dumps are ciphertext.** EDL reads the raw flash below the OS, so `userdata`
|
||||
comes out FBE-encrypted and is not decryptable without the keys (which live in secure hardware keyed
|
||||
by the lockscreen credential). EDL is therefore excellent for **small identity partitions**
|
||||
(EFS/persist/nvram — §4.3), full-flash forensic images, and un-bricking, but it is **not** a way to
|
||||
read a locked phone's app data. Treat downloaded dumps/tools as untrusted input: keep each in its own
|
||||
directory, pass paths as arguments, and run any helper scripts from outside the dump directory.
|
||||
|
||||
---
|
||||
|
||||
## 9. App-specific data that needs its own handling
|
||||
|
||||
Some high-value data lives behind app-level encryption and ignores the tier ladder:
|
||||
|
||||
- **WhatsApp** — local backup at `/sdcard/WhatsApp/Databases/msgstore.db.crypt15` (+ `/Media`). If
|
||||
the user enabled **end-to-end encrypted backups**, decryption needs the **64-digit key** (or the
|
||||
`encrypted_backup.key` file from `/data/data/com.whatsapp/files/`, root-only). Tools:
|
||||
[`wa-crypt-tools`](https://pypi.org/project/wa-crypt-tools/). Easiest migration is WhatsApp's own
|
||||
built-in transfer or Google-Drive backup with the same phone number — pull the `crypt15` + media
|
||||
as insurance, not as the primary path.
|
||||
- **Signal** — does **not** sync to any cloud. Migrate with Signal's in-app device-transfer, or its
|
||||
encrypted local backup (Settings → Chats → Backups) whose **30-digit passphrase you must record** —
|
||||
there is no recovery without it.
|
||||
- **Authenticator / 2FA** — Google Authenticator exports via its QR "export accounts" flow;
|
||||
**Aegis** (FOSS) exports an encrypted JSON vault. Do this *before* wiping; TOTP seeds not exported
|
||||
are gone.
|
||||
|
||||
---
|
||||
|
||||
## 10. What will NOT survive a wipe (plan around it)
|
||||
|
||||
No tier recovers these; the only mitigation is to migrate them *before* the wipe:
|
||||
|
||||
- **Hardware-backed Keystore keys** are non-exportable by design — key material never leaves the
|
||||
TEE/StrongBox. Any app that pinned a device-bound key (banking apps, some password managers,
|
||||
FIDO/passkeys stored in hardware) will treat the restored/new device as untrusted and re-enrol.
|
||||
- **Play Integrity / SafetyNet attestation** fails on rooted or custom-ROM devices independently of
|
||||
your data — so a perfectly-restored banking app may still refuse to run. That is an attestation
|
||||
problem, not a backup problem; don't chase it as data loss.
|
||||
- **SSAID / `ANDROID_ID`** changes on wipe (per app-signing-key + user), which can log apps out or
|
||||
reset entitlements even when their `/data/data` is restored byte-perfect.
|
||||
|
||||
Before wiping: export 2FA (§9), confirm password-manager vaults sync to their cloud, record Signal
|
||||
(§9) and WhatsApp keys, and re-enrol passkeys/banking on the new device rather than expecting a
|
||||
restore to carry them.
|
||||
|
||||
---
|
||||
|
||||
## 11. Pre-wipe checklist (the one-screen version)
|
||||
|
||||
1. **Decide the tier** (§1). Booting + unlockable → root (§4) or ADB (§6). Not booting → recovery
|
||||
(§3) → EDL (§8).
|
||||
2. **EFS/modem/persist first** (§4.3) — before any unlock or flash, off-device.
|
||||
3. **Media & docs** — `adb pull /sdcard` (§6.1).
|
||||
4. **App data** — root tar + dedicated tool (§4.1), or recovery-decrypted Data backup (§3).
|
||||
5. **Scoped dirs** — `Android/data`, `Android/obb` (§6.1).
|
||||
6. **PIM** — SMS/call-log/contacts via SMS Import-Export or vCard (§6.2).
|
||||
7. **App-specific** — WhatsApp/Signal/2FA via their own flows (§9).
|
||||
8. **App list + APKs** — `pm list packages -3`, pull splits (§6.4) to rebuild the app set.
|
||||
9. **System state the app tars miss** (§12.1) — keystore DB, SSAIDs, URI grants, doze whitelist,
|
||||
app-ops, launcher/shortcuts/QS/controls, Wi-Fi, SD-card UUIDs, root grants, old `packages.list`.
|
||||
10. **Baseline + app-sweep** the old ROM (`measure.sh`, `test-all-apps.sh`, §12.4) so the new one
|
||||
has something to be compared against.
|
||||
11. **Passwords apps keep themselves** — App Manager's KeyStore password, backup-app passphrases,
|
||||
Signal/WhatsApp keys (§9). Write them down *before* the wipe; none are recoverable after.
|
||||
12. **Verify the backups off-device** (open the archive, check `ls /data/data` really decrypted)
|
||||
*before* you type `fastboot -w` or confirm the reset. A backup you have not read back is not a
|
||||
backup.
|
||||
|
||||
---
|
||||
|
||||
## 12. Restoring onto a new ROM (root) — what a data tar does not carry
|
||||
|
||||
Learned from a full LineageOS-based A16 → Infinity-X A17 switch (Redmi Note 7, Magisk). Restoring
|
||||
`/data/data` + `user_de` + `Android/data` per §4.4 brings back most apps, but **every app gets a new
|
||||
UID** on the new ROM, and a lot of state is keyed by UID or lives in system files. The scripts in
|
||||
[`scripts/android/`](skills-public/scripts/android/) automate the parts below; they take the
|
||||
device from `ANDROID_SERIAL` like `adb` itself.
|
||||
|
||||
### 12.1 Grab these too, before the wipe (all root, all small)
|
||||
|
||||
| What | Where | Why it matters on restore |
|
||||
| :--- | :--- | :--- |
|
||||
| Old UID map | `/data/system/packages.list` | Remap everything keyed by UID (SSAID, keystore, grants) |
|
||||
| Keystore2 DB | `/data/misc/keystore/persistent.sqlite` | Apps whose keys vanish show "Keystore error"/log out (§12.3) |
|
||||
| SSAIDs | `/data/system/users/0/settings_ssaid.xml` | `ANDROID_ID` change logs apps out (GitHub, Bolt, Lovense, …) |
|
||||
| Persistent SAF grants | `/data/system/urigrants.xml` | Folder pickers (music players, camera, sync apps) lose their folders |
|
||||
| Doze whitelist | `/data/system/deviceidle.xml` | Notification/automation apps silently stop in background |
|
||||
| App-ops & special access | `cmd appops dump`, `cmd notification` policy access, `dumpsys package` | `MANAGE_EXTERNAL_STORAGE`, usage stats, DND, `READ_LOGS`, install-packages, notification listeners are **not** runtime permissions |
|
||||
| Launcher + pinned shortcuts | launcher DB, `/data/system_ce/0/shortcut_service/` | Home screen; pinned shortcuts (e.g. Ice Box) are owned by the system, not the launcher |
|
||||
| QS tiles, device controls | `settings get secure sysui_qs_tiles`; SystemUI `user_de` `files/controls_favorites.xml` + `shared_prefs/controls_prefs.xml` | Home Assistant controls, dev tiles |
|
||||
| Wi-Fi | `/data/misc/apex/com.android.wifi/` or `cmd wifi list-networks` + PSKs | Also note which SSIDs must **not** auto-join |
|
||||
| Root grants + Magisk settings | `magisk --sqlite "SELECT * FROM policies"`, `… FROM settings`, `… FROM denylist`, `/data/adb/modules` (note which are disabled) | UIDs change, grants don't follow; Zygisk off ⇒ DenyList silently does nothing |
|
||||
| Accounts | `dumpsys account`; `/data/system_ce/0/accounts_ce.db` | Shows which logins (2nd Google account, app-registered accounts) must be re-added by hand — they can't be transplanted |
|
||||
| SD-card volume IDs | `blkid` of each SD partition | Public volume ID = filesystem UUID; reformat ⇒ new ID ⇒ broken paths and grants |
|
||||
| Freezer state | `pm list packages -d` + the freezer app's own list | Updates/reinstalls unfreeze |
|
||||
|
||||
### 12.2 Restore order and traps
|
||||
|
||||
- **Update first, restore data second.** Installing the old APKs and then the store's update is
|
||||
fine; restoring data *into* an older APK than the data was written by is a downgrade — apps wipe
|
||||
or crash on it. Never re-run a full restore pass over updated apps; restore data only.
|
||||
- **Installer of record:** `pm set-installer` refuses even root. Reinstall the same APKs with
|
||||
`adb install-multiple -r -i <installer>` (keeps data) so Play/Aurora/F-Droid/Obtainium see their
|
||||
apps as updatable (`fix-installers.py`).
|
||||
- **New signature, same package** (forks such as Shizuku → Shizuku-Next for a newer Android): tar the
|
||||
data dirs, uninstall, install, untar, chown/chcon, re-grant, re-freeze — `reinstall-keep-data.sh`.
|
||||
- **System files are read only at `system_server` start.** Write `urigrants.xml`,
|
||||
`settings_ssaid.xml`, `shortcut_service/*` etc. (owner `system:system`, mode 600,
|
||||
`u:object_r:system_data_file:s0`), then `kill -9 $(pidof system_server)` — a soft restart that
|
||||
does **not** save over them first, which a normal reboot does. It locks the user: unlock with
|
||||
wake → swipe → `input text <PIN>` → ENTER (the bouncer screenshots black; retry if needed).
|
||||
- **fs-verity:** files under `shortcut_service` are verity-protected — `rm` then write, never edit.
|
||||
- **Restored caches can kill an app:** Ice Box OOM-crashed on a restored MMKV package cache. Drop
|
||||
`cache`, `code_cache`, WebView caches and per-app package-list caches; keep the real state.
|
||||
- **Re-map UID-keyed state** with the old/new `packages.list` (SSAID `name="<uid>"`, keystore
|
||||
`namespace`, Magisk `policies.uid`, doze lists are by package and need no remap).
|
||||
- **SD card:** if the partition had to be recreated, set the old UUID back
|
||||
(`tune2fs -U <old>` for ext4, `fsck` before and after) rather than fixing every app.
|
||||
- **Wi-Fi:** restored networks auto-join in their own order; re-add the ones that must stay
|
||||
manual with `cmd wifi add-network <ssid> wpa2 <psk> -d`.
|
||||
- **QS tiles** on A17: `settings put secure sysui_qs_tiles` is ignored; use
|
||||
`cmd statusbar set-tiles "<list>"`. Developer tiles must be enabled first
|
||||
(`pm enable 'com.android.settings/.development.qstile.DevelopmentTiles$SensorsOff'`).
|
||||
- **Launcher switch** (Nova had no build for the new Android): converting Nova's `favorites`
|
||||
table into Lawnchair's Launcher3 schema works (`nova-to-lawnchair.py`) — mark widgets
|
||||
`restored=1` so Launcher3 binds fresh widget IDs, and re-pin deep shortcuts for the new launcher.
|
||||
|
||||
**Banking-app root checks** are not a restore problem, but they show up right after one. Order
|
||||
that worked (2026-10, Magisk 31, A17): first the ROM's own spoofing settings, if it has any (PIF,
|
||||
TrickyStore target and keybox, hide developer status). If an app still crashes on purpose, look for
|
||||
an Xposed module that defuses that app's check. For PayPal 10.18 that was oslokrok on Vector, the
|
||||
LSPosed successor. Configure Vector headlessly with
|
||||
`/data/adb/modules/zygisk_vector/cli modules enable <pkg>` and `cli scope set <pkg> <app>/0`.
|
||||
**Take the app off the Magisk DenyList**: the DenyList strips every Zygisk module from the process,
|
||||
including the hook.
|
||||
|
||||
### 12.3 Keystore: transplanting app keys
|
||||
|
||||
Hardware-backed keys never leave the TEE, but keystore2 stores them **wrapped** in its SQLite DB.
|
||||
If the new ROM runs on the same TEE (compare `keyentry.km_uuid` old vs new — same device, same
|
||||
firmware ⇒ same), the old wrapped blobs still unwrap. Keys whose `blobmetadata` has tag 0
|
||||
(`EncryptedBy`) are additionally super-encrypted with the **old lock-screen credential's** key and
|
||||
cannot be moved. Domain 0 (APP) rows have `namespace = uid` → remap. Stop the apps, back up the live
|
||||
DB, insert `keyentry`/`blobentry`/`blobmetadata`/`keyparameter`/`keymetadata` with fresh ids, reboot.
|
||||
Apps that already regenerated keys on the new ROM also need their data re-restored to match.
|
||||
`scripts/android/keystore-transplant.py OLD_DB [--replace] OLD_UID:NEW_UID …` writes the SQL; dry-run
|
||||
it on a host copy of the live DB (`pragma integrity_check`) before applying with the device's own
|
||||
`sqlite3`. Tested: transplanted keys load with no keystore errors after a reboot. That does not
|
||||
bring back sessions the server ended, or apps that refuse to run because their build is too old.
|
||||
|
||||
### 12.4 Verify like a user would
|
||||
|
||||
- `test-all-apps.sh OUT` cold-starts every user app (unfreezing and re-freezing), screenshots it,
|
||||
and greps its log for crashes, ANRs and keystore/crypto errors → `results.tsv`. Look at every
|
||||
screenshot: logouts, licence checks, ToS/"what's new" dialogs and permission prompts don't crash.
|
||||
- Android 17 shell launches must match the activity's intent filter:
|
||||
`am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n 'pkg/act'`
|
||||
(single-quote it — component names contain `$`). `-n` alone gives "Access blocked".
|
||||
- `measure.sh OUT [pkg…]` snapshots memory, crashes/ANRs/lmk kills since boot and cold-launch
|
||||
times; run it on the old ROM before the wipe so there is a baseline.
|
||||
- Keep a running log of every security setting weakened during the restore (adb stay-awake, root
|
||||
grants, doze exemptions, …) with its revert command; the restore needs many of them and nobody
|
||||
remembers them afterwards.
|
||||
+141
-3
@@ -4,8 +4,10 @@ description: >-
|
||||
Use this skill whenever invoking a command-line tool, choosing between a CLI and an ad-hoc
|
||||
script, writing a standalone utility script, or orienting on unfamiliar work before starting.
|
||||
Covers discovering a CLI's real interface instead of guessing flags, non-interactive and
|
||||
machine-readable invocation, script placement and symlinking, and care around destructive
|
||||
or outward-facing commands.
|
||||
machine-readable invocation, checking an API's payload rather than its exit status, keeping
|
||||
credentials out of argv, eventually-consistent systems that briefly report the old state,
|
||||
script placement and symlinking, and care around destructive or outward-facing commands —
|
||||
including additive changes that silently take over a name already serving traffic.
|
||||
---
|
||||
|
||||
# CLI & Tooling — Discovery, Invocation, and Script Placement
|
||||
@@ -49,6 +51,24 @@ Before writing a script to do something, check whether:
|
||||
- the tool you are already using has a subcommand for it,
|
||||
- the project's own `scripts/` directory already contains it.
|
||||
|
||||
### Desktop Steam Client Lifecycle (`steamcli`)
|
||||
|
||||
> [!TIP]
|
||||
> **Never spawn Steam via raw shell backgrounding (`nohup steam &` or `steam -silent &`) or kill it with `pkill`.**
|
||||
> On desktop Linux/Bazzite, raw spawning leaves orphaned bash wrappers, misses custom client flags, or fails to initialize properly.
|
||||
> Instead, manage the desktop Steam client through **`steamcli client`**:
|
||||
> ```bash
|
||||
> # Graceful exit and wait for it to finish flushing configuration:
|
||||
> steamcli client shutdown
|
||||
>
|
||||
> # Launch the desktop client cleanly with configured defaults:
|
||||
> steamcli client launch
|
||||
>
|
||||
> # Launch with custom Steam arguments:
|
||||
> steamcli client launch -- -silent
|
||||
> ```
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 2. Non-Interactive Invocation
|
||||
@@ -83,6 +103,16 @@ tool list -o csv # structured columns for a spreadsheet or script
|
||||
|
||||
- **Never scrape a table meant for humans.** Column widths, truncation, colour codes, and
|
||||
alignment all change without notice; a structured format is a contract.
|
||||
- **Check the payload, not just the exit status.** Plenty of HTTP APIs answer `200 OK` with
|
||||
`{"success": false, "errors": [...]}` in the body, and `curl` exits `0` because the
|
||||
*request* worked. A script that trusts the exit code reports a clean success and writes
|
||||
nothing. Assert on the field the API uses for this, and print `errors[]` when it trips:
|
||||
```bash
|
||||
if [[ "$(jq -r '.success' <<<"$response")" != true ]]; then
|
||||
jq -r '.errors[]? | " - \(.message)"' <<<"$response" >&2
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
- Strip colour explicitly (`--color never`, `NO_COLOR=1`) when capturing output, or escape
|
||||
sequences end up in your parsed values.
|
||||
- Keep in mind that a "compact" or summary format is the right default for feeding an agent,
|
||||
@@ -166,9 +196,117 @@ system, not scattered at the repository root.
|
||||
- Anything that sends, publishes, deploys, or purchases is outward-facing: confirm first, and
|
||||
treat approval as applying to that one action rather than the category.
|
||||
|
||||
### Adding something can take something over
|
||||
|
||||
> [!CAUTION]
|
||||
> **A change that deletes nothing can still break a live service.** The dangerous operations
|
||||
> are not only the ones named `delete`. Adding a routing rule, a proxy entry, a redirect, a
|
||||
> DNS override or a higher-priority config can silently intercept a name that is already
|
||||
> serving traffic — while the thing it shadows still looks perfectly healthy in every
|
||||
> listing, because it was never touched.
|
||||
>
|
||||
> That is the worst shape a failure can take: the old config is still there, still correct,
|
||||
> still pointing where it should, and the service is dead anyway. Nothing in the inventory
|
||||
> says what happened.
|
||||
|
||||
**Before claiming a name, ask what currently answers to it** — and refuse by default when
|
||||
something does:
|
||||
|
||||
```bash
|
||||
# Only an unrouteable placeholder, or an entry this tool made, is safe to assume is free.
|
||||
if [[ "$(current_target "$host")" != "$PLACEHOLDER" && "$(owner "$host")" != "$MARKER" ]]; then
|
||||
[[ "$FORCE" == true ]] || die "$host already serves $(current_target "$host") — pass --force"
|
||||
fi
|
||||
```
|
||||
|
||||
Two things make that check possible:
|
||||
|
||||
- **Mark what you create.** A comment field, a tag, a description prefix — anything that
|
||||
distinguishes *your* entries from ones a human made. It turns "is this mine to replace?"
|
||||
into a lookup rather than a guess, and it is what lets the matching `--delete` refuse to
|
||||
remove something it did not create.
|
||||
- **Make `--force` the only way past**, and name in the refusal what is there now. A tool that
|
||||
silently overwrites is one nobody can safely run against production.
|
||||
|
||||
> [!TIP]
|
||||
> Test such a tool against a name you **expect** it to reject, not only against a free one.
|
||||
> The rejection path is where the damage would have been, and it is the half a happy-path test
|
||||
> never exercises.
|
||||
|
||||
---
|
||||
|
||||
## 7. Environment Constraints
|
||||
## 7. Credentials Belong on stdin, Not in `argv`
|
||||
|
||||
> [!WARNING]
|
||||
> **`argv` is world-readable.** Any user on the box can run `ps` and read the full command
|
||||
> line of every process. A token passed as `-H "Authorization: Bearer $TOKEN"`, `--password`,
|
||||
> or `--token` is therefore exposed for as long as the call runs — and lands in shell history
|
||||
> and process accounting too.
|
||||
|
||||
Most CLIs that accept a secret as a flag also accept it somewhere safer. Use that instead:
|
||||
|
||||
```bash
|
||||
# WRONG — the token is visible in `ps` for the life of the request
|
||||
curl -H "Authorization: Bearer $TOKEN" "$url"
|
||||
|
||||
# RIGHT — curl reads its options from stdin; argv holds nothing sensitive
|
||||
printf 'header = "Authorization: Bearer %s"\n' "$TOKEN" | curl --config - "$url"
|
||||
```
|
||||
|
||||
The equivalent elsewhere: `--password-stdin` (docker login, many package tools), `-u` with a
|
||||
`.netrc`, `--token-file`, or simply an environment variable the tool reads by name. Reach for
|
||||
the env var when nothing better exists — it is readable through `/proc/<pid>/environ` but only
|
||||
by the owner and root, which is a meaningfully smaller exposure than `argv`.
|
||||
|
||||
See [`code-languages/bash.md` §8](skills-public/code-languages/bash.md) for the surrounding rules on
|
||||
where secrets are stored and why they are never echoed.
|
||||
|
||||
---
|
||||
|
||||
## 8. Eventually-Consistent Systems Lie Briefly
|
||||
|
||||
Edge networks, DNS, CDNs, object stores, container registries and clustered schedulers all
|
||||
acknowledge a write before it is visible everywhere. **A read taken immediately after a
|
||||
successful write is not evidence of anything.**
|
||||
|
||||
The failure mode is believing the tool is broken and "fixing" it:
|
||||
|
||||
- A changed rule still returns the **old** value for a while — which reads as "my write was
|
||||
ignored".
|
||||
- A brand-new name returns an **error** for a while — a 5xx, an NXDOMAIN, a 404 — which reads
|
||||
much worse than it is.
|
||||
|
||||
So: take the write's own success response as the result, and treat the first few reads as
|
||||
unreliable. When a verification read matters, **say how long propagation takes** in the tool's
|
||||
own help or header comment, so the next person does not rediscover it at 2am. State it as a
|
||||
number — "~30s for a rule, ~40s for a new name" — because "eventually" tells nobody when to
|
||||
start worrying.
|
||||
|
||||
Do not wrap this in a polling loop (see
|
||||
[`code-languages/bash.md` §3](skills-public/code-languages/bash.md)); take one sample per
|
||||
invocation and let the caller run it again.
|
||||
|
||||
> [!WARNING]
|
||||
> **Checking too early can make the delay much worse.** A failed lookup is itself cached. Query
|
||||
> a DNS name before it exists and the resolver stores the `NXDOMAIN` for the SOA's negative TTL
|
||||
> — commonly 300-1800s — so the name stays invisible to *you* long after it is live for
|
||||
> everyone else. Polling during propagation is what creates this, which is a second reason not
|
||||
> to loop.
|
||||
>
|
||||
> When a check does come back negative, confirm where the negative lives before concluding
|
||||
> anything: query the authoritative source directly and compare. The answer is often "the
|
||||
> record is fine, my resolver is lying", and flushing the local cache will not help if the
|
||||
> stale entry is on an upstream resolver you do not control.
|
||||
>
|
||||
> ```bash
|
||||
> dig +short @1.1.1.1 "$name" # authoritative-ish: is it actually published?
|
||||
> dig +short "$name" # what my resolver believes
|
||||
> curl --resolve "$name:443:<ip>" "https://$name/" # bypass DNS entirely to test the service
|
||||
> ```
|
||||
|
||||
---
|
||||
|
||||
## 9. Environment Constraints
|
||||
|
||||
- **Entering a container or changing shells resets the working directory.** Chain `cd` inside
|
||||
the same command string rather than relying on it persisting:
|
||||
|
||||
@@ -0,0 +1,262 @@
|
||||
---
|
||||
name: gsc
|
||||
description: >-
|
||||
Use this skill when writing, reviewing, or debugging GSC (Game Script / Call of Duty script)
|
||||
or CSC — including entity-vs-level scope, threads and notify/wait semantics, endon/waittill
|
||||
discipline, undefined-value guards, the no-OOP/no-exceptions reality, asset precaching,
|
||||
per-frame cost, block scoping, and compiling off-engine with gsc-tool.
|
||||
---
|
||||
|
||||
# GSC — Rules & Preferences
|
||||
|
||||
Extends [`coding-dev.md`](skills-public/coding-dev.md). Everything there applies; this adds GSC specifics.
|
||||
|
||||
GSC is the Call of Duty engine's own scripting language (`.gsc` server-side, `.csc`
|
||||
client-side). It is **not Lua, not JavaScript, and not C** despite resembling all three. It has
|
||||
no type system, no exceptions, no classes, no closures in most dialects, and no test harness.
|
||||
It can be compiled off-engine (see §1), which catches syntax and scope errors — but nothing
|
||||
checks a type, a field that was never set, or a builtin that does not exist in the target title.
|
||||
So the code still has to be written so that it cannot be wrong.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Establish the dialect before writing anything.** IW3/IW4 (MW/MW2), IW5 (MW3), T5/T6 (Black
|
||||
> Ops 1/2), IW6, s1/s2, H1/H2 (MWR / MW2CR), and the various mod loaders (h2-mod, iw4x, plutonium,
|
||||
> t6 GSC Studio) each differ in available builtins, in how scripts get loaded, and in whether
|
||||
> `foreach`, `switch` fallthrough, or function pointers exist. A builtin from the wrong title is
|
||||
> a silent `undefined` or a hard crash.
|
||||
|
||||
---
|
||||
|
||||
## 1. The Gate — `gsc-tool`, Then The Game
|
||||
|
||||
**There is an off-engine GSC compiler: `gsc-tool`.** Reach for it first; it is
|
||||
easy to assume none exists and verify by loading a map instead.
|
||||
|
||||
```bash
|
||||
gsc-tool --mode comp --game <title> --system pc --dry path/to/script.gsc
|
||||
```
|
||||
|
||||
It exits non-zero and prints the engine's own diagnostic, e.g.
|
||||
`[ERROR]:compiler:script:97:25: local variable 'previous_count' not found`.
|
||||
|
||||
- **Use `--mode comp`, not `--mode parse`.** `parse` is syntax-only and happily
|
||||
accepts code the engine's compiler rejects, including the block-scope error
|
||||
above. `comp` with `--dry` writes nothing and does the real thing.
|
||||
- **`--game` must match the title**, and it is not the marketing name: MW2
|
||||
Campaign Remastered is `h2`, MWR is `h1`. The choices are `iw5 iw6 iw7 iw8
|
||||
iw9 s1 s2 s4 h1 h2 t6 t7 t8 t9 jup`. Wrong title, wrong builtin table.
|
||||
- Prebuilt binaries ship on its GitHub releases; there is no distro package.
|
||||
- Many mod trees already vendor it (Overlord has it under `deps/gsc-tool`).
|
||||
|
||||
What that still leaves:
|
||||
|
||||
| Check | How |
|
||||
| :--- | :--- |
|
||||
| **Syntax + semantics** | `gsc-tool --mode comp`. Put it in the build gate. |
|
||||
| **Builtin exists for this title** | `gsc-tool`'s own engine tables, e.g. `src/gsc/engine/h2_meth.cpp`. Grep before using a builtin you remember from another title. |
|
||||
| **Runtime errors** | In-game console / `developer 1`. Many mod loaders print a script stack trace. |
|
||||
| **Tests** | None off-host. Behaviour is verified by playing the map. |
|
||||
|
||||
So a compile is cheap and mandatory, and **behaviour is still unverified until
|
||||
the map runs.** Both halves matter:
|
||||
|
||||
- **`developer 1` (or the loader's equivalent) is mandatory during development.** Without it the
|
||||
engine suppresses script warnings and an undefined-value error looks like "the feature did
|
||||
nothing".
|
||||
- **Load the script and read the console on every change**, before judging behaviour. "Nothing
|
||||
happened" is far more often a parse error at load than a logic bug.
|
||||
- **Instrument with `iprintln` / `iprintlnbold` / `println`** on the success *and* the failure
|
||||
path. A function that silently returns on a guard is indistinguishable from one that never ran.
|
||||
- **Say plainly in your report that GSC behaviour is unverified** unless you actually loaded the
|
||||
map and watched it. A clean `gsc-tool` compile proves the script loads, not that it works.
|
||||
|
||||
---
|
||||
|
||||
## 2. Scope — `self`, `level`, `game`, and Locals
|
||||
|
||||
| Scope | Lifetime | Use for |
|
||||
| :--- | :--- | :--- |
|
||||
| local variable | the function call | everything it can hold |
|
||||
| `self` | the entity | per-entity state (a player's, an actor's) |
|
||||
| `level` | the current map/level | per-round state, map config, shared handles |
|
||||
| `game` | across map restarts | persistent state only |
|
||||
|
||||
- **Prefix your own fields.** `level.mymod_spawn_queue`, not `level.queue`. `level` and `self`
|
||||
are one flat namespace shared with the stock game scripts and every other mod; a collision
|
||||
overwrites engine state and the symptom appears somewhere unrelated.
|
||||
- **Never write to a stock field you did not create** unless that is the documented integration
|
||||
point. Read it, copy it, leave it.
|
||||
- **A function called *on* an entity gets it as `self`**: `player giveweapon("m4")`. Calling the
|
||||
same function without a prefix leaves `self` undefined and the first `self.x` access errors.
|
||||
- `self` is not inherited into a `thread`ed call's children automatically in the way callers
|
||||
expect — `ent thread f()` gives `f` `self == ent`, but a plain `f()` inside it does not.
|
||||
|
||||
---
|
||||
|
||||
## 3. Undefined Is The Default — Guard Everything External
|
||||
|
||||
GSC has no types and no nullability. **Reading a field that was never set yields `undefined`,
|
||||
and using it in arithmetic or a method call is a runtime error that aborts the thread** — often
|
||||
silently from the player's point of view.
|
||||
|
||||
```gsc
|
||||
// WRONG: any of these can be undefined
|
||||
ent = level.target;
|
||||
ent setorigin(level.player.origin + (0, 0, 64));
|
||||
|
||||
// RIGHT
|
||||
if (!isdefined(level.target) || !isdefined(level.player))
|
||||
return;
|
||||
```
|
||||
|
||||
- **`isdefined()` on every entity, every `level.*`/`self.*` read you did not write in the same
|
||||
function, and every return value from a lookup.**
|
||||
- **Entities die.** A handle held across a `wait` may be a dead entity by the time the thread
|
||||
resumes. Re-check `isdefined(ent)` *after* every `wait`, not only before.
|
||||
- `isalive(ent)` is the check for an actor you intend to act on; `isdefined` only proves the
|
||||
handle exists.
|
||||
- This is the one place where the general "don't over-guard" rule inverts: in GSC a value from
|
||||
outside the current function genuinely can be undefined, so the check is not dead code.
|
||||
- Arrays: `array.size` is safe on an empty array but errors on an undefined one. Guard the array,
|
||||
then the index.
|
||||
|
||||
---
|
||||
|
||||
## 4. Threads, `notify`, `waittill`, `endon`
|
||||
|
||||
```gsc
|
||||
monitor_thing()
|
||||
{
|
||||
level endon("game_ended");
|
||||
self endon("death");
|
||||
|
||||
for (;;)
|
||||
{
|
||||
self waittill("trigger_activated");
|
||||
do_work();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- **`thread` starts a coroutine and returns immediately.** It does not return the function's
|
||||
value — a `thread`ed function's `return` value is unreachable. Pass a handle or notify.
|
||||
- **Every long-lived thread needs an `endon`.** Without one, a loop keeps running after the
|
||||
entity dies or the level ends, and then errors on a dead `self` — or leaks, one copy per spawn.
|
||||
`level endon("game_ended")` plus the relevant entity `endon` is the baseline.
|
||||
- **`endon` must be registered before the first `wait`**, at the top of the function. Registering
|
||||
it later leaves a window.
|
||||
- **`waittill` is the event primitive — prefer it to polling.** `self waittill("death")` is
|
||||
correct; `while (isalive(self)) wait 0.05;` is the same thing, worse, and costs every frame
|
||||
forever. This is the general events-over-polling rule, and GSC gives you the event.
|
||||
- **A `notify` with no waiter is lost.** There is no queue. If a thread might not be waiting yet,
|
||||
set a flag as well as notifying, and have the waiter check the flag first.
|
||||
- `waittill("x", a, b)` receives the notify's extra parameters. The names are positional.
|
||||
- **`waittillframeend`** is the tool for "after everything else this frame" — use it instead of
|
||||
`wait 0.05` when the intent is ordering, not delay.
|
||||
- `wait 0` is not zero; it is one frame. `wait 0.05` is one server frame at 20Hz — write the
|
||||
intent, not a magic number, and name the constant.
|
||||
|
||||
---
|
||||
|
||||
## 5. Per-Frame Cost Is Your Budget
|
||||
|
||||
The script VM shares the server frame with everything else, and there is a **hard instruction
|
||||
budget per frame**. Exceeding it produces `script runtime error: exceeded maximum number of
|
||||
script variables` or a visible server hitch.
|
||||
|
||||
- **One polling loop per concern, never one per entity.** A `for(;;) { wait 0.05 }` loop on each
|
||||
of 60 actors is 60 wakeups a frame. Keep one manager loop over an array.
|
||||
- **Where an event exists, the polling loop should not.** See §4.
|
||||
- **Never loop without a `wait` inside.** A `for(;;)` with no `wait` hangs the server frame and
|
||||
takes the match down — this is the single most common fatal GSC mistake.
|
||||
- Keep the body of a per-frame loop trivial. Move anything expensive behind a condition or a
|
||||
longer interval, and say in a comment why that interval.
|
||||
- Spawning is expensive and entity counts are capped. Reuse and pool; delete what you spawned.
|
||||
|
||||
---
|
||||
|
||||
## 6. Assets Must Be Precached
|
||||
|
||||
```gsc
|
||||
init()
|
||||
{
|
||||
precachemodel("com_laptop_open");
|
||||
precacheshader("hud_icon_frag");
|
||||
precachestring(&"MYMOD_HINT");
|
||||
}
|
||||
```
|
||||
|
||||
- **Precaching only works during the level's load phase.** A `precachemodel` after the map is
|
||||
live either errors or silently fails, and the model then renders as a missing-asset
|
||||
placeholder or nothing at all.
|
||||
- Put every precache in the init path, unconditionally — not behind the feature flag that decides
|
||||
whether the asset gets used.
|
||||
- **A model or weapon name not present in the loaded level's zone will not spawn**, regardless of
|
||||
precaching. Asset availability is per-map; a name verified on one map proves nothing about
|
||||
another. State which map you verified on.
|
||||
- Localized strings use the `&"REFERENCE"` form and must exist in the string table; a missing one
|
||||
displays the raw reference.
|
||||
|
||||
---
|
||||
|
||||
## 7. Language Facts That Surprise People
|
||||
|
||||
- **No exceptions, no `try`.** An error aborts the current thread and nothing else. There is no
|
||||
recovery and no handler — which is why §3's guards are the error handling.
|
||||
- **No classes, no inheritance, no `new`.** "Objects" are entities with fields, or plain arrays
|
||||
used as structs. `spawnstruct()` is the idiom for a bag of fields with no world presence.
|
||||
- **Function pointers**: `::funcname` (and `[[ptr]]()` to call) exist in most dialects but not
|
||||
all. Verify before relying on them.
|
||||
- **No closures** in most dialects — an inner function cannot capture a local. Pass state
|
||||
explicitly or hang it on a struct.
|
||||
- `foreach (x in arr)` exists in IW4+ and T6 but not in the earliest dialects; `for (i = 0;
|
||||
i < arr.size; i++)` always works.
|
||||
- **Arrays are associative.** `array[]` with integer keys behaves like a list, but deleting an
|
||||
element leaves a hole and `.size` counts keys. Rebuild rather than delete-in-place.
|
||||
- String comparison is `==`; there is no `switch` on anything but constants, and fallthrough
|
||||
rules differ by dialect.
|
||||
- Vectors are `(x, y, z)` literals with `+`, `-`, and scalar `*`. Component access is `v[0]`,
|
||||
`v[1]`, `v[2]`. `anglestoforward` / `vectornormalize` / `distance` are builtins — do not
|
||||
hand-roll vector maths.
|
||||
|
||||
---
|
||||
|
||||
## 8. Style & Structure
|
||||
|
||||
- **Allman braces, 4 spaces** — the stock scripts' convention; match it, because GSC is almost
|
||||
always read next to engine scripts.
|
||||
- `main()` is the level entry point; `init()` is the convention for a module's setup called from
|
||||
it. Which one the loader calls is dialect- and loader-specific — verify, do not assume.
|
||||
- `#include maps\_utility;` (dialect-dependent path syntax) at the top; prefer the stock utility
|
||||
functions over reimplementing them. `maps\_utility` and `common_scripts\utility` already have
|
||||
flag, wait, and array helpers.
|
||||
- **`/* */` comments do not nest.** Commenting out a block that contains one breaks the file.
|
||||
- **A local declared inside a block is not visible after it.** Writing it in one `if` and reading
|
||||
it in a second does not compile — declare it unconditionally at function scope. Assigning
|
||||
`undefined` to an entity field unsets it, which is usually the right "restore nothing".
|
||||
- **No file-scope constants.** `MY_CONST = 0.2;` outside a function is not valid in most
|
||||
dialects; use a local at the point of use or a `level.` field set during init.
|
||||
- File and function size limits from the general rules apply, and matter more here than usual:
|
||||
there is no debugger worth the name, so a 300-line function is unanalysable.
|
||||
- **Delete dead code.** With no linter to flag it, commented-out GSC accumulates forever.
|
||||
|
||||
---
|
||||
|
||||
## 9. Side Effects on Level State
|
||||
|
||||
GSC runs inside someone else's running game. A change to shared state is not scoped to your
|
||||
feature and is not undone when your thread ends.
|
||||
|
||||
- **Mutating a stock spawner, trigger, or flag changes the mission.** Setting
|
||||
`spawner.count = 999` to get an extra AI makes that spawner infinite for the rest of the level.
|
||||
Save the old value and restore it, or clone rather than mutate.
|
||||
- **Anything you spawn, you delete.** Track handles in your own `level.mymod_*` array and provide
|
||||
a cleanup path; an orphaned entity survives until map change and counts against the cap.
|
||||
- **`setdvar` is global and persistent.** A dvar used as a command channel must be cleared
|
||||
immediately on read, and the clear must happen even on the failure path — otherwise the next
|
||||
poll re-fires the same command.
|
||||
- A dvar as an IPC channel is a **poll**, which §5 says to avoid; it is acceptable only because
|
||||
GSC cannot receive a host event, and the interval and the clear-on-read should both be
|
||||
explicit and commented.
|
||||
- Never leave the player in a modified state (god mode, `notarget`, a changed movement speed)
|
||||
when the feature that set it is turned off. Toggles need an off path that actually restores.
|
||||
@@ -139,9 +139,9 @@ Attach an `AbortController` to every long-lived `fetch` and abort it in cleanup.
|
||||
- **Imports**: prefer the configured alias (`@/…`) over relative climbs.
|
||||
- Indentation and quote/semicolon style: **match the file you are editing**. Do not introduce
|
||||
a new formatter or lint config into a project that already has one.
|
||||
- **One file until it earns a split** — around 250 lines for a self-contained plugin/feature,
|
||||
or when a second file is genuinely reusable (`utils.ts`, `types.ts`, `components/`).
|
||||
Hard limits from the general rules still apply: ESLint's `max-lines` and
|
||||
- **One file until it earns a split** — when it nears the 1000-line limit, or when a second
|
||||
file is genuinely reusable (`utils.ts`, `types.ts`, `components/`).
|
||||
The limits from the general rules (1000 lines per file, 200 per function) apply: ESLint's `max-lines` and
|
||||
`max-lines-per-function` cover both natively, so prefer those over a separate test here.
|
||||
|
||||
---
|
||||
|
||||
@@ -0,0 +1,232 @@
|
||||
---
|
||||
name: lua
|
||||
description: >-
|
||||
Use this skill when writing, reviewing, or debugging Lua — including Lua 5.1/5.2/5.3+ and
|
||||
LuaJIT dialect differences, HavokScript (HKS) as used by Call of Duty's LUI, luacheck
|
||||
configuration, locals over globals, pcall error handling, the integer-vs-float and
|
||||
table-length pitfalls, module layout, and embedded/sandboxed-host constraints.
|
||||
---
|
||||
|
||||
# Lua — Rules & Preferences
|
||||
|
||||
Extends [`coding-dev.md`](skills-public/coding-dev.md). Everything there applies; this adds Lua specifics.
|
||||
|
||||
Lua is almost always **embedded**: the host decides the dialect, the available standard
|
||||
library, the module loader, and whether an uncaught error takes the process down. **Establish
|
||||
which host and which dialect you are writing for before writing a line** — the same file is
|
||||
correct in one and a crash in another.
|
||||
|
||||
---
|
||||
|
||||
## 1. Pin the Dialect First
|
||||
|
||||
| Dialect | Notable traits |
|
||||
| :--- | :--- |
|
||||
| **5.1** | `loadstring`, `unpack`, `setfenv`/`getfenv`, `module()`, no integer type, no goto |
|
||||
| **LuaJIT** | 5.1 plus `bit`, `ffi`, `jit`; a few 5.2 extensions |
|
||||
| **5.2** | `load` replaces `loadstring`, `_ENV` replaces `setfenv`, `goto`, `table.unpack` |
|
||||
| **5.3** | distinct integer subtype, `//`, bitwise operators, `math.type` |
|
||||
| **5.4** | `<const>` / `<close>` attributes, generational GC |
|
||||
| **HavokScript (HKS)** | a 5.1 fork; see §8 |
|
||||
|
||||
- **Determine it from the host, not from habit.** `print(_VERSION)` in the target runtime, or
|
||||
read the host's embedding code. A `goto` in a 5.1 host is a parse error, and a `loadstring`
|
||||
in 5.4 is a nil call.
|
||||
- **Write for the host's dialect; do not write portable-across-all-five code** unless the
|
||||
module genuinely ships to several. Compatibility shims
|
||||
(`local unpack = table.unpack or unpack`) are worth it only where that is true.
|
||||
- Lua 5.1 has **no integer type** — every number is a double. `1/2` is `0.5`, and
|
||||
`tostring(3.0)` is `"3"` in 5.1 but `"3.0"` in 5.3. Never round-trip a number through a
|
||||
string and expect the same text across dialects.
|
||||
|
||||
---
|
||||
|
||||
## 2. The Gate
|
||||
|
||||
```bash
|
||||
luacheck . # lint; must be clean
|
||||
stylua --check . # or lua-format; formatting
|
||||
luac -p $(git ls-files '*.lua') # parse check against the TARGET dialect's luac
|
||||
busted # tests, where the code is testable off-host
|
||||
```
|
||||
|
||||
- **`luac -p` is the cheapest real check available** and the one most often skipped. It catches
|
||||
syntax errors, and when run with the host's own Lua version it catches dialect mismatches
|
||||
before the host does. For an embedded host whose scripts cannot be unit tested, this plus
|
||||
`luacheck` *is* the gate — wire it into the build script and never commit a file that fails it.
|
||||
- `luacheck` needs its globals declared or it reports the host's API as undefined. Declare them
|
||||
rather than disabling the check:
|
||||
```lua
|
||||
-- .luacheckrc
|
||||
std = "lua51"
|
||||
read_globals = { "Engine", "LUI", "GenericButtonSettings" }
|
||||
globals = { "MyModuleState" } -- only what this code deliberately writes
|
||||
max_line_length = 120
|
||||
```
|
||||
- `read_globals` for the host's API, `globals` only for names you intentionally define. If the
|
||||
list of writable globals is growing, that is the finding — see §3.
|
||||
|
||||
---
|
||||
|
||||
## 3. Locals, Not Globals
|
||||
|
||||
**Every variable is `local` unless it must be visible to the host.** An unqualified assignment
|
||||
creates a global, silently, with no warning from the language.
|
||||
|
||||
```lua
|
||||
-- WRONG: typo creates a new global; the real one keeps its old value
|
||||
local function update() cofnig.volume = 1 end
|
||||
|
||||
-- RIGHT
|
||||
local config = require("config")
|
||||
local function update() config.volume = 1 end
|
||||
```
|
||||
|
||||
- Locals are also **faster** — they are register slots, while a global is a hash lookup in
|
||||
`_ENV` on every access. Hoist hot library functions into locals at the top of a module:
|
||||
`local floor, insert = math.floor, table.insert`.
|
||||
- **Namespace what must be global.** One table (`MyMod.state`), never a scatter of
|
||||
`myModFoo` / `myModBar`. In a shared host every global is in one namespace with every other
|
||||
script's.
|
||||
- `luacheck` flags accidental globals. Keep it clean and the typo class disappears.
|
||||
- The 200-local-per-function limit is real; hitting it means the function needed splitting long ago.
|
||||
|
||||
---
|
||||
|
||||
## 4. Errors — `pcall` at the Boundary, Not Everywhere
|
||||
|
||||
- **`error()` for real failures; return `nil, message` for expected ones.** A file that may not
|
||||
exist returns `nil, err`; a caller passing the wrong type gets an `error`.
|
||||
- **`pcall` / `xpcall` at the boundary only** — a callback the host invokes, a plugin entry
|
||||
point, a request handler. Wrapping every internal call turns a stack trace into a string and
|
||||
hides where the fault was.
|
||||
- **`xpcall(f, debug.traceback)` when you need the trace.** Plain `pcall` discards it, and the
|
||||
message alone rarely identifies the line in a long callback chain.
|
||||
- **Never swallow it.** `pcall(f)` with the result ignored is an empty catch block:
|
||||
```lua
|
||||
-- WRONG
|
||||
pcall(risky)
|
||||
|
||||
-- RIGHT
|
||||
local ok, err = pcall(risky)
|
||||
if not ok then log.warn("risky failed: " .. tostring(err)) end
|
||||
```
|
||||
- `error("msg", 2)` blames the caller instead of the raising line — the right level for an
|
||||
argument-validation error.
|
||||
- `error()` can be given a **table**, which is how you carry structured error data; the
|
||||
`tostring` of it is then meaningless, so handlers must check `type(err)`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Tables — The Traps
|
||||
|
||||
- **`#t` is only defined for a sequence** (keys `1..n` with no holes). With a hole it may return
|
||||
either boundary, and the choice can change between runs. Track the count yourself when a
|
||||
table can be sparse, and never use `#t` on a map.
|
||||
- **`table.insert` / `table.remove` shift every later element** — O(n). Appending with
|
||||
`t[#t+1] = v` is cheaper than `table.insert(t, v)`; building a new table beats removing from
|
||||
the middle in a loop.
|
||||
- **Never mutate a table while iterating it with `pairs`.** Assigning to an existing key is
|
||||
defined; adding a new one is not. Collect keys first, then act.
|
||||
- **`pairs` order is unspecified** and varies run to run. Anything user-visible or compared
|
||||
between runs must sort its keys first.
|
||||
- `ipairs` stops at the first `nil`. Use it for sequences and `pairs` for maps — the choice
|
||||
documents which the table is.
|
||||
- `nil` cannot be stored in a table; `t[k] = nil` is a delete, and a sequence containing `nil`
|
||||
is not a sequence.
|
||||
- Beware `0` and `""` being **truthy**. Only `nil` and `false` are false, so
|
||||
`if count then` is true at zero — test `if count ~= 0 then`.
|
||||
- `1` and `1.0` are the **same key** in 5.3+ (the float is normalized), but a non-integral
|
||||
float is its own key.
|
||||
|
||||
---
|
||||
|
||||
## 6. Strings
|
||||
|
||||
- **Never build a string in a loop with `..`.** Each concatenation allocates; collect into a
|
||||
table and `table.concat` once.
|
||||
- **Lua patterns are not regex.** `%` is the escape (not `\`), there is no alternation and no
|
||||
`+`/`?` on groups, character classes are `%a %d %s %w %p %c %x` and their uppercase
|
||||
complements. Reach for `lpeg` when the grammar is real.
|
||||
- `string.format("%q", s)` produces a Lua-readable quoted literal — correct for serialization,
|
||||
wrong for display.
|
||||
- `%s` in `string.format` calls `tostring`, so a nil argument renders `"nil"` rather than
|
||||
erroring — the opposite of `..`, which errors on nil. That asymmetry is a frequent crash
|
||||
source in host callbacks: **prefer `string.format` over `..` for any message built from
|
||||
values that could be nil**, or guard with `tostring()`.
|
||||
- Indices are 1-based and inclusive on both ends: `s:sub(1, 3)` is the first three bytes.
|
||||
- Strings are **byte** strings. `#s` and `s:sub` count bytes, not characters — use `utf8` (5.3+)
|
||||
or a library for anything non-ASCII.
|
||||
|
||||
---
|
||||
|
||||
## 7. Modules & Structure
|
||||
|
||||
```lua
|
||||
local M = {}
|
||||
|
||||
local function helper(x) -- private: a local, not M.helper
|
||||
return x * 2
|
||||
end
|
||||
|
||||
function M.public(x)
|
||||
return helper(x)
|
||||
end
|
||||
|
||||
return M
|
||||
```
|
||||
|
||||
- **Return a local table. Never set globals, never use `module()`** (deprecated in 5.1,
|
||||
removed in 5.2).
|
||||
- `require` caches by module name — a module body runs once. Do not rely on re-`require` to
|
||||
reset state; expose an explicit `M.reset()` if a reset is needed.
|
||||
- `:` for methods that take `self`, `.` for everything else. Be consistent — a `.` call on a
|
||||
`:` method passes the first argument as `self` and fails confusingly.
|
||||
- File and function size limits from the general rules apply.
|
||||
|
||||
---
|
||||
|
||||
## 8. Embedded & Sandboxed Hosts
|
||||
|
||||
When the host owns the interpreter (a game engine, nginx, Redis, Neovim, a plugin API), its
|
||||
constraints outrank every preference above.
|
||||
|
||||
- **Verify an API exists before calling it.** The host's surface differs between versions and
|
||||
builds, and a missing function is a `nil` call that takes down whatever invoked you:
|
||||
```lua
|
||||
if type(Engine.SetDvarString) == "function" then
|
||||
Engine.SetDvarString(name, value)
|
||||
end
|
||||
```
|
||||
Prefer `type(x) == "function"` over a bare `if x then` — it also rejects a truthy non-callable.
|
||||
- **An uncaught error in a host callback may crash the process, not just the script.** In a C
|
||||
host the error unwinds through C frames that were not written to expect it. Treat every
|
||||
callback body as a trust boundary and `pcall` it.
|
||||
- **Do not monkey-patch host globals** (`print`, `require`, a host API function) outside a
|
||||
tightly scoped save/restore — and even then, remember the restore does not happen if the body
|
||||
errors, so do it in the `pcall` result path. Reentrancy makes the saved original a wrapper.
|
||||
- **Load order is the host's, not yours.** Do not assume a sibling script has run. Read shared
|
||||
state inside callbacks rather than at load time, and register your own state defensively.
|
||||
- **Check what the standard library actually is.** Many hosts remove `io`, `os`, and `debug`, or
|
||||
replace them with their own (`io.readfile` rather than `io.open`). Feature-test; do not assume.
|
||||
- Logging is whatever the host provides. Use it rather than `print`, and never leave a
|
||||
high-frequency print in a per-frame callback.
|
||||
|
||||
### HavokScript (HKS) — Call of Duty LUI
|
||||
|
||||
Call of Duty's UI (LUI) runs **HavokScript**, a proprietary Lua 5.1 fork:
|
||||
|
||||
- It is **5.1-based**: `loadstring`, `unpack`, `setfenv` — no `goto`, no integer division, no
|
||||
bitwise operators, no `table.unpack`.
|
||||
- There is **no `require` for siblings inside a UI script folder** in h2-mod-style loaders; a
|
||||
module directory's `__init__.lua` is loaded on its own. Keep each module self-contained in one
|
||||
file rather than splitting it and discovering the split does not load.
|
||||
- There is **no off-host test harness**. `luac -p` with a 5.1 `luac`, `luacheck --std lua51`,
|
||||
and reading the game's console output are the entire verification story — so lint and parse
|
||||
checks are not optional here, they are the only gate that exists.
|
||||
- **A dvar read is not type-safe.** `Engine.GetDvarString` on an unregistered dvar returns `nil`,
|
||||
and `nil .. "text"` is an error that propagates out of the LUI event dispatcher as an access
|
||||
violation. Register the dvar first, feature-test the accessor, and build messages with
|
||||
`string.format`.
|
||||
- **Hot-reload with `dofile` leaves the previous registration in place.** Guard re-entry with a
|
||||
module flag, and expect duplicate menu buttons when you do not.
|
||||
@@ -0,0 +1,402 @@
|
||||
---
|
||||
name: php
|
||||
description: >-
|
||||
Use this skill when writing, linting, analysing, testing, or releasing PHP — libraries,
|
||||
Composer packages, generated SDKs, and plain PHP-FPM web applications behind nginx. Covers
|
||||
strict_types, PHPStan at max, PER-CS 2.0 with risky rules, final readonly value objects,
|
||||
backed enums, PSR interfaces over implementations, output escaping, and PHPUnit strictness.
|
||||
---
|
||||
|
||||
# PHP — Rules & Preferences
|
||||
|
||||
Extends [`coding-dev.md`](skills-public/coding-dev.md). Everything there applies; this adds PHP specifics.
|
||||
|
||||
PHP here means one of two things, and they differ: a **library** published through Composer,
|
||||
which depends on PSR interfaces and ships no implementation, and a **plain PHP-FPM application**
|
||||
behind nginx with no framework, where the filesystem is the router. Both carry §1–§8; §9 is the
|
||||
library half and §10 the application half.
|
||||
|
||||
---
|
||||
|
||||
## 1. The Gate
|
||||
|
||||
```bash
|
||||
composer lint # php-cs-fixer fix --dry-run --diff
|
||||
composer stan # phpstan analyse
|
||||
composer test # phpunit
|
||||
composer check # all three, in that order
|
||||
```
|
||||
|
||||
Define those four as `scripts` in `composer.json` so the gate has one name. `composer check`
|
||||
must be clean before anything is called done — zero fixer diffs, zero PHPStan errors, zero
|
||||
test failures, and no warnings, notices or deprecations (see §7).
|
||||
|
||||
- Use `@php tools/x.php` rather than `php tools/x.php` in a composer script: it runs the
|
||||
interpreter Composer itself is running under, which matters on a machine with several.
|
||||
- `composer test` must never touch the network. Point `defaultTestSuite` at the unit suite so
|
||||
a bare run is safe to fire blind.
|
||||
- Name tool configs `.dist` — `phpstan.neon.dist`, `phpunit.xml.dist`, `.php-cs-fixer.dist.php` —
|
||||
so a developer can drop a local override beside them without dirtying the repo. Add the
|
||||
caches (`.php-cs-fixer.cache`, `.phpunit.cache`) to `.gitignore`.
|
||||
- Lockfile by artifact type: an **application commits `composer.lock`**, a **library ignores
|
||||
it**. Getting this backwards shows up in every diff.
|
||||
|
||||
---
|
||||
|
||||
## 2. There Is No PHP On The Host
|
||||
|
||||
On this workstation PHP is not installed on the host at all. `~/.local/bin/php` is a three-line
|
||||
bridge that execs into the development container, and **`composer` has no bridge** — see
|
||||
[`distrobox-buildbox-dev.md`](skills-private/distrobox-buildbox-dev.md) for the container, and
|
||||
[`coding-dev.md` §13](skills-public/coding-dev.md) for why toolchains never land on an immutable
|
||||
host root.
|
||||
|
||||
```bash
|
||||
# php works from the host. composer does not — go through the container, and chain the cd,
|
||||
# because entering one resets the working directory.
|
||||
distrobox enter <container> -- bash -c "cd /abs/path/to/project && composer check"
|
||||
```
|
||||
|
||||
> [!CAUTION]
|
||||
> **Inside the container, prepend `PATH=/usr/bin:/bin`.** The host `~/.local/bin` is on PATH
|
||||
> ahead of `/usr/bin` there, so the host's own `php` bridge shadows the container's real
|
||||
> interpreter and dies with `/usr/bin/distrobox: No such file or directory`. Every
|
||||
> `vendor/bin/*` tool is a PHP script, so `vendor/bin/phpstan` fails this way and the error
|
||||
> names distrobox rather than anything to do with PHP. `php vendor/bin/phpstan` with the fixed
|
||||
> PATH works.
|
||||
|
||||
Edit PHP sources with `Read`/`Edit`/`Write`, not with `sed` or a `php -r` patch script. The
|
||||
nested quoting inside `distrobox enter … bash -c "…"` mangles `$`, backslashes and quotes in
|
||||
exactly the places PHP uses them, and a half-applied patch script reports success.
|
||||
|
||||
---
|
||||
|
||||
## 3. Strictness — `strict_types` In Every File, PHPStan At Max
|
||||
|
||||
```php
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
```
|
||||
|
||||
**Every file, with no exceptions**, enforced mechanically by the fixer's `declare_strict_types`
|
||||
rule rather than by review. Without it a `"7"` from an API quietly becomes `int 7` and a
|
||||
hydrator whose whole purpose is to fail loudly on drift stops failing at all.
|
||||
|
||||
```neon
|
||||
# phpstan.neon.dist
|
||||
parameters:
|
||||
level: max
|
||||
paths:
|
||||
- src
|
||||
- tests
|
||||
- tools
|
||||
- examples
|
||||
treatPhpDocTypesAsCertain: false
|
||||
```
|
||||
|
||||
- **Analyse tests, tools and examples too**, not just `src`. Those are the three places authors
|
||||
assume are exempt, and they are where a convention actually breaks first.
|
||||
- `treatPhpDocTypesAsCertain: false` stops PHPStan trusting a PHPDoc that is narrower than the
|
||||
signature, so genuine runtime `is_string`/`is_array` guards are not reported as always-true.
|
||||
- **No baseline and no `ignoreErrors`.** Level max is reachable with standard tags and real
|
||||
guards; a baseline is a list of things nobody will ever go back to.
|
||||
- A suppression, when genuinely needed, is the narrowest `@phpstan-ignore-next-line` with the
|
||||
reason on it. Reaching for `@phpstan-*` or `@psalm-*` to describe a *type* is a smell — the
|
||||
standard tags in §5 carry everything level max needs.
|
||||
- **Never relax a strictness flag without an adjacent comment saying why.** This is the house
|
||||
habit across every config file in the repo, and it is what makes a relaxation reviewable
|
||||
instead of load-bearing folklore.
|
||||
|
||||
---
|
||||
|
||||
## 4. Formatting — PER-CS 2.0, Risky Rules On
|
||||
|
||||
```php
|
||||
// .php-cs-fixer.dist.php
|
||||
->setRiskyAllowed(true)
|
||||
->setRules([
|
||||
'@PER-CS2.0' => true,
|
||||
'@PHP82Migration' => true,
|
||||
'declare_strict_types' => true,
|
||||
'strict_param' => true,
|
||||
'strict_comparison' => true,
|
||||
'global_namespace_import' => ['import_classes' => false, 'import_functions' => false],
|
||||
'no_unused_imports' => true,
|
||||
'ordered_imports' => ['sort_algorithm' => 'alpha'],
|
||||
'php_unit_test_case_static_method_calls' => ['call_type' => 'self'],
|
||||
])
|
||||
```
|
||||
|
||||
php-cs-fixer and PHPStan, not Pint and not Psalm. The fixer config does more normative work
|
||||
than the prose does: these rules are where five of the conventions below stop being review
|
||||
items.
|
||||
|
||||
- `strict_comparison` + `strict_param` mean every comparison is `===`/`!==` and every
|
||||
`in_array()` carries its third argument. Loose `in_array` matches numeric strings.
|
||||
- `global_namespace_import => false` is why every global function, constant and class is written
|
||||
with a leading backslash: `\sprintf`, `\JSON_THROW_ON_ERROR`, `\DateTimeImmutable`,
|
||||
`\RuntimeException`. Inside a namespace an unqualified call makes PHP look up the namespaced
|
||||
name first and fall back to global — a per-call cost, and a hazard the day something defines
|
||||
`App\json_decode`.
|
||||
- Guard a fixer API that newer releases added, rather than calling it unconditionally and
|
||||
breaking the lint step on the oldest version `composer.json` still allows:
|
||||
`if (\method_exists($config, 'setUnsupportedPhpVersionAllowed')) { … }`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Banned Constructs
|
||||
|
||||
| Never | Use instead |
|
||||
| :--- | :--- |
|
||||
| `==` / `!=` | `===` / `!==` (the fixer enforces it) |
|
||||
| `in_array($x, $a)` | `\in_array($x, $a, true)` |
|
||||
| `stdClass` from `json_decode` | `\json_decode($b, true, 512, \JSON_THROW_ON_ERROR)` into `array<string, mixed>`, then a typed model |
|
||||
| `json_last_error()` checks | `\JSON_THROW_ON_ERROR` on every encode and decode |
|
||||
| `\DateTime` | `\DateTimeImmutable` — a mutable date in a `readonly` class is a hole in the guarantee |
|
||||
| `setX()` on a value object | `withX()` returning a new instance |
|
||||
| `http_build_query()` | a builder that drops nulls, spells booleans `true`/`false`, and repeats a key instead of emitting `k[0]=` |
|
||||
| `extract()`, `eval()`, `$$var` | named variables |
|
||||
| `@` error suppression | check the return value |
|
||||
| `array()` long syntax | `[]` |
|
||||
| A pure `enum` for a wire value | a backed `enum X: string` (§6) |
|
||||
| `echo $x` in HTML | `\htmlspecialchars($x, \ENT_QUOTES, 'UTF-8')` (§10) |
|
||||
| Unqualified `strlen`/`count`/`sprintf` inside a namespace | `\strlen`, `\count`, `\sprintf` |
|
||||
|
||||
- **PHPDoc only says what the signature cannot**: array shapes, generics, `class-string`,
|
||||
`list<T>`, `@template`, `@implements`, `@return never`. A `@param string $x` restating a
|
||||
declared scalar is a second source of truth that will drift from the first.
|
||||
- **`@return never` in a docblock over a native `: mixed`** is the deliberate exception: it
|
||||
gives PHPStan the unreachability while still letting the helper sit in a ternary tail
|
||||
position (`return \is_string($v) ? $v : $this->fail(…);`), which a native `: never` forbids.
|
||||
|
||||
---
|
||||
|
||||
## 6. Types — `final readonly` By Default
|
||||
|
||||
- **`final readonly class` is the default shape** for anything carrying data. Plain
|
||||
`final class` only for things that memoise, hold mutable state, or are pure behaviour.
|
||||
`readonly` on the class is what makes "models are immutable, build a new one" enforced by the
|
||||
engine instead of promised in a docblock.
|
||||
- **Constructor property promotion for every property**, with `private readonly` written out
|
||||
explicitly even inside a `final` class. The one property declared the long way should be the
|
||||
one the constructor has to run through a normaliser — promotion cannot transform a value, so
|
||||
that asymmetry is a readable signal rather than an inconsistency.
|
||||
- **`new` in a parameter default**, not nullable-plus-`??`: `private readonly LoggerInterface
|
||||
$logger = new NullLogger()`. The signature then states the real type and PHPStan never sees a
|
||||
nullable to guard. Keep `?X $x = null` only where a *static factory* forwards the argument and
|
||||
"the caller said nothing" has to stay distinguishable.
|
||||
- **Named arguments** wherever a call has more than two or three arguments or skips one. A
|
||||
wide constructor built positionally silently mis-wires the day two same-typed fields swap
|
||||
order.
|
||||
- **First-class callable syntax** (`Model::fromArray(...)`) over a `'Model::fromArray'` string —
|
||||
it is checked at compile time. Mark a closure that needs no binding `static`, so it cannot
|
||||
capture `$this` into a long-lived callable and keep the owner alive.
|
||||
- Model properties may keep the **API's own `snake_case` names** rather than being renamed to
|
||||
camelCase. A rename table has to be maintained in both directions, silently drops whatever
|
||||
the mapping forgot, and makes a devtools capture stop matching the code.
|
||||
|
||||
---
|
||||
|
||||
## 7. Enums & Wire Values
|
||||
|
||||
- **Backed enums only** (`enum Provider: string`). A backed case is what lets one branch
|
||||
collapse any enum to its wire form and a generic reader be typed
|
||||
`class-string<T of \BackedEnum>`.
|
||||
- **An unrecognised wire value throws**, naming the model, the field, the value, the enum class
|
||||
and the remedy. `::from()` raises a bare `\ValueError` with no field name; an `Unknown`
|
||||
fallback case hands the caller a value the API's own type does not have and defers the failure
|
||||
to someone else's `match`.
|
||||
- Where a field genuinely carries any JSON value, type it **`mixed` and explain it in prose**. A
|
||||
PHPDoc union nothing actually validates is a lie PHPStan will then trust.
|
||||
|
||||
---
|
||||
|
||||
## 8. Errors & Exceptions
|
||||
|
||||
- **The root of the hierarchy is an interface**, `interface AppException extends \Throwable {}`,
|
||||
not a base class. Callers can then `catch (AppException $e)` without catching unrelated
|
||||
`\RuntimeException`s, while each exception still extends the most apt SPL class — which an
|
||||
abstract base makes impossible the moment two of them want different parents.
|
||||
- One **static factory mapping status to class** (`match (true)` over 400/401/403/404/409/422/
|
||||
429/`>= 500`). Declare the base's constructor `final public function __construct` so `new
|
||||
$class(…)` in that factory stays safe.
|
||||
- Carry the **parsed error envelope, the raw body, and the request method and URI** on the
|
||||
exception, and truncate the raw body — a 2 MB proxy error page does not belong in a log line.
|
||||
Document that callers branch on an error *code*, never on the message, which is prose.
|
||||
- **A failure to parse the error body must never mask the real status.** A 502 from a load
|
||||
balancer returns HTML; if the envelope parser throws, the caller sees a JSON parse error
|
||||
instead of "the service is down". Return null from the parser on every one of those paths.
|
||||
- **Retry only idempotent methods** — `GET`, `HEAD`, `OPTIONS`, `PUT`, `DELETE`. POST and PATCH
|
||||
need an explicit opt-in, because a timeout on a request that actually succeeded is how a
|
||||
giveaway gets drawn twice. Honour `Retry-After` over the computed backoff, and inject the
|
||||
sleeper so the retry test does not take 30 seconds of wall clock.
|
||||
|
||||
---
|
||||
|
||||
## 9. Libraries — PSR Interfaces, Not Implementations
|
||||
|
||||
Follow the global ordering (reuse → fork → write, [`coding-dev.md` §3](skills-public/coding-dev.md)).
|
||||
For a published package specifically:
|
||||
|
||||
- **Require PSR interfaces and ship no implementation.** `psr/http-client` (18),
|
||||
`psr/http-factory` (17), `psr/http-message` (7), `psr/log` (3), plus `php-http/discovery`.
|
||||
Guzzle and a PSR-7 implementation go in `require-dev` with a `suggest` entry saying any
|
||||
implementation works. Pinning a client forces a version on every consuming application.
|
||||
- Widen the constraint on an interface package that has had a major (`"psr/http-message":
|
||||
"^1.1 || ^2.0"`) so a consumer's choice does not conflict with yours.
|
||||
- **Wrap the PSR client behind your own one-method transport interface.** That is what lets the
|
||||
whole unit suite run against a fake and assert on exact bytes, instead of building PSR-7
|
||||
mocks in every test.
|
||||
- Convert a discovery miss into your own configuration exception whose message says what to
|
||||
`composer require`.
|
||||
- **Secrets enter through an interface**, never a string parameter: a token provider whose
|
||||
implementation reads the environment, with a null return treated as legitimate because some
|
||||
endpoints are public. A token passed as a constructor literal lands in every stack trace and
|
||||
`var_dump`. See [`coding-dev.md` §10](skills-public/coding-dev.md) for the general rule.
|
||||
- **Partial updates**: distinguish "field absent" from "field is null" per field. Serialise a
|
||||
fetched model, PATCH it back, and a blanket `array_filter(…, fn($v) => $v !== null)` wipes
|
||||
every server-side value the caller never mentioned — while the opposite default breaks the
|
||||
fields for which null *is* the value.
|
||||
- Accept an `int` where the schema says `number`. JSON has one number type, so a whole number
|
||||
arrives as an int and strict `is_float()` rejects every such value. This wants a named
|
||||
regression test.
|
||||
|
||||
**Generated code** lives in directories that are wiped and rewritten whole, never overwritten
|
||||
file by file — otherwise a deleted endpoint lingers as a class that still compiles and 404s.
|
||||
Every generated file opens with a banner carrying the exact regeneration command, generated and
|
||||
hand-written namespaces never share a directory, and **nothing is added to a generated
|
||||
namespace** — a method on a generated enum is deleted by the next run, so it belongs on a
|
||||
hand-written sibling instead.
|
||||
|
||||
---
|
||||
|
||||
## 10. Web Applications — Escaping, Writes, Sessions
|
||||
|
||||
A plain PHP-FPM app with no framework still gets **one `require __DIR__ . '/../vendor/autoload.php'`
|
||||
per entry point** and PSR-4 for everything else. Hand-rolled `require` graphs between
|
||||
application files are what produce double-inclusion and load-order bugs.
|
||||
|
||||
- **Entry points are three lines over a class.** Two copies of the same handler are how one of
|
||||
them loses its CSRF check.
|
||||
- **Compute first, echo second.** With no template engine the discipline is the separation: all
|
||||
logic above the `?>`, then markup with `<?= ?>` echoes of values already shaped. Logic mixed
|
||||
into markup is where an unescaped echo hides.
|
||||
- **Escape at every echo**, with the explicit form `\htmlspecialchars($t, \ENT_QUOTES, 'UTF-8')`.
|
||||
There are no trusted values.
|
||||
|
||||
> [!CAUTION]
|
||||
> **`json_encode` into an inline `<script>` is an XSS hole**, and `JSON_UNESCAPED_SLASHES`
|
||||
> opens it by removing the `<\/script>` escaping that would otherwise have saved you. Any
|
||||
> `</script>` or `<!--` inside the data closes the element and the rest is parsed as HTML. Use
|
||||
> `JSON_HEX_TAG | JSON_HEX_AMP | JSON_HEX_APOS | JSON_HEX_QUOT`, or emit a
|
||||
> `<script type="application/json">` block and `JSON.parse` it.
|
||||
|
||||
- **The document root stays unwritable.** A web server that can create files in its own
|
||||
document root is one bug away from writing a `.php` there. Anything PHP writes goes under
|
||||
`sys_get_temp_dir()`; a file it must update in place gets permission on *the file*, not on
|
||||
the directory. If an upload directory must live under the root anyway, sniff content
|
||||
(`getimagesize()` plus an image-type→extension allowlist, never the uploaded name) **and**
|
||||
deny `.php` under it in the server config.
|
||||
- **Check the return value of every write.** Both halves of the uid trap recorded in this
|
||||
machine's memory were silent: a cache the web user could not create meant every page view
|
||||
re-scraped three third-party APIs, and an `.env` the web user could not read looked exactly
|
||||
like "no token configured".
|
||||
- **Atomic writes, two routes.** Write a temp file beside the target and `rename()` it, copying
|
||||
the victim's mode *and group* first — a rename-created file lands in PHP's group and locks
|
||||
everyone else out of editing it. Fall back to `r+` plus `flock(LOCK_EX)` plus `ftruncate`,
|
||||
never `w`, which truncates before the lock and happily creates a stray file at a typo'd path.
|
||||
Treat a short write as a failure.
|
||||
- **Optimistic concurrency by content hash**, not mtime — the mtime of a file on a network
|
||||
mount is not something to bet a save on. Validate the whole document before writing; a
|
||||
partial save is never attempted.
|
||||
- **Validate a filename against a listing**, not by sanitising it: `in_array($name,
|
||||
array_column($this->backups(), 'name'), true)`. Cleaning the string is how `../../.env` gets
|
||||
through, and for a path under a base directory only comparing `realpath()` results catches a
|
||||
symlink.
|
||||
- **Sessions in one place.** One guarded `session_start()`, an explicit `save_path` you know is
|
||||
writable (a session that silently fails to persist looks exactly like a login that silently
|
||||
fails), `use_strict_mode=1`, `use_only_cookies=1`, and an `httponly` + `samesite=Lax` +
|
||||
`secure` cookie. Decide `secure` by **defaulting to true and excepting loopback**: behind a
|
||||
terminating proxy `$_SERVER['HTTPS']` is empty on a real HTTPS visit, and reading
|
||||
`X-Forwarded-Proto` lets a forged header strip the flag. Expire by inactivity, which the client
|
||||
does not control, and `session_regenerate_id(true)` the moment a visitor gains privilege.
|
||||
- **Re-check authorisation per request**, never from an `is_admin` flag stamped into the session
|
||||
at login — otherwise revoking access means waiting for a session to expire. Request the minimum
|
||||
OAuth scope and discard the access token after use; storing it turns a session leak into an
|
||||
account compromise.
|
||||
- **Discard unsafe input rather than cleaning it**: a redirect target must start with `/`, must
|
||||
not start with `//`, and must have no control characters or backslashes; strip CR/LF from any
|
||||
header value built from a query string.
|
||||
- **An authenticated JSON API answers 401 with JSON**, not a 302 to an HTML login page, which is
|
||||
useless to `fetch()` and surfaces as a parse error. Every admin call is a POST carrying a CSRF
|
||||
token compared with `hash_equals` — **including the reads**, since a GET is reachable from
|
||||
another site's page — dispatched through `match` with a `default` that 400s.
|
||||
- **One response shape**, `{ok: bool, …}` with a snake_case `reason` code, and set the content
|
||||
type, `X-Content-Type-Options: nosniff` and the cache policy *before* anything can fail so the
|
||||
error path carries them too. Choose per endpoint whether upstream trouble is a 200 with
|
||||
`ok: false` (a public endpoint the page polls) or a real status code (a command channel).
|
||||
- **A multipart request has already been consumed into `$_POST`/`$_FILES`**, so check for the
|
||||
upload action *before* reading `php://input` — otherwise the generic JSON path reports "not
|
||||
valid JSON" for a body PHP ate.
|
||||
- **Every outbound call gets a timeout and a connect timeout**, `http_errors => false` so a 4xx
|
||||
body can be read for its reason, and redirects pinned to `https`. No timeout means a hung third
|
||||
party holding an FPM worker.
|
||||
- **A missing credential is a supported state**: separate "configured" from "not configured",
|
||||
answer with a machine-readable reason, and render a page saying it is not set up rather than a
|
||||
500 nobody can diagnose.
|
||||
|
||||
---
|
||||
|
||||
## 11. Testing
|
||||
|
||||
- **PHPUnit with attributes**, never docblock annotations: `#[CoversClass]`, `#[DataProvider]`,
|
||||
`#[Group]`. Annotations are deprecated in PHPUnit 11 and the config below turns that into a
|
||||
failure rather than a warning.
|
||||
- Turn on every escalation in `phpunit.xml.dist`: `failOnWarning`, `failOnNotice`,
|
||||
`failOnDeprecation`, `failOnRisky`, `failOnEmptyTestSuite`, `beStrictAboutOutputDuringTests`.
|
||||
`failOnSkipped="false"` is the one justified exception, because a suite that skips itself
|
||||
without credentials is the intended outcome on a machine without them — say so in an XML
|
||||
comment next to it.
|
||||
- Test classes are `final`; assertions are `self::assertSame(…)` (the fixer enforces it), while
|
||||
expectation setup, which is genuinely instance state, stays on `$this`.
|
||||
- Test method names are full sentences: `testAWrongTypeIsReportedRatherThanCoerced()`.
|
||||
- **Mock at the transport boundary with a fake that fails loudly.** Running out of queued
|
||||
responses throws, naming the method, the URI and how many were already served — a fake
|
||||
returning null turns "one more request than expected" into a type error three frames away.
|
||||
Record every send so a test can assert on exact bytes.
|
||||
- **Fixtures are sanitised real captures**, not hand-written JSON. A hand-written fixture
|
||||
encodes what the author *thinks* the API returns, so a drift test passes against an imagined
|
||||
payload. Keep shape, keys and enum values exactly; replace every identifier, handle, display
|
||||
name, avatar URL and message body with a deterministic placeholder so the diff stays stable.
|
||||
The raw capture stays out of the repository.
|
||||
- **A suite needing credentials is a separate testsuite**, tagged, not the default, and skips
|
||||
per test by re-reading the environment in a helper — so the skip message is attached to the
|
||||
test that needed it. `\getenv()` returns `false` when unset, so guard with `is_string` before
|
||||
`trim`. Everything in it is a read, because the only credentials anyone will run it with
|
||||
belong to something live. **Regression test every fixed bug.**
|
||||
- **Put the file and function size limits in a test**, since neither PHPStan nor the fixer has a
|
||||
length rule — see [`coding-dev.md` §4](skills-public/coding-dev.md). Measure function length
|
||||
by walking `\PhpToken::tokenize()`, not by counting braces in text: a `{` inside a string or a
|
||||
comment looks exactly like a block opener. Skip generated files, whose length is a property of
|
||||
the API surface. Carry over-limit files in an `ACCEPTED` map whose value is the written reason,
|
||||
and fail when a listed file drops back under so the entry gets removed.
|
||||
|
||||
---
|
||||
|
||||
## 12. CI
|
||||
|
||||
A PHP workflow runs `shivammathur/setup-php@v2` with `tools: composer:v2`, then
|
||||
`composer install --prefer-dist --no-interaction --no-progress`, then the gate in order: fixer
|
||||
dry-run, PHPStan, unit suite. Set `permissions: contents: read` explicitly. Put a
|
||||
credential-needing suite behind a `workflow_dispatch` input and pass the secret as `env:`,
|
||||
never as a command-line argument ([`cli-tooling.md` §7](skills-public/cli-tooling.md)).
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Under this account, only `workflow_dispatch` may be live.** Every automatic trigger —
|
||||
> `push`, `pull_request`, `schedule` — stays in the file but commented out, under a comment
|
||||
> saying why, so a fork restores CI by deleting `#` characters. See the GitHub Actions section
|
||||
> of the agent profile; a workflow written with `on: push` looks correct and never runs.
|
||||
|
||||
A job that reports upstream drift should `::warning::` rather than fail: the API moving is
|
||||
information, not a defect in this repository.
|
||||
+164
-19
@@ -2,8 +2,9 @@
|
||||
name: python
|
||||
description: >-
|
||||
Use this skill when writing, linting, type-checking, testing, or packaging Python —
|
||||
including async code, CLI tools, bot cogs, and library packages. Covers strict typing with
|
||||
mypy, ruff configuration, virtualenv discipline, async patterns, and packaging.
|
||||
including async code, CLI tools, MCP servers, Red-DiscordBot cogs, Home Assistant custom
|
||||
components, and library packages. Covers strict typing with mypy, ruff configuration,
|
||||
virtualenv discipline, async patterns, framework conventions, and packaging.
|
||||
---
|
||||
|
||||
# Python — Rules & Preferences
|
||||
@@ -11,6 +12,9 @@ description: >-
|
||||
Extends [`coding-dev.md`](skills-public/coding-dev.md). Everything there applies; this adds Python specifics.
|
||||
For the surrounding build and deploy scripts, see [`bash.md`](skills-public/code-languages/bash.md).
|
||||
|
||||
§1–§9 are the general rules. §10–§12 are the three frameworks most of this Python is written
|
||||
against, each of which overrides a general rule somewhere — and says so.
|
||||
|
||||
---
|
||||
|
||||
## 1. The Gate
|
||||
@@ -23,7 +27,16 @@ pytest
|
||||
```
|
||||
|
||||
All four clean before anything is called done. Configure them in `pyproject.toml` so the
|
||||
editor, the CLI, and CI all read the same settings.
|
||||
editor, the CLI, and the hooks all read the same settings.
|
||||
|
||||
**The gate runs locally and through `pre-commit`, not through CI** — workflows do not run under
|
||||
this account (see the GitHub Actions section of the agent profile), so a green repository with no
|
||||
local run means nothing was checked. A `.pre-commit-config.yaml` with `ruff-pre-commit`
|
||||
(check + format) and `mirrors-mypy` scoped `files: src` is where enforcement actually lives.
|
||||
|
||||
Scale the gate to what the code is: **a new owner-authored project gets all four**. A vendored
|
||||
fork gets lint only — match the surrounding style, keep the patch minimal and reviewable, and
|
||||
do not open a mass reformat of someone else's package (`coding-dev.md` §8 scope discipline).
|
||||
|
||||
---
|
||||
|
||||
@@ -47,6 +60,11 @@ disallow_untyped_defs = true
|
||||
- `@dataclass(frozen=True, slots=True)` for value objects.
|
||||
- Third-party library missing stubs: install `types-*` if it exists, otherwise a narrowly
|
||||
scoped `# type: ignore[import-untyped]` with a comment — never a blanket module-level ignore.
|
||||
For a **framework with no stubs at all**, the per-import ignore does not scale; use a named
|
||||
`[[tool.mypy.overrides]]` block listing those modules with `ignore_missing_imports = true`
|
||||
and a comment saying why, so the exemption is one reviewable list rather than scattered.
|
||||
- Worth adding beyond `strict`: `no_implicit_optional`, `warn_redundant_casts`,
|
||||
`warn_unused_ignores`, `warn_unused_configs`, `show_error_codes`, `allow_redefinition = false`.
|
||||
|
||||
### Value types over primitives
|
||||
|
||||
@@ -70,15 +88,32 @@ choices are:
|
||||
## 3. Linting
|
||||
|
||||
`ruff` is the linter and formatter (it replaces flake8, isort, black, pyupgrade, and most
|
||||
plugins). Enable a broad rule set rather than the default minimum:
|
||||
plugins). The preferred shape is **everything on, minus a documented ignore list**:
|
||||
|
||||
```toml
|
||||
[tool.ruff.lint]
|
||||
select = ["ALL"]
|
||||
ignore = [
|
||||
"COM812", # conflicts with the formatter
|
||||
"D203", "D213", # mutually exclusive with the docstring style chosen below
|
||||
"E501", # the formatter owns line length
|
||||
]
|
||||
```
|
||||
|
||||
The explicit set below is the floor, for a codebase where `ALL` would bury real findings:
|
||||
|
||||
```toml
|
||||
select = ["E", "F", "W", "I", "N", "UP", "B", "A", "C4", "SIM", "RUF", "ASYNC", "S", "PTH"]
|
||||
```
|
||||
|
||||
`S` (bandit security rules) and `ASYNC` are worth the noise. As always: fix, don't suppress —
|
||||
and when you must suppress, use a specific code (`# noqa: S603`) with a reason.
|
||||
- `S` (bandit security rules) and `ASYNC` are worth the noise. `PTH` keeps filesystem access on
|
||||
`pathlib`. Add `PLR0915` and `C901` as the function-size proxies (§8).
|
||||
- **Comment the ignore list too.** Every entry gets the reason beside it, exactly as a `# noqa`
|
||||
does — an uncommented ignore list is a set of rules nobody can tell were disabled on purpose.
|
||||
- As always: fix, don't suppress — and when you must suppress, use a specific code
|
||||
(`# noqa: S603`) with a reason.
|
||||
- Do not add `black` alongside ruff. Where both appear it is an inherited upstream toolchain,
|
||||
not a choice; ruff's formatter is the one to keep in new work.
|
||||
|
||||
---
|
||||
|
||||
@@ -88,8 +123,26 @@ and when you must suppress, use a specific code (`# noqa: S603`) with a reason.
|
||||
Never install into the system interpreter, and never `pip install --user` as a workaround
|
||||
for an immutable OS — use a container or a venv.
|
||||
- `uv` is the preferred installer/resolver where free choice exists; it is dramatically faster
|
||||
and lockfile-native.
|
||||
- Commit the lockfile. Pin the Python version in `pyproject.toml` (`requires-python`).
|
||||
and lockfile-native. **Check it exists before relying on it** (`command -v uv`) — a committed
|
||||
`uv.lock` is not evidence that the tool is installed on this machine.
|
||||
|
||||
> [!CAUTION]
|
||||
> **Bare `python` and bare `pip` can be different interpreters.** On this host `python` is the
|
||||
> system 3.14 while `pip`/`pip3` on PATH are bridge wrappers into the development container's
|
||||
> 3.13 — so `pip install x` followed by `python x.py` produces a `ModuleNotFoundError` that
|
||||
> looks like a bad install. Always go through a project venv, or `python -m pip` with the
|
||||
> interpreter named explicitly. See [`distrobox-buildbox-dev.md`](skills-private/distrobox-buildbox-dev.md).
|
||||
|
||||
- **Lockfile by artifact type**: an application commits its lock, a library does not. A
|
||||
throwaway script directory legitimately stays on a `requirements.txt` with floor pins and no
|
||||
lock at all.
|
||||
- Dev dependencies go in PEP 735 `[dependency-groups] dev = [...]`. `[tool.uv]
|
||||
dev-dependencies` is the older uv-specific form — fine to leave in a fork, not what new work
|
||||
uses.
|
||||
- Pin the Python version in `pyproject.toml` (`requires-python`). Where there is no `[project]`
|
||||
table at all — a plugin or custom component rather than a package — the pin lives in ruff's
|
||||
`target-version` and mypy's `python_version` instead, and **those two will drift apart unless
|
||||
something checks them**; set them in the same edit.
|
||||
- `pathlib.Path` over `os.path` everywhere (ruff's `PTH` rules enforce this).
|
||||
|
||||
---
|
||||
@@ -103,8 +156,13 @@ and when you must suppress, use a specific code (`# noqa: S603`) with a reason.
|
||||
- Custom exception types deriving from a single package-level base, so callers can catch the
|
||||
whole family.
|
||||
- **`logging` module, never `print`,** in anything that isn't a CLI writing to stdout as its
|
||||
actual output. One `logger = logging.getLogger(__name__)` per module, at module scope.
|
||||
actual output. One `logger = logging.getLogger(__name__)` per module, at module scope — unless
|
||||
the framework owns the logger tree, in which case its prefix wins (§11).
|
||||
- **Never log secrets** — tokens, auth headers, message content, personal data.
|
||||
- A blanket `contextlib.suppress(Exception)` is permitted in exactly one situation: a
|
||||
development-ergonomics escape hatch whose every suppression carries a written paragraph saying
|
||||
what staleness or platform quirk it absorbs. The rule is not "never suppress", it is
|
||||
**suppression is allowed when the comment carries the argument**.
|
||||
|
||||
---
|
||||
|
||||
@@ -114,11 +172,19 @@ and when you must suppress, use a specific code (`# noqa: S603`) with a reason.
|
||||
synchronous `requests`, no blocking file I/O in a coroutine. Use `asyncio.to_thread` for
|
||||
genuinely blocking calls.
|
||||
- `httpx` (or the framework's own client) over `requests` in async code.
|
||||
- **`asyncio.TaskGroup`** (3.11+) over bare `create_task` — it propagates exceptions and
|
||||
cancels siblings instead of losing them to an unawaited task.
|
||||
- **`asyncio.TaskGroup`** (3.11+) over bare `create_task` **where one failure should cancel the
|
||||
rest** — it propagates exceptions instead of losing them to an unawaited task.
|
||||
- **For fan-out over N external calls, TaskGroup is the wrong default**: cancel-on-first-error
|
||||
means one dead provider takes down the other nine. Use a bounded helper instead — an
|
||||
`asyncio.Semaphore(limit)` around each worker plus `asyncio.gather(..., return_exceptions=True)`
|
||||
— and keep it in one shared function rather than re-deriving it per caller.
|
||||
- **Every `create_task` result must be held in a reference**, or the garbage collector can
|
||||
cancel it mid-flight.
|
||||
- Cancellation is cooperative: clean up in `finally`, and never swallow `CancelledError`.
|
||||
cancel it mid-flight. Clear it in `finally`.
|
||||
- Cancellation is cooperative: clean up in `finally`, and never swallow `CancelledError` —
|
||||
but **distinguish a cancellation you asked for from a framework shutdown**. Re-raise unless a
|
||||
stop was requested; converting a real shutdown into an application error strands it.
|
||||
- Where a blocking library runs **its own** background thread, don't wrap it in `to_thread` —
|
||||
bridge back onto the loop with `loop.call_soon_threadsafe(future.set_result, …)`.
|
||||
- **Events over polling.** Subscribe to the library's event/callback API; do not loop on a
|
||||
timer. Where polling is genuinely unavoidable (no event source exists), use a **staggered
|
||||
backoff** rather than a fixed interval, and say why polling was necessary.
|
||||
@@ -141,7 +207,13 @@ When talking to a rate-limited API (a chat platform, a store, a tracker):
|
||||
|
||||
## 8. Structure
|
||||
|
||||
- Package layout: `src/<package>/`, tests in `tests/`.
|
||||
- Package layout: `src/<package>/`, tests in `tests/`. A single-binary stdlib-only CLI may stay
|
||||
flat; a plugin or component follows its framework's layout instead.
|
||||
- A tool that has to run on an immutable host is worth keeping **dependency-free** where the
|
||||
stdlib covers it (`argparse`, `subprocess`, `shlex`, `json`, `pathlib`) — nothing to provision
|
||||
is the strongest form of "it works on that machine".
|
||||
- Entry points are `[project.scripts]`. `typer` for a CLI with subcommands, stdlib `argparse`
|
||||
for everything else; do not add a CLI framework to a two-flag script.
|
||||
- **Shared behavior goes in a real shared module**, imported by each consumer — not
|
||||
copy-pasted between sibling packages. When three components grow the same queue/retry/
|
||||
formatting logic, that is the signal to extract it.
|
||||
@@ -153,17 +225,90 @@ When talking to a rate-limited API (a chat platform, a store, a tracker):
|
||||
function-length rule — `PLR0915` (`too-many-statements`, default 50) and `C901`
|
||||
(cyclomatic complexity) are the closest proxies and are worth enabling, but neither
|
||||
measures lines. Put **both** the function-length and file-length limits in a pytest
|
||||
case (see [`coding-dev.md` §4](skills-public/coding-dev.md)).
|
||||
case (see [`coding-dev.md` §4](skills-public/coding-dev.md)): file 1000 lines max, function
|
||||
200 lines max measured by walking the `ast` rather than counting lines of text. **Guard the guard** with a test asserting the glob matched anything at
|
||||
all — a pattern that matches nothing makes every limit vacuous and still passes.
|
||||
|
||||
---
|
||||
|
||||
## 9. Testing
|
||||
|
||||
- `pytest`, with `pytest-asyncio` for coroutines.
|
||||
- `pytest`, with `pytest-asyncio`. Set `asyncio_mode = "auto"` in
|
||||
`[tool.pytest.ini_options]` — it removes the per-test marker entirely — plus
|
||||
`asyncio_default_fixture_loop_scope = "function"`.
|
||||
- Fixtures over setup/teardown methods; `tmp_path` for anything touching the filesystem.
|
||||
- **Regression test every fixed bug.**
|
||||
- Mock at the network boundary (`respx`, `responses`), not by monkeypatching your own
|
||||
internals — a test that patches the code under test proves nothing.
|
||||
- **Regression test every fixed bug**, and give them a home: a `test_regressions.py` is where
|
||||
they stop being scattered and start being a list.
|
||||
- **Mock at a boundary, not inside your own code.** A test that patches the code under test
|
||||
proves nothing. The boundary can be a network mock (`respx`, `responses`), a **real local fake
|
||||
service** the test drives end to end, or **recorded payloads fed to the provider seam** — the
|
||||
last two are usually better evidence than a mocked HTTP layer, because they exercise the
|
||||
parsing too.
|
||||
- **Never exercise destructive operations against real accounts or real data.** Temporary
|
||||
fixtures only.
|
||||
- `pytest -p no:randomly` only when order genuinely matters, and note why.
|
||||
|
||||
---
|
||||
|
||||
## 10. MCP Servers
|
||||
|
||||
> [!CAUTION]
|
||||
> **On the stdio transport, stdout is the protocol.** A stray `print`, a `logging`
|
||||
> `StreamHandler` pointed at stdout, or a library's progress bar corrupts the frame and the
|
||||
> client sees a parse error rather than your message. Log to **stderr** only, and keep §5's
|
||||
> no-`print` rule absolute here.
|
||||
|
||||
- Make the transport an explicit option (`--transport stdio|http`) rather than hardcoding one.
|
||||
- **Tool schemas are derived from the signature and the docstring**, so §2's "every function
|
||||
annotated" rule has a second payoff: an unannotated parameter produces a useless client-facing
|
||||
schema. Never hand-write JSON schema.
|
||||
- Set the behavioural annotations on every tool — `readOnlyHint`, `openWorldHint`, a human
|
||||
`title`.
|
||||
- **Errors are raised, not returned.** Raise (FastMCP serialises it) or return
|
||||
`isError=True` with a `TextContent` on the low-level SDK. A tool that returns the string
|
||||
`"ERROR: …"` as its normal value is indistinguishable from data to the client and defeats
|
||||
its retry logic.
|
||||
- `pydantic-settings` for layered config is good, but note what using a settings value as a
|
||||
**tool parameter default** means: it is frozen at import time *and* published in the advertised
|
||||
schema. Fine for a hostname, a secret leak for anything else.
|
||||
|
||||
---
|
||||
|
||||
## 11. Red-DiscordBot Cogs
|
||||
|
||||
- `self.config = Config.get_conf(self, identifier=<int>, force_registration=True)`, then
|
||||
`register_global`/`register_guild` for every key. **Never a side-car JSON file** beside the cog.
|
||||
- Read-modify-write goes through the context manager — `async with
|
||||
self.config.guild(ctx.guild).channels() as channels:` — because a plain get/set around a
|
||||
mutable list is a lost-update bug.
|
||||
- Lifecycle: `async def setup(bot)` in `__init__.py`; `async def cog_load(self)` for I/O-bearing
|
||||
startup; `@commands.Cog.listener()` for gateway events.
|
||||
- **`cog_unload` is synchronous.** It cannot await, so it cancels held tasks and fires teardown
|
||||
off into `self.bot.loop.create_task(self.api.close())`. Start long-lived tasks the same way,
|
||||
on the bot's loop.
|
||||
- `async def red_delete_data_for_user(self, *, _requester: str, _user_id: int) -> None` is a
|
||||
mandatory contract, not optional boilerplate.
|
||||
- **The logger name overrides §5**: `logging.getLogger("red.<owner>.<cog>")`, not `__name__`.
|
||||
Under `__name__` the cog sits outside the `red.*` tree where Red's own log-level controls
|
||||
apply, so the generic rule produces a logger nobody can turn up.
|
||||
- Red loads each cog directory independently, so a cross-cog shared module is awkward by design.
|
||||
The answer is a per-cog `utils/` package, not a copy-pasted `lib.py` — §8's shared-module rule
|
||||
still applies *within* a cog.
|
||||
|
||||
---
|
||||
|
||||
## 12. Home Assistant Custom Components
|
||||
|
||||
- `from __future__ import annotations` at the top of every module.
|
||||
- The `async_setup_entry` / `async_unload_entry` / `_async_update_listener` triple, with state on
|
||||
**`entry.runtime_data`**, not `hass.data`.
|
||||
- **Never create your own `aiohttp.ClientSession`** — take `async_get_clientsession(hass)`, or
|
||||
you leak a session per reload and HA logs it.
|
||||
- A `DataUpdateCoordinator[T]` subclass whose `_async_update_data` raises `UpdateFailed`. **No
|
||||
provider exception may escape it.**
|
||||
- Declare dependencies in `manifest.json` `requirements`, not in `pyproject.toml` — the component
|
||||
is not a package.
|
||||
- Fire events in size-bounded chunks with a run id in the envelope; a single huge event is not
|
||||
a thing the bus handles gracefully.
|
||||
- PEP 695 syntax is current here: `type MyConfigEntry = ConfigEntry[MyCoordinator]`, and
|
||||
`def f[T](…)` for generics.
|
||||
@@ -73,8 +73,12 @@ workspace = true
|
||||
wrong for the *entire* codebase — not a shortcut for one stubborn call site. It silences
|
||||
the rule in code nobody has written yet.
|
||||
- **Re-test every `#[allow]` periodically.** If removing one produces no warning, delete it.
|
||||
- `too_many_lines` is the mechanical enforcement of the 100-line function limit. A function
|
||||
that trips it wants extracting, not an allow.
|
||||
- `too_many_lines` is the mechanical enforcement of the 200-line function limit. Clippy's
|
||||
default threshold is 100, so set it in `clippy.toml`:
|
||||
```toml
|
||||
too-many-lines-threshold = 200
|
||||
```
|
||||
A function that trips it wants extracting, not an allow.
|
||||
- Clippy has **no file-length lint**, so the 1000-line file limit belongs in a test — see
|
||||
[`coding-dev.md` §4](skills-public/coding-dev.md) for the pattern. Put it in the workspace
|
||||
root's `tests/` so one test covers every member crate, walking `src/` plus each
|
||||
|
||||
+30
-19
@@ -5,7 +5,7 @@ description: >-
|
||||
in any language. Defines standing preferences on strictness, dependency choice, file and function
|
||||
size limits, test discipline, build output handling, error/warning policy, and commit hygiene.
|
||||
Links to per-language rules under code-languages/ for Rust, TypeScript/JavaScript, Bash, Python,
|
||||
Go, C#/.NET, and C/C++.
|
||||
PHP, Go, C#/.NET, and C/C++.
|
||||
---
|
||||
|
||||
# Coding & Development — Standing Rules and Preferences
|
||||
@@ -26,6 +26,14 @@ reasons attached, not style trivia — most exist because ignoring them caused a
|
||||
> When in doubt about which mode you are in, **ask** — it changes the answer on strictness,
|
||||
> file size, typing, and test policy all at once.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **A new software idea starts with research, not code.** When the user describes a tool, app,
|
||||
> service, or other project they want built, follow
|
||||
> [`software-vision.md`](skills-public/software-vision.md) first: find existing projects to use,
|
||||
> extend, or fork (web searches and user reviews first, then training-data recall, then
|
||||
> throwaway clones), and present the options before scaffolding anything — unless the user
|
||||
> explicitly opted out of research.
|
||||
|
||||
Per-language specifics live in [`code-languages/`](skills-public/code-languages/):
|
||||
|
||||
| Language | Rules |
|
||||
@@ -34,9 +42,12 @@ Per-language specifics live in [`code-languages/`](skills-public/code-languages/
|
||||
| TypeScript / JavaScript | [`code-languages/js-typescript.md`](skills-public/code-languages/js-typescript.md) |
|
||||
| Bash / Shell | [`code-languages/bash.md`](skills-public/code-languages/bash.md) |
|
||||
| Python | [`code-languages/python.md`](skills-public/code-languages/python.md) |
|
||||
| PHP | [`code-languages/php.md`](skills-public/code-languages/php.md) |
|
||||
| Go | [`code-languages/go.md`](skills-public/code-languages/go.md) |
|
||||
| C# / .NET | [`code-languages/csharp-dotnet.md`](skills-public/code-languages/csharp-dotnet.md) |
|
||||
| C / C++ | [`code-languages/c-cpp.md`](skills-public/code-languages/c-cpp.md) |
|
||||
| Lua | [`code-languages/lua.md`](skills-public/code-languages/lua.md) |
|
||||
| GSC (Call of Duty script) | [`code-languages/gsc.md`](skills-public/code-languages/gsc.md) |
|
||||
|
||||
---
|
||||
|
||||
@@ -112,7 +123,9 @@ a fresh implementation will rediscover slowly and incompletely.
|
||||
|
||||
Before hand-rolling any subsystem — a protocol client, a parser, a date formatter, a
|
||||
debounce, a retry/backoff, a semver comparator — **search first and state the finding.**
|
||||
Prefer official/first-party SDKs over third-party wrappers.
|
||||
Prefer official/first-party SDKs over third-party wrappers. The same ordering applies to a
|
||||
whole project idea, not just a subsystem — see
|
||||
[`software-vision.md`](skills-public/software-vision.md).
|
||||
|
||||
### The corollary: new generic code becomes its own package
|
||||
|
||||
@@ -135,19 +148,23 @@ depends on those.
|
||||
|
||||
## 4. Size Limits — Files and Functions
|
||||
|
||||
| Unit | Soft target | Split at | Hard limit |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **Source file** | ~400 lines | 600 lines | **1000 lines** |
|
||||
| **Function** | under 50 lines | — | **100 lines** |
|
||||
| **Function parameters** | 3–4 | — | ~5, then pass a struct/object |
|
||||
| **Nesting depth** | 2 | — | 3, then extract |
|
||||
| Unit | Limit |
|
||||
| :--- | :--- |
|
||||
| **Source file** | **1000 lines** max |
|
||||
| **Function** | **200 lines** max |
|
||||
| **Function parameters** | ~5, then pass a struct/object |
|
||||
| **Nesting depth** | 3, then extract |
|
||||
|
||||
These are the limits, not targets to stay well under: there is no earlier "split at" point.
|
||||
A file of 700 lines with one coherent responsibility is fine and is not split for its size.
|
||||
|
||||
### Enforce both mechanically, never by eye
|
||||
|
||||
A limit nobody checks is a suggestion. Pick the **highest** applicable option:
|
||||
|
||||
1. **A linter rule**, where one exists — the function limit is usually covered natively
|
||||
(Clippy's `too_many_lines`, ESLint's `max-lines-per-function`, `golangci-lint`'s `funlen`).
|
||||
(Clippy's `too_many_lines` with `too-many-lines-threshold = 200`, ESLint's
|
||||
`max-lines-per-function`, `golangci-lint`'s `funlen`), configured to 200 lines.
|
||||
A function that trips it wants extracting, not a suppression. Note that some linters only
|
||||
offer *proxies* for length — Ruff's `PLR0915` counts statements, not lines — in which case
|
||||
enable the proxy and put the real limit in a test.
|
||||
@@ -157,24 +174,18 @@ A limit nobody checks is a suggestion. Pick the **highest** applicable option:
|
||||
repository rather than living in one machine's shell script.
|
||||
3. **The build script**, only as a fallback when the project has no test suite to hang it on.
|
||||
|
||||
Whichever you use, it should **fail over the hard limit** and separately *name* anything past
|
||||
the soft split point — so the seam gets found while splitting is still cheap, rather than at
|
||||
the moment the hard limit finally trips.
|
||||
Whichever you use, it should **fail over the limit** and name the offending files.
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn no_source_file_exceeds_limits() {
|
||||
const HARD: usize = 1000;
|
||||
const WARN: usize = 600;
|
||||
const LIMIT: usize = 1000;
|
||||
let mut over = Vec::new();
|
||||
|
||||
for path in source_files() {
|
||||
let lines = std::fs::read_to_string(&path).unwrap().lines().count();
|
||||
if lines > HARD {
|
||||
over.push(format!("{} — {lines} lines (limit {HARD})", path.display()));
|
||||
} else if lines > WARN {
|
||||
// Not a failure: surfaces the seam while splitting is still cheap.
|
||||
eprintln!("note: {} is {lines} lines, approaching the {HARD} limit", path.display());
|
||||
if lines > LIMIT {
|
||||
over.push(format!("{} — {lines} lines (limit {LIMIT})", path.display()));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Executable
+49
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Reinstall each restored app with its original installer (pm set-installer refuses root).
|
||||
|
||||
Same APKs, `install-multiple -r -i <installer>` keeps data. The device comes from ANDROID_SERIAL, as for adb. Usage: fix-installers.py BACKUP_DIR WORK_DIR
|
||||
"""
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
backup, work = Path(sys.argv[1]), Path(sys.argv[2])
|
||||
log_file = (work / "restore.log").open("a")
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
line = f"{time.strftime('%H:%M:%S')} {msg}"
|
||||
print(line, flush=True)
|
||||
log_file.write(line + "\n")
|
||||
log_file.flush()
|
||||
|
||||
|
||||
def adb(*args: str, timeout: int = 900) -> str:
|
||||
r = subprocess.run(["adb", *args], capture_output=True, text=True, timeout=timeout)
|
||||
return (r.stdout + r.stderr).strip()
|
||||
|
||||
|
||||
def installers(text: str) -> dict:
|
||||
# `pm list packages -i` prints "package:PKG", with -f "package:PATH=PKG".
|
||||
return dict(re.findall(r"[:=]([\w.]+)\s+installer=(\S+)", text))
|
||||
|
||||
|
||||
want = installers((backup / "rooted-fresh/packages-3.txt").read_text())
|
||||
now = installers(adb("shell", "pm list packages -3 -i"))
|
||||
disabled = set(re.findall(r"package:([\w.]+)", adb("shell", "pm list packages -d")))
|
||||
log("=== installer fix-up start")
|
||||
for pkg, inst in sorted(want.items()):
|
||||
if inst == "null" or pkg not in now or now[pkg] == inst or inst not in now | {"com.android.vending": 1}:
|
||||
continue
|
||||
apks = sorted(str(p) for p in (backup / "apks" / pkg).glob("*.apk"))
|
||||
if not apks:
|
||||
continue
|
||||
# Reinstalling re-enables a frozen app; disable it again afterwards.
|
||||
out = adb("install-multiple", "-r", "-d", "-i", inst, "--bypass-low-target-sdk-block", *apks)
|
||||
if pkg in disabled:
|
||||
adb("shell", f"su -c 'pm disable-user --user 0 {pkg}'")
|
||||
log(f"{pkg}: installer {now[pkg]} -> {inst}: {'OK' if 'Success' in out else 'FAILED ' + out[-200:]}")
|
||||
log("=== installer fix-up done")
|
||||
Executable
+74
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Emit SQL that copies apps' keystore2 keys from an old ROM's DB into the live one, under the new UIDs.
|
||||
|
||||
Only works when both ROMs use the same TEE (compare keyentry.km_uuid). Keys super-encrypted with the
|
||||
old lock-screen credential (blobmetadata tag 0, EncryptedBy) cannot be moved and are skipped.
|
||||
|
||||
Usage: keystore-transplant.py OLD_DB [--replace] OLD_UID:NEW_UID [...] > transplant.sql
|
||||
--replace first delete the app's keys that already exist on the new ROM (needed when it regenerated them)
|
||||
Apply with the apps force-stopped: `su -c 'sqlite3 /data/misc/keystore/persistent.sqlite < transplant.sql'`, reboot.
|
||||
"""
|
||||
|
||||
import secrets
|
||||
import sqlite3
|
||||
import sys
|
||||
|
||||
DOMAIN_APP = 0
|
||||
TAG_ENCRYPTED_BY = 0
|
||||
|
||||
|
||||
def lit(v: object) -> str:
|
||||
if v is None:
|
||||
return "NULL"
|
||||
if isinstance(v, bytes):
|
||||
return f"X'{v.hex()}'"
|
||||
if isinstance(v, str):
|
||||
return "'" + v.replace("'", "''") + "'"
|
||||
return str(v)
|
||||
|
||||
|
||||
def delete_keys(where: str) -> list[str]:
|
||||
ids = f"(SELECT id FROM keyentry WHERE {where})"
|
||||
return [f"DELETE FROM blobmetadata WHERE blobentryid IN (SELECT id FROM blobentry WHERE keyentryid IN {ids});",
|
||||
f"DELETE FROM blobentry WHERE keyentryid IN {ids};",
|
||||
f"DELETE FROM keyparameter WHERE keyentryid IN {ids};",
|
||||
f"DELETE FROM keymetadata WHERE keyentryid IN {ids};",
|
||||
f"DELETE FROM grant WHERE keyentryid IN {ids};",
|
||||
f"DELETE FROM keyentry WHERE {where};"]
|
||||
|
||||
|
||||
args = sys.argv[1:]
|
||||
old = sqlite3.connect(f"file:{args.pop(0)}?mode=ro", uri=True)
|
||||
replace = "--replace" in args
|
||||
pairs = [tuple(map(int, a.split(":"))) for a in args if a != "--replace"]
|
||||
# Older keystore2 schemas have no blobentry.state; every blob there is current (0).
|
||||
blob_state = "state" if any(c[1] == "state" for c in old.execute("PRAGMA table_info(blobentry)")) else "0"
|
||||
|
||||
out = ["BEGIN;"]
|
||||
for old_ns, new_ns in pairs:
|
||||
if replace:
|
||||
out += delete_keys(f"domain={DOMAIN_APP} AND namespace={new_ns}")
|
||||
keys = old.execute("SELECT id, key_type, alias, state, km_uuid FROM keyentry WHERE domain=? AND namespace=?",
|
||||
(DOMAIN_APP, old_ns)).fetchall()
|
||||
for kid, key_type, alias, state, km_uuid in keys:
|
||||
blobs = old.execute(f"SELECT id, subcomponent_type, blob, {blob_state} FROM blobentry WHERE keyentryid=?", (kid,)).fetchall()
|
||||
if any(old.execute("SELECT 1 FROM blobmetadata WHERE blobentryid=? AND tag=?", (b[0], TAG_ENCRYPTED_BY)).fetchone()
|
||||
for b in blobs):
|
||||
print(f"-- skip {old_ns} {alias!r}: super-encrypted with the old lock-screen credential", file=sys.stderr)
|
||||
continue
|
||||
new_id = secrets.randbits(63) * (1 if secrets.randbits(1) else -1)
|
||||
# The old key wins over one the app generated under the same alias on the new ROM.
|
||||
out += delete_keys(f"domain={DOMAIN_APP} AND namespace={new_ns} AND alias={lit(alias)}")
|
||||
out.append(f"INSERT INTO keyentry (id,key_type,domain,namespace,alias,state,km_uuid) VALUES "
|
||||
f"({new_id},{key_type},{DOMAIN_APP},{new_ns},{lit(alias)},{state},{lit(km_uuid)});")
|
||||
for bid, sub, blob, bstate in blobs:
|
||||
out.append(f"INSERT INTO blobentry (subcomponent_type,keyentryid,blob,state) VALUES ({sub},{new_id},{lit(blob)},{bstate});")
|
||||
for tag, data in old.execute("SELECT tag, data FROM blobmetadata WHERE blobentryid=?", (bid,)):
|
||||
out.append(f"INSERT INTO blobmetadata (blobentryid,tag,data) VALUES ((SELECT max(id) FROM blobentry WHERE keyentryid={new_id}),{tag},{lit(data)});")
|
||||
for tag, data, level in old.execute("SELECT tag, data, security_level FROM keyparameter WHERE keyentryid=?", (kid,)):
|
||||
out.append(f"INSERT INTO keyparameter (keyentryid,tag,data,security_level) VALUES ({new_id},{tag},{lit(data)},{level});")
|
||||
for tag, data in old.execute("SELECT tag, data FROM keymetadata WHERE keyentryid=?", (kid,)):
|
||||
out.append(f"INSERT INTO keymetadata (keyentryid,tag,data) VALUES ({new_id},{tag},{lit(data)});")
|
||||
print(f"-- {old_ns}->{new_ns} {alias!r}: {len(blobs)} blobs", file=sys.stderr)
|
||||
out.append("COMMIT;")
|
||||
print("\n".join(out))
|
||||
Executable
+8
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# Cold-start an app's launcher activity the way Android 17 accepts from the shell. Usage: launch.sh PKG [WAIT] [SHOT.png]
|
||||
p=$1 w=${2:-6} shot=${3:-}
|
||||
act=$(adb shell "cmd package resolve-activity --brief -c android.intent.category.LAUNCHER $p" </dev/null | tr -d '\r' | tail -1)
|
||||
adb shell "am start -S -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n '$act'" </dev/null >/dev/null
|
||||
sleep "$w"
|
||||
[ -n "$shot" ] && adb exec-out screencap -p > "$shot" </dev/null
|
||||
true
|
||||
Executable
+42
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env bash
|
||||
# Snapshot phone health: memory, load, crashes/ANRs since boot, low-memory kills, log errors, cold launches.
|
||||
# Usage: measure.sh OUT_FILE [launch-package ...]
|
||||
set -uo pipefail
|
||||
out=$1; shift
|
||||
a() { adb shell "$@" </dev/null; }
|
||||
|
||||
{
|
||||
echo "# $(date '+%F %T') $(a getprop ro.build.display.id) / Android $(a getprop ro.build.version.release)"
|
||||
echo "## system"
|
||||
a 'uptime; cat /proc/loadavg'
|
||||
a 'grep -E "MemTotal|MemAvailable|Cached:|SwapTotal|SwapFree" /proc/meminfo'
|
||||
a 'su -c "cat /proc/sys/vm/swappiness; cat /sys/block/zram0/mm_stat 2>/dev/null"'
|
||||
echo "procs: $(a 'ls -d /proc/[0-9]* | wc -l')"
|
||||
echo "## top cpu"
|
||||
a 'top -b -n 1 -m 12 -o PID,USER,%CPU,%MEM,NAME' | tail -13
|
||||
echo "## dropbox since boot"
|
||||
boot=$(a 'date -d @$(( $(date +%s) - $(cut -d. -f1 /proc/uptime) )) "+%Y-%m-%d %H:%M:%S"')
|
||||
a "dumpsys dropbox" | awk -v b="$boot" '$1" "$2 >= b && $3 ~ /anr|crash|watchdog|tombstone|wtf|lowmem|native/ {print $3}' | sort | uniq -c
|
||||
echo "## crash buffer (FATAL / ANR per process)"
|
||||
a 'logcat -d -b crash' | grep -oE "Process: [^,]+|FATAL EXCEPTION|>>> [^ ]+ <<<" | sort | uniq -c | sort -rn | head -20
|
||||
a 'logcat -d -b events' | grep -oE "am_anr: \[[^]]*" | awk -F, '{print "ANR " $3}' | sort | uniq -c
|
||||
echo "## lmkd / am_kill"
|
||||
a 'logcat -d -b events' | grep -cE "am_kill|lmk_kill" | sed 's/^/kills: /'
|
||||
a 'logcat -d -b events' | grep -oE "am_kill: \[[^]]*" | awk -F, '{print $5}' | sort | uniq -c | sort -rn | head -10
|
||||
echo "## logcat error tags (top 25)"
|
||||
a 'logcat -d *:E' | awk '{print $6}' | sort | uniq -c | sort -rn | head -25
|
||||
echo "## thermal / battery"
|
||||
a 'dumpsys battery | grep -E "level|temperature|status"; for z in /sys/class/thermal/thermal_zone*; do echo "$(cat $z/type) $(cat $z/temp)"; done 2>/dev/null | sort -k2 -n | tail -5'
|
||||
if [ $# -gt 0 ]; then
|
||||
echo "## cold launches (ms, am start -W -S, 3 runs)"
|
||||
for p in "$@"; do
|
||||
t=""
|
||||
for _ in 1 2 3; do
|
||||
act=$(a "cmd package resolve-activity --brief -c android.intent.category.LAUNCHER $p" | tr -d '\r' | tail -1)
|
||||
v=$(a "am start -W -S -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n '$act'" | awk '/TotalTime/{print $2}')
|
||||
t="$t ${v:-fail}"; a 'input keyevent KEYCODE_HOME'; sleep 2
|
||||
done
|
||||
echo "$p:$t"
|
||||
done
|
||||
fi
|
||||
} 2>&1 | tee "$out"
|
||||
Executable
+88
@@ -0,0 +1,88 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild the Nova home screen in Lawnchair, plus the Ice Box shortcuts Nova had pinned.
|
||||
|
||||
Usage: nova-to-lawnchair.py NOVA_DB LAWNCHAIR_DB OLD_ICEBOX_SHORTCUT_XML OUT_DIR
|
||||
|
||||
Writes OUT_DIR/launcher_5_5_5.db (Lawnchair schema, Nova's items), OUT_DIR/icebox.xml
|
||||
(shortcut_service package file with only the pinned shortcuts) and OUT_DIR/lawnchair-pins.xml
|
||||
(shortcut_service launcher file pinning them for Lawnchair).
|
||||
"""
|
||||
|
||||
import re
|
||||
import shutil
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
LAUNCHER = "app.lawnchair.nightly"
|
||||
ICEBOX = "com.catchingnow.icebox"
|
||||
# From /data/system/packages.xml on the new ROM: versionCode, lastUpdateTime, SHA-256 of the cert.
|
||||
LAUNCHER_INFO = ("1600020100", "1792700106881", "dHw2RbNXJYsuI+hR5TyWdH/grdAH5bos2X6MhVcuTcU=")
|
||||
ICEBOX_INFO = ("154821395", "1792699567277", "gvW+cC4DIBdGDKg3b22r1yWp5S3SSSDdXl17WF+/4bY=")
|
||||
# Nova-only containers (gesture/dock-swipe slots); Lawnchair has nothing to put them in.
|
||||
SKIP_CONTAINERS = {-207, -208}
|
||||
ITEM_APP, ITEM_FOLDER, ITEM_WIDGET, ITEM_DEEP_SHORTCUT = 0, 2, 4, 6
|
||||
FLAG_ID_NOT_VALID = 1 # Launcher3 restore flag: allocate and bind a fresh widget id on load
|
||||
MISSING_PACKAGES = {"com.dot.gallery"} # not installed on the new ROM
|
||||
# Google's smartspace widget on row 0 of the first page: Lawnchair reserves that row for its own.
|
||||
SKIP_IDS = {56}
|
||||
|
||||
nova_db, lc_db, icebox_xml, out = map(Path, sys.argv[1:5])
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def package_info(version: str, updated: str, sig: str) -> str:
|
||||
return (f'<package-info version="{version}" last_udpate_time="{updated}" shadow="0" allow-backup="1" '
|
||||
f'allow-backup-initialized="1" bk_src_version="-1" bk_src_backup-allowed="0">\n'
|
||||
f'<signature hash="{sig}" />\n</package-info>\n')
|
||||
|
||||
|
||||
nova = sqlite3.connect(nova_db)
|
||||
nova.row_factory = sqlite3.Row
|
||||
rows = [r for r in nova.execute("select * from favorites where _id >= 0 order by _id")
|
||||
if r["container"] not in SKIP_CONTAINERS and r["_id"] not in SKIP_IDS]
|
||||
folders = {r["_id"] for r in rows if r["itemType"] == ITEM_FOLDER}
|
||||
kept = []
|
||||
for r in rows:
|
||||
pkg = re.search(r"component=([\w.]+)/", r["intent"] or "")
|
||||
if pkg and pkg.group(1) in MISSING_PACKAGES:
|
||||
print(f"skip {r['title']}: {pkg.group(1)} not installed")
|
||||
continue
|
||||
if r["container"] >= 0 and r["container"] not in folders:
|
||||
continue
|
||||
kept.append(r)
|
||||
|
||||
db = out / "launcher_5_5_5.db"
|
||||
shutil.copy(lc_db, db)
|
||||
lc = sqlite3.connect(db)
|
||||
lc.execute("delete from favorites")
|
||||
pins = []
|
||||
for r in kept:
|
||||
widget = r["itemType"] == ITEM_WIDGET
|
||||
hotseat = r["container"] == -101
|
||||
in_folder = r["container"] >= 0
|
||||
x, y = int(max(r["cellX"] or 0, 0)), int(max(r["cellY"] or 0, 0))
|
||||
lc.execute(
|
||||
"insert into favorites (_id,title,intent,container,screen,cellX,cellY,spanX,spanY,itemType,"
|
||||
"appWidgetId,appWidgetProvider,restored,profileId,rank,options) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(r["_id"], r["title"], r["intent"], r["container"],
|
||||
x if hotseat else (0 if in_folder else r["screen"]),
|
||||
x, 0 if hotseat else y, int(r["spanX"]), int(r["spanY"]), r["itemType"],
|
||||
r["appWidgetId"] if widget else -1, r["appWidgetProvider"],
|
||||
FLAG_ID_NOT_VALID if widget else 0, 0, r["rank"], 0))
|
||||
if r["itemType"] == ITEM_DEEP_SHORTCUT:
|
||||
pins.append(re.search(r"S\.shortcut_id=([^;]+)", r["intent"]).group(1))
|
||||
lc.commit()
|
||||
print(f"{len(kept)} items written, pins: {pins}")
|
||||
|
||||
old = Path(icebox_xml).read_text()
|
||||
blocks = [m.group(0) for m in re.finditer(r'<shortcut id="([^"]+)".*?</shortcut>\n', old, re.S) if m.group(1) in pins]
|
||||
assert len(blocks) == len(pins), f"found {len(blocks)} of {len(pins)} shortcuts"
|
||||
head = re.search(r"<package name=[^>]*>\n", old).group(0)
|
||||
(out / "icebox.xml").write_text("<?xml version='1.0' encoding='utf-8' standalone='yes' ?>\n" + head
|
||||
+ package_info(*ICEBOX_INFO) + "".join(blocks) + "</package>\n")
|
||||
(out / "lawnchair-pins.xml").write_text(
|
||||
"<?xml version='1.0' encoding='utf-8' standalone='yes' ?>\n"
|
||||
f'<launcher-pins package-name="{LAUNCHER}" launcher-user="0">\n' + package_info(*LAUNCHER_INFO)
|
||||
+ f'<package package-name="{ICEBOX}" package-user="0">\n'
|
||||
+ "".join(f'<pin value="{p}" />\n' for p in pins) + "</package>\n</launcher-pins>\n")
|
||||
Executable
+40
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env bash
|
||||
# Replace an app with a differently signed build while keeping its data, permissions,
|
||||
# installer and frozen state. Data is also saved to BACKUP_DIR before the uninstall.
|
||||
# Usage: reinstall-keep-data.sh PACKAGE BACKUP_DIR APK [APK ...]
|
||||
set -euo pipefail
|
||||
pkg=$1 bak=$2; shift 2
|
||||
a() { adb shell "$@" </dev/null; }
|
||||
su_() { a "su -c '$1'"; }
|
||||
|
||||
mkdir -p "$bak"
|
||||
installer=$(a "pm list packages -3 -i $pkg" | tr -d '\r' | grep "package:$pkg " | sed 's/.*installer=//')
|
||||
disabled=$(a "pm list packages -d $pkg" | tr -d '\r' | grep -cx "package:$pkg" || true)
|
||||
perms=$(a "dumpsys package $pkg" | sed -n '/runtime permissions:/,/^ *[a-z]* .*:$/p' | grep -oE '[a-zA-Z0-9_.]+: granted=true' | cut -d: -f1 | sort -u | tr '\n' ' ')
|
||||
echo "$pkg: installer=$installer disabled=$disabled perms=[$perms]"
|
||||
|
||||
dirs="/data/data/$pkg /data/user_de/0/$pkg"
|
||||
[ "$(su_ "[ -d /data/media/0/Android/data/$pkg ] && echo y")" = y ] && dirs="$dirs /data/media/0/Android/data/$pkg"
|
||||
su_ "am force-stop $pkg"
|
||||
# The phone writes the tar to stdout; exec-out keeps it binary-safe.
|
||||
adb exec-out "su -c 'tar -cf - --exclude=$pkg/cache --exclude=$pkg/code_cache $dirs 2>/dev/null'" > "$bak/$pkg.tar"
|
||||
echo "saved $(stat -c %s "$bak/$pkg.tar") bytes to $bak/$pkg.tar"
|
||||
[ "$(stat -c %s "$bak/$pkg.tar")" -gt 10240 ] || { echo "backup suspiciously small, aborting"; exit 1; }
|
||||
|
||||
a "pm uninstall $pkg"
|
||||
iopt=(); [ -n "$installer" ] && [ "$installer" != null ] && iopt=(-i "$installer")
|
||||
adb install-multiple -r "${iopt[@]}" --bypass-low-target-sdk-block "$@"
|
||||
|
||||
uid=$(su_ "stat -c %u /data/data/$pkg")
|
||||
cache_gid=$((uid - 10000 + 20000))
|
||||
adb exec-in "su -c 'tar -xf - -C /'" < "$bak/$pkg.tar"
|
||||
su_ "while pgrep -x tar >/dev/null; do sleep 0.3; done"
|
||||
for d in /data/data/$pkg /data/user_de/0/$pkg; do
|
||||
su_ "chown -R $uid:$uid $d; for c in cache code_cache; do mkdir -p $d/\$c; chown $uid:$cache_gid $d/\$c; chmod 2771 $d/\$c; done; chcon -hR \"\$(stat -c %C $d)\" $d"
|
||||
done
|
||||
case " $dirs " in *Android/data*)
|
||||
su_ "d=/data/media/0/Android/data/$pkg; chown -R $uid:1078 \$d; find \$d -type d -exec chmod 2770 {} +; find \$d -type f -exec chmod 660 {} +; chcon -R u:object_r:media_rw_data_file:s0 \$d" ;;
|
||||
esac
|
||||
for p in $perms; do a "pm grant $pkg $p" 2>/dev/null || true; done
|
||||
[ "$disabled" = 1 ] && su_ "pm disable-user --user 0 $pkg"
|
||||
echo "$pkg: reinstalled as uid $uid, $(a "dumpsys package $pkg" | grep -m1 versionName | xargs)"
|
||||
Executable
+144
@@ -0,0 +1,144 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Restore user apps, app data and permissions onto a freshly flashed, rooted phone.
|
||||
|
||||
Sources (from the backup folder):
|
||||
apks/<pkg>/*.apk APK + splits
|
||||
rooted-fresh/dumpsys-package.txt installer, granted runtime permissions
|
||||
rooted-fresh/packages-3.txt which packages are user apps
|
||||
rooted-fresh/packages-disabled.txt frozen/disabled state
|
||||
rooted/apps-manifest.json special app-ops (overlay, all-files, ...)
|
||||
<work>/data/<pkg>, <work>/user_de/0/<pkg> unpacked data-fresh.tar
|
||||
|
||||
The device comes from ANDROID_SERIAL, as for adb. Usage: restore.py BACKUP_DIR WORK_DIR [--data-only] [pkg ...] (no pkgs = all user apps)
|
||||
--data-only only put the data back (apps already installed and updated): keeps the current APK,
|
||||
installer, permissions and frozen state. Never re-run a full restore over updated apps.
|
||||
Every action and its result is appended to WORK_DIR/restore.log.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
SKIP = {"com.topjohnwu.magisk"} # installed fresh, its data must not be reused
|
||||
# Dropped on restore: regenerated by the app, and the bulk of most backups.
|
||||
VOLATILE = ["cache", "code_cache", "app_webview/Default/Cache", "app_webview/Default/Code Cache"]
|
||||
|
||||
backup, work = Path(sys.argv[1]), Path(sys.argv[2])
|
||||
data_only = "--data-only" in sys.argv[3:]
|
||||
only = set(sys.argv[3:]) - {"--data-only"}
|
||||
log_file = (work / "restore.log").open("a")
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
line = f"{time.strftime('%H:%M:%S')} {msg}"
|
||||
print(line, flush=True)
|
||||
log_file.write(line + "\n")
|
||||
log_file.flush()
|
||||
|
||||
|
||||
def adb(*args: str, timeout: int = 300, stdin=None) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(["adb", *args], capture_output=True, text=stdin is None,
|
||||
timeout=timeout, stdin=stdin)
|
||||
|
||||
|
||||
def su(cmd: str, timeout: int = 300) -> str:
|
||||
r = adb("shell", f"su -c {shlex.quote(cmd)}", timeout=timeout)
|
||||
return (r.stdout + r.stderr).strip()
|
||||
|
||||
|
||||
def parse_dumpsys(text: str) -> dict:
|
||||
"""installer + granted runtime permissions per package."""
|
||||
out = {}
|
||||
for block in re.split(r"\n Package \[", text)[1:]:
|
||||
name = block.split("]", 1)[0]
|
||||
installer = re.search(r"\n installerPackageName=(\S+)", block)
|
||||
runtime = block.split("runtime permissions:", 1)
|
||||
perms = re.findall(r"\n ([\w.]+): granted=true", runtime[1]) if len(runtime) > 1 else []
|
||||
out[name] = {"installer": installer.group(1) if installer else None, "perms": sorted(set(perms))}
|
||||
return out
|
||||
|
||||
|
||||
fresh = backup / "rooted-fresh"
|
||||
meta = parse_dumpsys((fresh / "dumpsys-package.txt").read_text(errors="replace"))
|
||||
manifest = json.loads((backup / "rooted/apps-manifest.json").read_text())["apps"]
|
||||
user_pkgs = sorted(set(re.findall(r"=([\w.]+)\s+installer=", (fresh / "packages-3.txt").read_text())) - SKIP)
|
||||
disabled = set(re.findall(r"package:([\w.]+)", (fresh / "packages-disabled.txt").read_text()))
|
||||
targets = [p for p in user_pkgs if not only or p in only]
|
||||
log(f"=== restore start: {len(targets)} packages")
|
||||
|
||||
# Pass 1: install every APK set, so installers like Obtainium exist before pass 2 names them.
|
||||
for pkg in [] if data_only else targets:
|
||||
apks = sorted(str(p) for p in (backup / "apks" / pkg).glob("*.apk"))
|
||||
if not apks:
|
||||
log(f"{pkg}: NO APK, skipped")
|
||||
continue
|
||||
# --bypass-low-target-sdk-block: Android 14+ refuses targetSdk < 24 (Automagic workarounds).
|
||||
r = subprocess.run(["adb", "install-multiple", "-r", "-d", "-g",
|
||||
"--bypass-low-target-sdk-block", *apks],
|
||||
capture_output=True, text=True, timeout=900)
|
||||
log(f"{pkg}: install {'OK' if 'Success' in r.stdout else 'FAILED ' + (r.stdout + r.stderr).strip()[-200:]}")
|
||||
|
||||
installed = set(re.findall(r"package:([\w.]+)", adb("shell", "pm list packages").stdout))
|
||||
|
||||
# Pass 2: data, ownership, SELinux labels, installer, permissions, disabled state.
|
||||
for pkg in targets:
|
||||
if pkg not in installed:
|
||||
continue
|
||||
info = meta.get(pkg, {})
|
||||
su(f"am force-stop {pkg}")
|
||||
uid = su(f"stat -c %u /data/data/{pkg}")
|
||||
if not uid.isdigit():
|
||||
log(f"{pkg}: no data dir after install ({uid}), data skipped")
|
||||
continue
|
||||
cache_gid = int(uid) - 10000 + 20000 # u0_aNNN_cache
|
||||
|
||||
members = [m for m in (f"data/{pkg}", f"user_de/0/{pkg}") if (work / m).is_dir()]
|
||||
if members:
|
||||
for m in members:
|
||||
su(f"find /data/{m} -mindepth 1 -maxdepth 1 ! -name lib -exec rm -rf {{}} +")
|
||||
excludes = [f"--exclude={m}/{v}" for m in members for v in VOLATILE]
|
||||
tar = subprocess.Popen(["tar", "-cf", "-", "-C", str(work), *excludes, *members],
|
||||
stdout=subprocess.PIPE)
|
||||
r = adb("exec-in", "su -c 'tar -xf - -C /data'", stdin=tar.stdout, timeout=1800)
|
||||
tar.wait()
|
||||
# exec-in returns once our side is sent; the phone's tar may still be writing.
|
||||
su("while pgrep -x tar >/dev/null; do sleep 0.3; done", timeout=600)
|
||||
for m in members:
|
||||
su(f"chown -R {uid}:{uid} /data/{m}; "
|
||||
f"for d in cache code_cache; do mkdir -p /data/{m}/$d; chown {uid}:{cache_gid} /data/{m}/$d; "
|
||||
# restorecon gives only app_data_file:s0; the per-app MLS categories that
|
||||
# installd put on the top directory must be copied onto its contents.
|
||||
f"chmod 2771 /data/{m}/$d; done; chcon -hR \"$(stat -c %C /data/{m})\" /data/{m}",
|
||||
timeout=900)
|
||||
wrong = su(f"find {' '.join('/data/' + m for m in members)} ! -user {uid} | wc -l")
|
||||
log(f"{pkg}: data {'+'.join(members)} rc={r.returncode} uid={uid} wrong_owner={wrong}")
|
||||
else:
|
||||
log(f"{pkg}: no data in backup")
|
||||
if data_only:
|
||||
continue
|
||||
|
||||
inst = info.get("installer")
|
||||
if inst and inst != "null" and inst in installed | {"com.android.vending"}:
|
||||
log(f"{pkg}: installer {inst} -> {su(f'pm set-installer {pkg} {inst}') or 'OK'}")
|
||||
|
||||
# -g granted everything at install; revoke what was not granted before.
|
||||
wanted = set(info.get("perms", []))
|
||||
now = re.findall(r"\n\s+([\w.]+): granted=true", su(f"dumpsys package {pkg} | sed -n '/runtime permissions:/,$p'"))
|
||||
revoked = [p for p in set(now) - wanted]
|
||||
for p in revoked:
|
||||
su(f"pm revoke {pkg} {p}")
|
||||
for p in wanted - set(now):
|
||||
su(f"pm grant {pkg} {p}")
|
||||
ops = manifest.get(pkg, {}).get("specialAppOps", {})
|
||||
for op, mode in ops.items():
|
||||
su(f"appops set {pkg} {op} {mode}")
|
||||
log(f"{pkg}: perms granted={len(wanted)} revoked={len(revoked)} appops={','.join(ops) or '-'}")
|
||||
|
||||
if pkg in disabled:
|
||||
log(f"{pkg}: disable-user -> {su(f'pm disable-user --user 0 {pkg}')}")
|
||||
|
||||
log("=== restore done")
|
||||
Executable
+40
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env bash
|
||||
# Launch every user app with a launcher activity, one by one: unfreeze if frozen, cold start,
|
||||
# wait, screenshot, collect its crash/ANR/keystore log lines, check it is alive and in front,
|
||||
# then stop it and re-freeze. Writes OUT_DIR/results.tsv, OUT_DIR/<pkg>.png, OUT_DIR/<pkg>.log.
|
||||
# Usage: test-all-apps.sh OUT_DIR [pkg ...]
|
||||
set -uo pipefail
|
||||
out=$1; shift
|
||||
mkdir -p "$out"
|
||||
a() { adb shell "$@" </dev/null; }
|
||||
WAIT=${WAIT:-8}
|
||||
|
||||
if [ $# -gt 0 ]; then pkgs=("$@"); else
|
||||
mapfile -t pkgs < <(a 'pm list packages -3' | tr -d '\r' | sed 's/package://' | sort)
|
||||
fi
|
||||
frozen=$(a 'pm list packages -d' | tr -d '\r' | sed 's/package://')
|
||||
printf 'pkg\twas_frozen\tlaunch_ms\talive\tforeground\tcrash\tanr\tkeystore\tnote\n' > "$out/results.tsv"
|
||||
|
||||
for p in "${pkgs[@]}"; do
|
||||
wf=0; grep -qx "$p" <<<"$frozen" && wf=1
|
||||
[ $wf = 1 ] && a "su -c 'pm enable --user 0 $p'" >/dev/null
|
||||
act=$(a "cmd package resolve-activity --brief -c android.intent.category.LAUNCHER $p" | tr -d '\r' | tail -1)
|
||||
if [[ "$act" != */* ]]; then
|
||||
[ $wf = 1 ] && a "su -c 'pm disable-user --user 0 $p'" >/dev/null
|
||||
printf '%s\t%s\t-\t-\t-\t-\t-\t-\tno launcher activity\n' "$p" "$wf" >> "$out/results.tsv"; continue
|
||||
fi
|
||||
a 'logcat -c'
|
||||
ms=$(a "am start -W -S -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n '$act'" | awk '/TotalTime/{print $2}')
|
||||
sleep "$WAIT"
|
||||
adb exec-out screencap -p > "$out/$p.png" </dev/null
|
||||
pid=$(a "pidof $p" | tr -d '\r')
|
||||
fg=$(a 'dumpsys activity activities' | grep -m1 -E 'topResumedActivity|mResumedActivity' | grep -c "$p/")
|
||||
a 'logcat -d' > "$out/$p.log"
|
||||
crash=$(grep -cE "FATAL EXCEPTION|Process: $p|Fatal signal" "$out/$p.log")
|
||||
anr=$(grep -cE "ANR in $p" "$out/$p.log")
|
||||
ks=$(grep -ciE "AEADBadTag|KeyStoreException|VERIFICATION_FAILED|KeyPermanentlyInvalidated|UnrecoverableKey|BadPaddingException|file is not a database|SQLiteDatabaseCorrupt" "$out/$p.log")
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t\n' "$p" "$wf" "${ms:-?}" "${pid:+yes}" "$fg" "$crash" "$anr" "$ks" >> "$out/results.tsv"
|
||||
echo "$p frozen=$wf ms=${ms:-?} alive=${pid:+yes} fg=$fg crash=$crash anr=$anr ks=$ks"
|
||||
a "am force-stop $p; input keyevent KEYCODE_HOME"
|
||||
[ $wf = 1 ] && a "su -c 'pm disable-user --user 0 $p'" >/dev/null
|
||||
done
|
||||
@@ -0,0 +1,201 @@
|
||||
---
|
||||
name: software-vision
|
||||
description: >-
|
||||
Use this skill whenever the user describes an idea for a new piece of software, a tool, an app,
|
||||
a service, a bot, a plugin, a mod, or an automation — "I want something that …", "build me a …",
|
||||
"could we make a tool for …", "I need a service that …" — before choosing a language, scaffolding
|
||||
a repository, or writing any code. Requires prior-art research first: web searches for existing
|
||||
projects, forks, skeletons, libraries and what users say about them (reviews, complaints,
|
||||
comparisons), only then merged with training-data recall as a secondary source, a
|
||||
throwaway shallow clone of promising candidates to check them against the requirements, and a
|
||||
set of options (fork, depend on, extend, start fresh) for the user to choose from.
|
||||
---
|
||||
|
||||
# Software Vision — Find What Exists Before Building
|
||||
|
||||
When the user has an idea for a piece of software, the first job is **not** to design it. It is
|
||||
to find out what already exists that does all or most of it. Building a full stack from nothing
|
||||
when a project covering 80% of the requirements is one `git clone` away wastes days, and the
|
||||
result is usually worse than the thing that already had years of bug reports behind it.
|
||||
|
||||
This is the project-level form of the dependency ordering in
|
||||
[`coding-dev.md` §3](skills-public/coding-dev.md): use what exists, fork what is close, write
|
||||
your own only when nothing fits. That section governs individual subsystems; this one governs
|
||||
the whole idea.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Never assume nothing exists.** However niche the idea sounds, someone may have built it,
|
||||
> half-built it, abandoned it, drafted a skeleton, written a blog post about the protocol, or
|
||||
> published research on the hard part. "I couldn't think of anything" is not a finding — a list
|
||||
> of the searches you ran and what they turned up is.
|
||||
|
||||
---
|
||||
|
||||
## 1. When This Applies — and the Only Opt-Out
|
||||
|
||||
Run this whole process whenever the user describes software they want to exist: a new app,
|
||||
CLI, service, bot, plugin, mod, integration, library, or automation — whether it is a fresh
|
||||
repository or a large new component inside an existing one.
|
||||
|
||||
**The only way to skip it is an explicit opt-out from the user**, such as "no online research",
|
||||
"don't look at other projects", or "I want to write this from scratch myself". Then:
|
||||
|
||||
- Skip the web searches and the clones (§3–§4).
|
||||
- Still use the libraries rule in [`coding-dev.md` §3](skills-public/coding-dev.md) for
|
||||
individual subsystems unless the user excluded that too.
|
||||
- Do not argue for research afterwards; the user has already decided.
|
||||
|
||||
Urgency, a small-sounding scope, or "it's just a quick script" are **not** opt-outs. A few
|
||||
searches take a minute; reinventing a maintained tool takes far longer.
|
||||
|
||||
---
|
||||
|
||||
## 2. Pin Down the Requirements First
|
||||
|
||||
Before searching, turn the user's description into a short checklist, because that list is what
|
||||
every candidate gets measured against:
|
||||
|
||||
- **Must-haves** — what the software has to do to be useful at all.
|
||||
- **Nice-to-haves** — features the user mentioned that are not deal-breakers.
|
||||
- **Constraints** — platform (Linux, Windows, Wine, Android, a game engine), runtime, language
|
||||
preferences, self-hosted vs cloud, license needs, offline use, performance.
|
||||
|
||||
Show the checklist to the user with the results (§6) so they can correct it. If a requirement
|
||||
is genuinely ambiguous *and* would change which candidates qualify, ask — otherwise state the
|
||||
assumption and carry on.
|
||||
|
||||
---
|
||||
|
||||
## 3. Search First, Then Merge in Recall
|
||||
|
||||
The searches happen anyway, so they come **first**, and the decision rests on what they show.
|
||||
Training data can be months or years out of date: projects get abandoned, renamed, relicensed,
|
||||
or overtaken, and newer ones may never have been seen at all. Starting from recall anchors the
|
||||
whole evaluation on a stale picture, so recall comes second and only fills gaps.
|
||||
|
||||
### 3.1 Web searches first
|
||||
|
||||
**Run at least three to five searches**, varied in phrasing and source, before settling on any
|
||||
candidate. Cover:
|
||||
|
||||
| Where | What it finds |
|
||||
| :--- | :--- |
|
||||
| General web search with the problem phrased several ways ("self-hosted X", "open source alternative to Y", "X for Linux") | Products, blog posts, comparisons |
|
||||
| GitHub / GitLab / Codeberg search, topics, and `awesome-*` lists | Projects, forks, skeletons, abandoned prototypes |
|
||||
| The package registry for the likely language (crates.io, npm, PyPI, NuGet, pkg.go.dev, …) | Libraries that do the hard part |
|
||||
| Forums and communities (Reddit, Hacker News, the domain's own Discord/forum, issue trackers of adjacent projects) | "Does anything do X?" threads, half-finished attempts, known dead ends |
|
||||
| Papers, specs, reverse-engineering notes for niche domains | The research behind the hard part, even if nobody shipped an app |
|
||||
|
||||
Search for **parts** as well as the whole: an idea with no complete match often splits into
|
||||
pieces that each have a mature library or tool.
|
||||
|
||||
For each promising hit, note what can be checked without cloning: last commit or release date,
|
||||
license, language, open issue count and how maintainers respond, stars or downloads as a weak
|
||||
signal, and whether its README claims the must-haves.
|
||||
|
||||
### 3.2 What people say about it
|
||||
|
||||
A project's own README describes what it *claims* to do. What users say shows how it actually
|
||||
holds up, so search for opinions on every serious candidate too: reviews, "X vs Y" comparisons,
|
||||
Reddit and Hacker News threads, forum posts, and the complaints in its issue tracker. Look for:
|
||||
|
||||
- Recurring pain points (crashes, data loss, missing features people keep requesting)
|
||||
- Whether people are moving to it or away from it, and to what
|
||||
- How the maintainers respond to bugs and pull requests
|
||||
- Workarounds people rely on, which often point at the gaps a fork would need to fill
|
||||
|
||||
Weight recent opinions over old ones, and many independent voices over a single loud one. Put
|
||||
this in the comparison (§6) next to the technical facts.
|
||||
|
||||
### 3.3 Then merge in recall
|
||||
|
||||
Only after searching, add what you already know from training data: projects, products,
|
||||
libraries, and standards that the searches missed. Every recalled item must be checked with a
|
||||
quick search before it goes into the comparison (does it still exist, is it maintained, has the
|
||||
license changed?). Where recall and current sources disagree, **current sources win.**
|
||||
|
||||
---
|
||||
|
||||
## 4. Clone the Promising Candidates — Then Delete Them
|
||||
|
||||
When a candidate looks close but it is unclear whether it ticks every must-have, **look inside
|
||||
it** rather than guessing from the README.
|
||||
|
||||
1. **Shallow-clone into a throwaway directory** — the session's scratchpad or a fresh `mktemp -d`,
|
||||
never inside a project tree:
|
||||
```bash
|
||||
dir=$(mktemp -d) && git clone --depth 1 <url> "$dir/<name>"
|
||||
```
|
||||
2. **Read, don't run.** A clone is untrusted code. Inspect the README, directory layout, entry
|
||||
points, dependency manifest, build files, tests, and open issues. Do not run its install
|
||||
scripts, build, or test suite as part of evaluation — that is a decision for after the user
|
||||
picks it.
|
||||
3. **Map it against the checklist from §2**: which must-haves it covers, which are partial, which
|
||||
are missing, and roughly how much work each gap is. Note the architecture (does it have a
|
||||
plugin/extension point where the gaps could go?), code quality, and the license.
|
||||
4. **Delete the clone when done.** Nothing from evaluation stays on disk or in a project. If the
|
||||
user later chooses to fork it, that is a fresh, deliberate clone or fork in the right place.
|
||||
|
||||
Clone only the few candidates that are genuinely in contention, not every search hit.
|
||||
|
||||
---
|
||||
|
||||
## 5. Pick Likely Libraries
|
||||
|
||||
Whether the plan ends up as a fork or a fresh start, identify the libraries the work would lean
|
||||
on, for the language that fits best (the existing candidate's language if forking, otherwise the
|
||||
best fit for the constraints and the user's preferences). For each major subsystem — protocol,
|
||||
file format, UI toolkit, storage, networking, parsing — name one or two maintained candidates
|
||||
and why. The per-language rules under
|
||||
[`code-languages/`](skills-public/code-languages/) apply to whatever is chosen.
|
||||
|
||||
---
|
||||
|
||||
## 6. Present Options — the User Chooses
|
||||
|
||||
Report back **before** scaffolding or writing code, with:
|
||||
|
||||
1. **The requirements checklist** (§2), so the user can correct it.
|
||||
2. **What was searched** — the queries and sources, briefly, so an empty result is credible.
|
||||
3. **A comparison of the real candidates**:
|
||||
|
||||
| Candidate | Covers | Missing / partial | License | Health | User opinion | Fit |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
|
||||
4. **Options**, typically some of:
|
||||
- **Use it as is** — it already does the job; maybe configure or script around it.
|
||||
- **Extend it** — via its plugin/extension API, with no fork.
|
||||
- **Fork and modify** — close, but the gaps need changes to its core.
|
||||
- **Contribute upstream** — the gaps are useful to everyone; send them as PRs (then the
|
||||
project's own rules win, per the precedence table in
|
||||
[`coding-dev.md`](skills-public/coding-dev.md)).
|
||||
- **Combine** — several existing tools or libraries glued together.
|
||||
- **Start fresh** — on top of the libraries from §5, with a short, explicit reason why none of
|
||||
the candidates were a usable base.
|
||||
5. **A recommendation**, with the reason — but the decision is the user's. Do not quietly start
|
||||
on the recommended option. When the user has said to work autonomously and decide without
|
||||
them, take the recommendation and record the choice and the alternatives where they will see
|
||||
it.
|
||||
|
||||
If nothing usable exists, say so plainly, list what was checked, and still mention near misses
|
||||
worth borrowing ideas, formats, or code from (license permitting).
|
||||
|
||||
**Keep hosting out of the proposal.** The options are about *what software* to run, not how to
|
||||
deploy it. Containers, proxies, tunnels, extra helper services, and the database layout belong to
|
||||
the environment's own infrastructure skill, which gets read once an option is chosen and before
|
||||
anything is deployed. Note a hard requirement that affects the choice (it needs Postgres, it
|
||||
needs a GPU) and stop there.
|
||||
|
||||
---
|
||||
|
||||
## 7. Checklist
|
||||
|
||||
- [ ] Requirements turned into must-haves / nice-to-haves / constraints
|
||||
- [ ] At least three to five varied web searches first, covering the whole idea and its parts
|
||||
- [ ] User reviews, comparisons, and complaints gathered for each serious candidate
|
||||
- [ ] Training-data recall merged in afterwards, each item checked against current sources
|
||||
- [ ] Promising candidates shallow-cloned to a temp directory, read without running, mapped
|
||||
against the checklist, then deleted
|
||||
- [ ] Libraries identified for the likely language
|
||||
- [ ] Options and a recommendation presented; nothing built before the user chose (or before
|
||||
an autonomous decision was recorded)
|
||||
Reference in new issue
Block a user