Compare commits
4
Commits
6207de1a97
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
  BluscreamandClaude Opus 5.5
|
5e6abadb95 | ||
|
BluscreamandClaude Sonnet 5.5
|
a16a879b1e | ||
|
BluscreamandClaude Opus 5.5
|
44c2d1502c | ||
|
BluscreamandClaude Opus 5
|
e385e01993 |
No files matched your search
@@ -9,8 +9,10 @@ A collection of public, reusable agent skills, cheatsheets, and operational work
|
||||
| Skill File | Scope / Topics Covered |
|
||||
| :--- | :--- |
|
||||
| **[`coding-dev.md`](skills-public/coding-dev.md)** | Standing rules for all code work: maximum linter/compiler strictness, never filtering build output, reuse-before-reinvent dependency ordering, file/function size limits, test & regression discipline, verification gates, scope control, commit hygiene, and honest reporting. |
|
||||
| **[`software-vision.md`](skills-public/software-vision.md)** | Starting from a software idea: turn it into a requirements checklist, search the web first (projects, forks, skeletons, libraries, user reviews), then merge in training-data recall checked against current sources, shallow-clone promising candidates to a temp dir to check them, then present use / extend / fork / start-fresh options before building anything. |
|
||||
| **[`cli-tooling.md`](skills-public/cli-tooling.md)** | Using command-line tools correctly: reading the help tree instead of guessing flags, preferring an existing CLI over an ad-hoc script, non-interactive invocation, machine-readable output formats, orienting on prior work before starting, script placement & symlinking, and care around destructive commands. |
|
||||
| **[`afk-sleep-work.md`](skills-public/afk-sleep-work.md)** | Operating unattended and autonomously: continuous forward momentum, avoiding polling traps and deadlocks, host preservation and resource budgeting (disk, packages, processes), isolated tool provisioning, avoiding "last breath" traps on token/quota exhaustion, bypassing or driving GUI launchers, and wake-up checkpoints. |
|
||||
| **[`android-backup-restore.md`](skills-public/android-backup-restore.md)** | Extracting/backing up/restoring Android data by access tier before a wipe or migration: EDL/firehose & bootloader, custom recovery (TWRP/OrangeFox/CWM), root, device-owner, Shizuku, plain ADB, and on-device apps; FBE decryption limits, per-partition dumps (userdata, EFS/modem/IMEI, persist), SELinux/UID-correct restores, the death of `adb backup`, and app-specific data (WhatsApp, Signal, TOTP/Keystore) that does not survive. |
|
||||
|
||||
### 🔤 Per-Language Rules ([`code-languages/`](skills-public/code-languages/))
|
||||
|
||||
@@ -25,7 +27,7 @@ A collection of public, reusable agent skills, cheatsheets, and operational work
|
||||
| **C# / .NET** | [`csharp-dotnet.md`](skills-public/code-languages/csharp-dotnet.md) | `TreatWarningsAsErrors`, nullable enabled, no `.Result`, trimming/single-file publish caveats, SDK shims |
|
||||
| **C / C++** | [`c-cpp.md`](skills-public/code-languages/c-cpp.md) | full warning set + `-Werror`, sanitizers, RAII/Rule of Zero, target-based CMake, FFI ownership |
|
||||
| **Lua** | [`lua.md`](skills-public/code-languages/lua.md) | pin the dialect first, `luac -p` + luacheck as the only gate, locals over globals, `pcall` at the boundary, `#t`/pattern traps, embedded & HavokScript hosts |
|
||||
| **GSC** | [`gsc.md`](skills-public/code-languages/gsc.md) | no compiler or test harness, `isdefined` on everything external, `endon` before the first wait, per-frame budget, precache at load, level-state side effects |
|
||||
| **GSC** | [`gsc.md`](skills-public/code-languages/gsc.md) | compile off-engine with `gsc-tool --mode comp`, `isdefined` on everything external, `endon` before the first wait, per-frame budget, precache at load, level-state side effects |
|
||||
|
||||
Each language file extends `coding-dev.md` rather than repeating it. A project's own
|
||||
`AGENTS.md` / `CLAUDE.md` / `.rules` always overrides both.
|
||||
|
||||
+1
-1
@@ -120,7 +120,7 @@ When remaining budget or context drops below a safe margin:
|
||||
|
||||
## 7. Checkpointing & The "Wake-Up" Audit Trail
|
||||
|
||||
Maintain a persistent status file (`PROGRESS.md` or session artifact) updated at each milestone:
|
||||
Maintain a persistent status file (for the tsw-plugins project `tsw-plugins/tasks/{todo,in-progress,done}.md`; otherwise a session artifact) updated at each milestone:
|
||||
|
||||
| Section | Contents |
|
||||
| :--- | :--- |
|
||||
|
||||
@@ -0,0 +1,562 @@
|
||||
---
|
||||
name: android-backup-restore
|
||||
description: >-
|
||||
Use this skill when extracting, backing up, migrating, or restoring data from an Android
|
||||
device — especially before a wipe, ROM flash, factory reset, or device-to-device move, and
|
||||
when deciding what is salvageable at a given access level. Covers the full ladder of access
|
||||
tiers (EDL/firehose & bootloader, custom recovery CWM/TWRP/OrangeFox, root, device-owner,
|
||||
Shizuku, plain ADB, and on-device apps), file-based-encryption decryption limits, per-partition
|
||||
dumps (userdata, EFS/modem/IMEI, persist), SELinux/UID-correct restores, the death of
|
||||
`adb backup`, and app-specific data (WhatsApp, Signal, TOTP/Keystore) that does not survive.
|
||||
---
|
||||
|
||||
# Android Backup & Restore — Extraction by Access Level
|
||||
|
||||
How to pull as much as possible off an Android device before wiping, flashing, or migrating —
|
||||
and how to put it back. Organised as a **ladder of access tiers**: the more privileged the
|
||||
access, the more you can extract. Work out which tier you have, pull from the highest tier
|
||||
available, and fall back down the ladder for whatever that tier cannot reach.
|
||||
|
||||
This is a *data-recovery and migration* skill for devices you own or are authorised to service.
|
||||
Shell orchestration here follows [`bash.md`](skills-public/code-languages/bash.md) (strict mode,
|
||||
hard `timeout`s, no wait-loops); CLI invocation follows
|
||||
[`cli-tooling.md`](skills-public/cli-tooling.md) (read `<tool> --help` before trusting any flag
|
||||
below — tool surfaces drift, and the exact partition/block paths are **per-device**).
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **The single biggest variable is encryption.** Every modern Android device uses File-Based
|
||||
> Encryption (FBE); the `/data` partition is a ciphertext blob until something that holds the
|
||||
> user credential derives the keys. That "something" is normally the booted OS. A custom
|
||||
> recovery can only read plaintext `/data` if it was *built to decrypt your exact Android
|
||||
> version* and you type the lockscreen PIN/password into it. See §3. This is why "boot it
|
||||
> normally and pull over ADB/root" beats "boot to recovery and dd" whenever the device still
|
||||
> boots.
|
||||
|
||||
---
|
||||
|
||||
## 1. The Access Ladder (decision table)
|
||||
|
||||
Pick the highest row you can reach. Each tier can extract everything below it plus its own column.
|
||||
|
||||
| Tier | Access you need | Reaches raw partitions? | Reaches decrypted `/data/data` app data? | Reaches EFS/IMEI/persist? | Notes |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| **EDL / firehose / BROM** (§8) | Vendor edl/mtk loader, device in 9008/BROM | ✅ full flash, offline | ⚠️ only as ciphertext (still FBE-locked) | ✅ | Last resort for dead/bricked devices; dumps are encrypted blobs unless you also have the keys. |
|
||||
| **Bootloader / fastboot** (§8) | Unlocked bootloader | ⚠️ some (`fastboot fetch` on newer; mostly flash-only) | ❌ | partial | Mostly a *flashing* interface, not a dumping one. |
|
||||
| **Custom recovery** TWRP/OrangeFox/CWM (§3) | Unlocked bootloader + matching recovery | ✅ every partition via `dd`/built-in backup | ⚠️ **only if recovery decrypts your FBE** + you enter the PIN | ✅ | Classic "nandroid". Decryption is the catch on Android 10+. |
|
||||
| **Root (booted OS)** (§4) | Magisk/KernelSU/APatch on a booting device | ✅ (block devices readable) | ✅ **fully decrypted** (OS is running) | ✅ | The best tier for a *working* device — plaintext data + raw partitions together. |
|
||||
| **Device Owner / DPC** (§5) | `dpm set-device-owner` on a fresh/unprovisioned device | ❌ | ❌ (own managed scope only) | ❌ | Niche; grants/denies permissions, reads some providers; not a bulk dumper. |
|
||||
| **Shizuku / ADB-shell privilege** (§6) | Shizuku (wireless-debug or one-time ADB) | ❌ | ⚠️ APKs, `Android/data`, `Android/obb`, media — **not** `/data/data` of release apps | ❌ | `adb shell` UID (2000) reach, on-device, no PC tether. |
|
||||
| **Plain ADB** (§6) | USB debugging authorised | ❌ | ⚠️ same as Shizuku + `run-as` for *debuggable* apps only | ❌ | `adb backup` is effectively dead (§7). Pull storage, providers, bugreport. |
|
||||
| **On-device app, no root** (§6.4) | Just the unlocked phone | ❌ | ❌ (own data only) | ❌ | Vendor cloud/D2D, Seedvault, per-app export. Last-mile for the non-technical case. |
|
||||
|
||||
**Rule of thumb for a pre-wipe sweep:** if it boots and you can unlock it → get **root** (§4) if
|
||||
possible, else **ADB** (§6). If it does *not* boot → **custom recovery** (§3), else **EDL** (§8).
|
||||
Always grab **EFS/IMEI/persist** (§4.3) before any flash regardless of tier.
|
||||
|
||||
---
|
||||
|
||||
## 2. First: inventory what is actually worth saving
|
||||
|
||||
Before dumping 128 GB of userdata, enumerate. Most of the value is small and specific.
|
||||
|
||||
- **Media & documents** — `/sdcard` (a.k.a. `/storage/emulated/0`): DCIM, Pictures, Download,
|
||||
Documents, Music, ringtones. Reachable at *every* tier including plain ADB. Pull this first; it
|
||||
is the easy win and the bulk of what users actually miss.
|
||||
- **App data** — `/data/data/<pkg>` and `/data/user/0/<pkg>` (credential-encrypted, "CE") and
|
||||
`/data/user_de/0/<pkg>` (device-encrypted, "DE"). Needs root or recovery-decrypt. This is where
|
||||
game saves, app logins, notes, 2FA seeds live.
|
||||
- **Scoped external app data** — `Android/data/<pkg>` and `Android/obb/<pkg>` on `/sdcard`.
|
||||
Blocked from ordinary file managers since Android 11/13 but reachable via ADB/Shizuku/root.
|
||||
- **Telephony/PIM** — SMS/MMS, call log, contacts, calendar. See §6.2 (providers) and §9.
|
||||
- **Identity partitions** — EFS/modem (`modemst1`, `modemst2`, `fsg`), `persist`, `efs`. IMEI,
|
||||
Wi-Fi/BT MAC, sensor calibration. §4.3. **Irreplaceable — back up before any flash.**
|
||||
- **The things that will NOT survive** no matter the tier — hardware-backed Keystore keys, and
|
||||
by extension many 2FA/banking apps. §10. Plan migrations for these *before* wiping.
|
||||
|
||||
---
|
||||
|
||||
## 3. Custom Recovery (TWRP / OrangeFox / CWM) — the offline nandroid
|
||||
|
||||
Use when the device will not boot normally, or you want a block-level image independent of the OS.
|
||||
Requires an **unlocked bootloader** (itself a wipe on most devices — so unlock *before* you have
|
||||
data worth keeping, or accept the loss). Boot without flashing when you can:
|
||||
`fastboot boot twrp.img` leaves the installed recovery untouched.
|
||||
|
||||
> [!TIP]
|
||||
> **Try the *installed* recovery before downloading or temp-booting one.** `adb reboot recovery`,
|
||||
> then `adb shell id` — many custom ROMs ship a current OrangeFox/TWRP whose `adbd` already runs as
|
||||
> **root** and, crucially, is new enough to **decrypt FBE on the installed Android version** (so
|
||||
> `/data/data` shows real package names and readable file contents). This beats `fastboot boot`-ing
|
||||
> an old recovery image: a recovery built from a much older branch often won't boot on a new ROM at
|
||||
> all (its kernel panics and the device falls back to the OS — you'll land back in Android as
|
||||
> `uid=2000(shell)`, the tell-tale that it didn't take). Confirm decryption with
|
||||
> `ls /data/data` (real names, not hex) and by reading one `*/shared_prefs/*.xml`.
|
||||
|
||||
### 3.1 The decryption problem (read this before trusting a backup)
|
||||
|
||||
TWRP/OrangeFox can only back up **plaintext** `/data` if its decryption code matches your device's
|
||||
FBE implementation *and* you enter the lockscreen credential in recovery (Keymaster derives the
|
||||
keys from it). Reality on Android 10+:
|
||||
|
||||
- Use a recovery **built from the branch matching your Android version** (e.g. an `android-12.1`+
|
||||
build for Android 12+). A recovery built from an old branch shows `/data` as garbage folder names
|
||||
or fails to mount.
|
||||
- Devices that **shipped on Android ≤11 and were upgraded** to 12+ often cannot be decrypted by any
|
||||
TWRP build — the keystore/Keymaster path changed. OrangeFox sometimes succeeds where TWRP fails;
|
||||
try both. If neither decrypts, you can still image the partition but only as ciphertext (useless
|
||||
without the keys).
|
||||
- If decryption fails, **do not factory-reset to "fix" it** — that destroys the keys and the data.
|
||||
|
||||
Verify decryption worked before relying on it: in the TWRP terminal, `ls -la /data/data` should
|
||||
show real package names (`com.whatsapp`, …), not hex blobs.
|
||||
|
||||
### 3.2 Built-in backup vs. raw `dd`
|
||||
|
||||
- **Built-in backup** (TWRP "Backup" / OrangeFox): tick System, Data, Boot, EFS, etc. Produces a
|
||||
nandroid set under `/sdcard/TWRP/BACKUPS/`. If `/data` decrypted, the Data archive holds plaintext
|
||||
app data; if not, it backs up the raw (encrypted) partition. Pull the whole folder off afterwards.
|
||||
- **Raw partition image** when you want a specific partition or the built-in backup won't cover it.
|
||||
From the recovery `adb shell` or terminal — **block paths are per-device**, find them first:
|
||||
|
||||
```bash
|
||||
# from recovery shell — enumerate by name, DO NOT guess the numbers
|
||||
ls -l /dev/block/by-name/ # symlinks: userdata, boot_a, modemst1, efs, persist, …
|
||||
cat /proc/partitions
|
||||
|
||||
# image a named partition to portable storage (not back onto /data)
|
||||
timeout 1800 dd if=/dev/block/by-name/userdata of=/sdcard/userdata.img bs=8M
|
||||
```
|
||||
|
||||
Pull with `adb pull /sdcard/userdata.img` (or write to an SD card / OTG drive). A raw `userdata`
|
||||
image is only restorable/mountable if you also have the FBE keys — prefer the decrypted built-in
|
||||
Data backup for app data, and reserve raw images for small partitions (§4.3).
|
||||
|
||||
### 3.3 Restore
|
||||
|
||||
Restore the built-in nandroid through the recovery UI, or `dd` a raw image back **only onto the
|
||||
exact same partition of the same device** (never another device's EFS — §4.3). After a decrypted
|
||||
Data restore across Android versions, expect some apps to misbehave; a clean reinstall + selective
|
||||
data restore (§4.2) is often more reliable than a whole-partition nandroid across versions.
|
||||
|
||||
---
|
||||
|
||||
## 4. Root on a booting device — the best tier for a live phone
|
||||
|
||||
If the device boots and you can unlock it, root (Magisk / KernelSU / APatch) is the strongest
|
||||
position: the OS has already derived the FBE keys, so **all app data is plaintext**, and you still
|
||||
have raw block-device access. (For installing root on this workstation's own test devices, Magisk
|
||||
APKs/modules may already be on the NAS — see
|
||||
[check-nas-before-downloading](check-nas-before-downloading.md) before fetching.)
|
||||
|
||||
### 4.1 Bulk app-data pull
|
||||
|
||||
```bash
|
||||
# as root, on device or via `adb shell su -c`
|
||||
# tar preserves owners/modes/symlinks — plain `cp`/`adb pull` loses them (see §4.4)
|
||||
su -c 'tar -cpf /sdcard/appdata-whatsapp.tar \
|
||||
/data/data/com.whatsapp /data/user_de/0/com.whatsapp'
|
||||
# then from the PC:
|
||||
adb pull /sdcard/appdata-whatsapp.tar
|
||||
```
|
||||
|
||||
Pull `/data/data/<pkg>` (CE) **and** `/data/user_de/0/<pkg>` (DE) for each app — some apps split
|
||||
state across both. For a whole-device grab, `tar` `/data/data`, `/data/user/0`, `/data/user_de/0`
|
||||
and `/data/misc` (wifi configs, etc.), but mind disk budget on the pulling host.
|
||||
|
||||
> [!TIP]
|
||||
> **Stream straight off the device with `adb exec-out` — no on-device staging.** When `/data` has
|
||||
> little free space (you can't write a multi-GB tar back onto the partition you're reading), pipe
|
||||
> the archive directly to the host, and `dd` images the same way (`exec-out` is binary-safe; plain
|
||||
> `adb shell` mangles bytes):
|
||||
> ```bash
|
||||
> adb exec-out 'cd /data && tar -cp -f - data user_de system misc vendor adb' > data-full.tar
|
||||
> adb exec-out 'dd if=/dev/block/by-name/boot status=none' > boot.img
|
||||
> ```
|
||||
> Toybox/busybox `tar` can't store SELinux contexts and exits non-zero when it skips sockets
|
||||
> (`unknown file type '140000'` — harmless). Capture contexts separately as a restore sidecar:
|
||||
> `for d in /data/data/* /data/user_de/0/*; do stat -c '%a %U %G %C %n' "$d"; done`. The per-app
|
||||
> SELinux *categories* (`:c123,c456`) encode the app sandbox and are **not** regenerable by
|
||||
> `restorecon` alone — this sidecar is how you put them back.
|
||||
>
|
||||
> **Record restore metadata while you have the device.** A `dumpsys package` capture gives, per
|
||||
> app: `codePath` (→ system/vendor/product/`/data/app`=user), `pkgFlags` (SYSTEM/UPDATED_SYSTEM_APP),
|
||||
> `installerPackageName` (Play `com.android.vending` vs sideloaded `null`), versionCode, and granted
|
||||
> runtime permissions; `cmd appops dump` adds special ops (SYSTEM_ALERT_WINDOW,
|
||||
> MANAGE_EXTERNAL_STORAGE, …). Parse these into a manifest so you can later restore *only* user apps
|
||||
> and re-grant their exact permissions.
|
||||
|
||||
Prefer dedicated tools over hand-rolled tar when restoring to a *different* device or Android
|
||||
version, because they record ownership/SELinux/permissions that a naive tar restore drops (§4.4):
|
||||
|
||||
- **Swift Backup** (root; Shizuku for a weaker subset) — app + data, permissions, SSAID.
|
||||
- **Neo Backup** (FOSS, OAndBackupX fork; root incl. KernelSU/APatch) — app + data, schedules.
|
||||
- **OAndBackupX / Migrate** — FOSS nandroid-style app+data sets.
|
||||
|
||||
### 4.2 Raw partitions while rooted
|
||||
|
||||
Same `dd` from `/dev/block/by-name/` as §3.2, but from the running OS (`su -c dd …`). Good for
|
||||
grabbing `boot`/`vendor_boot` to patch, or EFS/persist (§4.3) without rebooting to recovery.
|
||||
|
||||
### 4.3 EFS / modem / IMEI / persist — back these up before ANY flash
|
||||
|
||||
These hold IMEI, baseband calibration, Wi-Fi/BT MAC, and sensor calibration. Corrupting them bricks
|
||||
the radio permanently and there is no download that restores *your* unique values. **Partition names
|
||||
and block numbers are per-device — look them up, never copy numbers from a guide.**
|
||||
|
||||
```bash
|
||||
su -c 'ls -l /dev/block/by-name/' | grep -Ei 'efs|modemst|fsg|fsc|persist|nvram|nvdata|protect'
|
||||
# Qualcomm: commonly modemst1, modemst2, fsg, fsc, persist
|
||||
# MediaTek: commonly nvram, nvdata, protect_f, protect_s, persist
|
||||
for p in modemst1 modemst2 fsg persist; do
|
||||
su -c "dd if=/dev/block/by-name/$p of=/sdcard/efs-$p.img bs=4M"
|
||||
done
|
||||
```
|
||||
|
||||
Pull the images **off the device and keep copies on a host/NAS** — a backup that lives only on the
|
||||
partition you are about to wipe is no backup. Restore the same way (`dd … of=/dev/block/by-name/$p`)
|
||||
**only to the identical partition on the identical device**. Never flash one phone's EFS onto
|
||||
another — you would overwrite its unique identity with useless values.
|
||||
|
||||
### 4.4 Restoring raw `/data/data` tars correctly (UID + SELinux)
|
||||
|
||||
A tar of `/data/data/<pkg>` restored by hand will *not* launch unless you fix two things Android
|
||||
assigns at install time:
|
||||
|
||||
1. **Install the app first** (same package, ideally same version) so Android creates the data dir
|
||||
with the correct per-app UID and SELinux context.
|
||||
2. Restore the files, then **fix ownership and labels**:
|
||||
|
||||
```bash
|
||||
# find the app's real UID from the freshly-created dir, don't assume 100xx
|
||||
su -c 'ls -ld /data/data/com.example' # e.g. u0_a142 u0_a142
|
||||
su -c 'tar -xpf /sdcard/appdata-example.tar -C /'
|
||||
su -c 'chown -R u0_a142:u0_a142 /data/data/com.example'
|
||||
su -c 'restorecon -RFD /data/data/com.example' # argument order matters; -v to see work
|
||||
```
|
||||
|
||||
Then reboot before launching. Skipping the UID chown or `restorecon` is the usual reason a
|
||||
"successful" restore produces an app that crashes on open. Dedicated backup apps (§4.1) do this
|
||||
bookkeeping for you — hand-tar only when you must.
|
||||
|
||||
### 4.5 Getting root headlessly (no GUI taps) when the device boots
|
||||
|
||||
If the bootloader is unlocked and you need root to reach §4 but can't tap through the Magisk app,
|
||||
patch and flash boot entirely from the shell:
|
||||
|
||||
1. **Get the *exact* stock boot.img.** The authoritative copy is the live partition — dump it from
|
||||
a root recovery (§3) with `dd`/`exec-out`. Don't flash a boot.img of unknown origin: check its
|
||||
cmdline first (`androidboot.hardware=qcom` = Qualcomm vs MediaTek's `bootopt=…` — a wrong-SoC
|
||||
image bricks the device).
|
||||
2. **Patch it** with Magisk's own tooling. The Magisk APK is a zip: its `assets/boot_patch.sh` needs
|
||||
the arm64 libs renamed (`lib/arm64-v8a/libmagiskboot.so`→`magiskboot`, `libmagiskinit.so`→
|
||||
`magiskinit`, `libmagisk.so`→`magisk`, `libinit-ld.so`→`init-ld`) plus `stub.apk` and
|
||||
`util_functions.sh` in the same dir, then `sh boot_patch.sh boot.img` → `new-boot.img`.
|
||||
- Set the flags **explicitly** (standalone `boot_patch.sh` treats unset as false, which can
|
||||
bootloop): for a modern FBE device with a vbmeta partition and dynamic partitions use
|
||||
`KEEPVERITY=true KEEPFORCEENCRYPT=true PATCHVBMETAFLAG=false RECOVERYMODE=false LEGACYSAR=false`.
|
||||
`Failed to patch` on the kernel DTB step is benign (nothing to patch).
|
||||
- The recovery installer (`META-INF/com/google/android/update-binary`, which auto-detects these
|
||||
flags via `get_flags()`) is cleaner **but** fails outside a real sideload: it tries to mount
|
||||
`/system`, which on super/dynamic-partition ROMs is `/dev/mapper/system` and won't mount in a
|
||||
plain `adb shell`. `boot_patch.sh` with explicit flags sidesteps that.
|
||||
3. **Flash** `new-boot.img` (verify it unpacks: `magiskboot unpack new-boot.img`) with
|
||||
`fastboot flash boot` or, from a root recovery, `dd … of=/dev/block/by-name/boot`. Keep the stock
|
||||
image for instant rollback. Reboot: `magiskd` in `ps -A` + `magisk -V` confirms root took and
|
||||
survives reboot.
|
||||
4. **Grant su to the ADB shell** (Magisk denies it by default, silently): Magisk app → Superuser
|
||||
access = "Apps and ADB" + Automatic response = "Grant", or seed a `policies` row
|
||||
(`uid=2000 policy=2`) in `/data/adb/magisk.db`. Until then `adb shell su -c …` returns
|
||||
`Permission denied` even though root is installed and working.
|
||||
|
||||
---
|
||||
|
||||
## 5. Device Owner (DPC) — narrow, not a bulk extractor
|
||||
|
||||
A device-owner app (set with `adb shell dpm set-device-owner <pkg>/<receiver>` on a device with
|
||||
**no accounts added yet**) gets elevated *policy* powers, not filesystem powers. It is relevant to
|
||||
extraction only in that it can silently grant its own runtime permissions and read some content
|
||||
providers that would otherwise prompt — useful for a managed enrolment flow, not for cleaning out a
|
||||
phone before a wipe. Hard limits:
|
||||
|
||||
- `setPermissionGrantState()` for **sensor** permissions (location, camera, mic, body sensors,
|
||||
activity recognition) is restricted/deprecated for *profile* owners on Android 12+ (silently
|
||||
returns false); device owners retain more but the surface is shrinking each release.
|
||||
- It cannot read other apps' `/data/data`, cannot image partitions, and cannot reach EFS.
|
||||
|
||||
If you find yourself reaching for device-owner to extract data, the real answer is almost always
|
||||
root (§4) or ADB providers (§6.2) instead.
|
||||
|
||||
---
|
||||
|
||||
## 6. ADB & Shizuku — no root, device boots
|
||||
|
||||
Plain ADB runs as the `shell` user (UID 2000). Shizuku grants that **same** privilege to on-device
|
||||
apps (started via wireless debugging or a one-time ADB command), so their reach is identical to ADB
|
||||
— handy when you have no PC.
|
||||
|
||||
### 6.1 External storage & scoped app dirs
|
||||
|
||||
```bash
|
||||
adb pull /sdcard/ ./sdcard-backup/ # DCIM, Download, Documents, etc. — works everywhere
|
||||
# Android/data and Android/obb are blocked from on-device file managers (11/13+) but
|
||||
# the adb shell UID can still read them:
|
||||
adb pull /sdcard/Android/data/ ./android-data/
|
||||
adb pull /sdcard/Android/obb/ ./android-obb/
|
||||
```
|
||||
|
||||
### 6.2 Content providers (telephony/PIM without root)
|
||||
|
||||
`adb shell content query` reaches providers the `shell` UID is allowed to read. What actually works
|
||||
on a stock device is uneven:
|
||||
|
||||
```bash
|
||||
adb shell content query --uri content://contacts/phones # contacts — usually readable
|
||||
adb shell content query --uri content://com.android.calendar/events
|
||||
# SMS/MMS and call_log generally FAIL for the shell UID (need READ_SMS / be default app):
|
||||
adb shell content query --uri content://sms # typically permission denied
|
||||
```
|
||||
|
||||
For SMS/MMS, call logs, and contacts as restorable files, use an on-device FOSS app rather than
|
||||
fighting providers over ADB:
|
||||
|
||||
- **SMS Import / Export** (`com.github.tmo1.sms_ie`, F-Droid) — SMS+MMS+call log+contacts to
|
||||
JSON/ZIP, no root, and it *restores* too (set it as default SMS app during restore).
|
||||
- **Open Android Backup** (mrrfv) — PC-driven, rootless, uses a companion app (not `adb backup`):
|
||||
restores apps-without-data, internal storage, and contacts (vCard); exports SMS (CSV) and call
|
||||
log (txt) as **view-only** (no restore, no MMS attachments).
|
||||
- Contacts also export cleanly from the stock Contacts app ("Export to .vcf") or via Google account
|
||||
sync — often the least-effort path for the non-technical case.
|
||||
|
||||
### 6.3 `run-as` — app data for *debuggable* apps only
|
||||
|
||||
```bash
|
||||
adb shell run-as com.example.debug tar -cpf - ./ > example-data.tar
|
||||
```
|
||||
|
||||
`run-as` only works for apps with `android:debuggable="true"` (i.e. your own dev builds, not Play
|
||||
Store releases). Shizuku-based file managers use the same `run-as` path and inherit the same limit.
|
||||
Release-app `/data/data` is **not** reachable at this tier — that needs root (§4) or a
|
||||
recovery-decrypt (§3).
|
||||
|
||||
### 6.4 Bugreport & app lists (cheap, always worth grabbing)
|
||||
|
||||
```bash
|
||||
adb bugreport ./bugreport.zip # logs, settings dump, installed-package metadata
|
||||
adb shell pm list packages -3 -f # third-party packages + APK paths, to reinstall later
|
||||
adb shell cmd package list packages --show-versioncode
|
||||
```
|
||||
|
||||
To re-pull the APKs themselves for reinstall:
|
||||
|
||||
```bash
|
||||
adb shell pm path com.example # -> package:/data/app/.../base.apk + splits
|
||||
adb pull /data/app/~~.../com.example-.../base.apk # repeat for each split; reinstall with
|
||||
adb install-multiple base.apk split_config.*.apk # install-multiple
|
||||
```
|
||||
|
||||
### 6.5 Vendor D2D and cloud (no-tech last mile)
|
||||
|
||||
- **Google** cloud backup + Pixel/Android cable "device-to-device" restore at setup. App data rides
|
||||
along *only* if the Google **account** is transferred (Block Store is account-bound); untick
|
||||
nothing under "Accounts". `allowBackup="false"` stops cloud backup but D2D still transfers app data
|
||||
unless the app sets `dataExtractionRules`/`android:fullBackupOnly` to opt out.
|
||||
- **Samsung Smart Switch** (PC or cable): backs up most app data, but **encrypted app data (e.g.
|
||||
WhatsApp), banking logins, and sideloaded APKs do not transfer** — handle those per §9/§10.
|
||||
- **Seedvault** (CalyxOS/GrapheneOS/LineageOS): the FOSS backup transport. On Android 15 builds its
|
||||
**device-to-device** mode (Expert settings) is default-on and captures far more app data than the
|
||||
old cloud-style transport, because it presents as a D2D migration to each app.
|
||||
|
||||
---
|
||||
|
||||
## 7. The death of `adb backup` (don't build a plan on it)
|
||||
|
||||
`adb backup` was deprecated in Android 10 and gutted in Android 12 (API 31): app data is excluded by
|
||||
default and only included if the app is **debuggable**, so on essentially all release apps across
|
||||
Android 12–16 `adb backup` produces an archive with no app data. Treat it as unavailable:
|
||||
|
||||
- Don't rely on it for a migration. Use root tooling (§4.1), on-device FOSS exporters (§6.2), or
|
||||
vendor/Seedvault D2D (§6.5) instead.
|
||||
- `adb restore` likewise cannot restore encrypted backups and is a dead end for modern data.
|
||||
- The `bmgr`/LocalTransport backup-manager plumbing still exists for *developers testing their own
|
||||
backup agents* (`adb shell bmgr …`), but it is a per-app developer tool, not a device-wide
|
||||
extractor — out of scope for a pre-wipe sweep.
|
||||
|
||||
---
|
||||
|
||||
## 8. EDL / firehose / BROM — dead or bootloader-locked devices
|
||||
|
||||
The lowest level, for devices that will not boot to Android *or* recovery. These talk to the SoC's
|
||||
emergency loader below the OS:
|
||||
|
||||
- **Qualcomm EDL (9008)** — [`bkerler/edl`](https://github.com/bkerler/edl) with a firehose
|
||||
programmer (`.mbn`/`.elf`) matching the SoC. `edl printgpt` lists partitions; `edl rl dumps
|
||||
--skip=userdata --genxml` dumps all partitions (minus the huge userdata) to a folder with a
|
||||
`rawprogram` XML; `edl r <partition> <out.img>` grabs one. Newer SoCs need `--loader=<file>`
|
||||
because loader autodetection no longer works.
|
||||
- **MediaTek BROM** — [`bkerler/mtkclient`](https://github.com/bkerler/mtkclient). `./mtk printgpt`,
|
||||
`./mtk rl out` dumps all partitions to `out/`, `./mtk r <partition> <out.img>` for one; `--preloader`
|
||||
for BROM mode.
|
||||
|
||||
The hard limit: **these dumps are ciphertext.** EDL reads the raw flash below the OS, so `userdata`
|
||||
comes out FBE-encrypted and is not decryptable without the keys (which live in secure hardware keyed
|
||||
by the lockscreen credential). EDL is therefore excellent for **small identity partitions**
|
||||
(EFS/persist/nvram — §4.3), full-flash forensic images, and un-bricking, but it is **not** a way to
|
||||
read a locked phone's app data. Treat downloaded dumps/tools as untrusted input: keep each in its own
|
||||
directory, pass paths as arguments, and run any helper scripts from outside the dump directory.
|
||||
|
||||
---
|
||||
|
||||
## 9. App-specific data that needs its own handling
|
||||
|
||||
Some high-value data lives behind app-level encryption and ignores the tier ladder:
|
||||
|
||||
- **WhatsApp** — local backup at `/sdcard/WhatsApp/Databases/msgstore.db.crypt15` (+ `/Media`). If
|
||||
the user enabled **end-to-end encrypted backups**, decryption needs the **64-digit key** (or the
|
||||
`encrypted_backup.key` file from `/data/data/com.whatsapp/files/`, root-only). Tools:
|
||||
[`wa-crypt-tools`](https://pypi.org/project/wa-crypt-tools/). Easiest migration is WhatsApp's own
|
||||
built-in transfer or Google-Drive backup with the same phone number — pull the `crypt15` + media
|
||||
as insurance, not as the primary path.
|
||||
- **Signal** — does **not** sync to any cloud. Migrate with Signal's in-app device-transfer, or its
|
||||
encrypted local backup (Settings → Chats → Backups) whose **30-digit passphrase you must record** —
|
||||
there is no recovery without it.
|
||||
- **Authenticator / 2FA** — Google Authenticator exports via its QR "export accounts" flow;
|
||||
**Aegis** (FOSS) exports an encrypted JSON vault. Do this *before* wiping; TOTP seeds not exported
|
||||
are gone.
|
||||
|
||||
---
|
||||
|
||||
## 10. What will NOT survive a wipe (plan around it)
|
||||
|
||||
No tier recovers these; the only mitigation is to migrate them *before* the wipe:
|
||||
|
||||
- **Hardware-backed Keystore keys** are non-exportable by design — key material never leaves the
|
||||
TEE/StrongBox. Any app that pinned a device-bound key (banking apps, some password managers,
|
||||
FIDO/passkeys stored in hardware) will treat the restored/new device as untrusted and re-enrol.
|
||||
- **Play Integrity / SafetyNet attestation** fails on rooted or custom-ROM devices independently of
|
||||
your data — so a perfectly-restored banking app may still refuse to run. That is an attestation
|
||||
problem, not a backup problem; don't chase it as data loss.
|
||||
- **SSAID / `ANDROID_ID`** changes on wipe (per app-signing-key + user), which can log apps out or
|
||||
reset entitlements even when their `/data/data` is restored byte-perfect.
|
||||
|
||||
Before wiping: export 2FA (§9), confirm password-manager vaults sync to their cloud, record Signal
|
||||
(§9) and WhatsApp keys, and re-enrol passkeys/banking on the new device rather than expecting a
|
||||
restore to carry them.
|
||||
|
||||
---
|
||||
|
||||
## 11. Pre-wipe checklist (the one-screen version)
|
||||
|
||||
1. **Decide the tier** (§1). Booting + unlockable → root (§4) or ADB (§6). Not booting → recovery
|
||||
(§3) → EDL (§8).
|
||||
2. **EFS/modem/persist first** (§4.3) — before any unlock or flash, off-device.
|
||||
3. **Media & docs** — `adb pull /sdcard` (§6.1).
|
||||
4. **App data** — root tar + dedicated tool (§4.1), or recovery-decrypted Data backup (§3).
|
||||
5. **Scoped dirs** — `Android/data`, `Android/obb` (§6.1).
|
||||
6. **PIM** — SMS/call-log/contacts via SMS Import-Export or vCard (§6.2).
|
||||
7. **App-specific** — WhatsApp/Signal/2FA via their own flows (§9).
|
||||
8. **App list + APKs** — `pm list packages -3`, pull splits (§6.4) to rebuild the app set.
|
||||
9. **System state the app tars miss** (§12.1) — keystore DB, SSAIDs, URI grants, doze whitelist,
|
||||
app-ops, launcher/shortcuts/QS/controls, Wi-Fi, SD-card UUIDs, root grants, old `packages.list`.
|
||||
10. **Baseline + app-sweep** the old ROM (`measure.sh`, `test-all-apps.sh`, §12.4) so the new one
|
||||
has something to be compared against.
|
||||
11. **Passwords apps keep themselves** — App Manager's KeyStore password, backup-app passphrases,
|
||||
Signal/WhatsApp keys (§9). Write them down *before* the wipe; none are recoverable after.
|
||||
12. **Verify the backups off-device** (open the archive, check `ls /data/data` really decrypted)
|
||||
*before* you type `fastboot -w` or confirm the reset. A backup you have not read back is not a
|
||||
backup.
|
||||
|
||||
---
|
||||
|
||||
## 12. Restoring onto a new ROM (root) — what a data tar does not carry
|
||||
|
||||
Learned from a full LineageOS-based A16 → Infinity-X A17 switch (Redmi Note 7, Magisk). Restoring
|
||||
`/data/data` + `user_de` + `Android/data` per §4.4 brings back most apps, but **every app gets a new
|
||||
UID** on the new ROM, and a lot of state is keyed by UID or lives in system files. The scripts in
|
||||
[`scripts/android/`](skills-public/scripts/android/) automate the parts below; they take the
|
||||
device from `ANDROID_SERIAL` like `adb` itself.
|
||||
|
||||
### 12.1 Grab these too, before the wipe (all root, all small)
|
||||
|
||||
| What | Where | Why it matters on restore |
|
||||
| :--- | :--- | :--- |
|
||||
| Old UID map | `/data/system/packages.list` | Remap everything keyed by UID (SSAID, keystore, grants) |
|
||||
| Keystore2 DB | `/data/misc/keystore/persistent.sqlite` | Apps whose keys vanish show "Keystore error"/log out (§12.3) |
|
||||
| SSAIDs | `/data/system/users/0/settings_ssaid.xml` | `ANDROID_ID` change logs apps out (GitHub, Bolt, Lovense, …) |
|
||||
| Persistent SAF grants | `/data/system/urigrants.xml` | Folder pickers (music players, camera, sync apps) lose their folders |
|
||||
| Doze whitelist | `/data/system/deviceidle.xml` | Notification/automation apps silently stop in background |
|
||||
| App-ops & special access | `cmd appops dump`, `cmd notification` policy access, `dumpsys package` | `MANAGE_EXTERNAL_STORAGE`, usage stats, DND, `READ_LOGS`, install-packages, notification listeners are **not** runtime permissions |
|
||||
| Launcher + pinned shortcuts | launcher DB, `/data/system_ce/0/shortcut_service/` | Home screen; pinned shortcuts (e.g. Ice Box) are owned by the system, not the launcher |
|
||||
| QS tiles, device controls | `settings get secure sysui_qs_tiles`; SystemUI `user_de` `files/controls_favorites.xml` + `shared_prefs/controls_prefs.xml` | Home Assistant controls, dev tiles |
|
||||
| Wi-Fi | `/data/misc/apex/com.android.wifi/` or `cmd wifi list-networks` + PSKs | Also note which SSIDs must **not** auto-join |
|
||||
| Root grants + Magisk settings | `magisk --sqlite "SELECT * FROM policies"`, `… FROM settings`, `… FROM denylist`, `/data/adb/modules` (note which are disabled) | UIDs change, grants don't follow; Zygisk off ⇒ DenyList silently does nothing |
|
||||
| Accounts | `dumpsys account`; `/data/system_ce/0/accounts_ce.db` | Shows which logins (2nd Google account, app-registered accounts) must be re-added by hand — they can't be transplanted |
|
||||
| SD-card volume IDs | `blkid` of each SD partition | Public volume ID = filesystem UUID; reformat ⇒ new ID ⇒ broken paths and grants |
|
||||
| Freezer state | `pm list packages -d` + the freezer app's own list | Updates/reinstalls unfreeze |
|
||||
|
||||
### 12.2 Restore order and traps
|
||||
|
||||
- **Update first, restore data second.** Installing the old APKs and then the store's update is
|
||||
fine; restoring data *into* an older APK than the data was written by is a downgrade — apps wipe
|
||||
or crash on it. Never re-run a full restore pass over updated apps; restore data only.
|
||||
- **Installer of record:** `pm set-installer` refuses even root. Reinstall the same APKs with
|
||||
`adb install-multiple -r -i <installer>` (keeps data) so Play/Aurora/F-Droid/Obtainium see their
|
||||
apps as updatable (`fix-installers.py`).
|
||||
- **New signature, same package** (forks such as Shizuku → Shizuku-Next for a newer Android): tar the
|
||||
data dirs, uninstall, install, untar, chown/chcon, re-grant, re-freeze — `reinstall-keep-data.sh`.
|
||||
- **System files are read only at `system_server` start.** Write `urigrants.xml`,
|
||||
`settings_ssaid.xml`, `shortcut_service/*` etc. (owner `system:system`, mode 600,
|
||||
`u:object_r:system_data_file:s0`), then `kill -9 $(pidof system_server)` — a soft restart that
|
||||
does **not** save over them first, which a normal reboot does. It locks the user: unlock with
|
||||
wake → swipe → `input text <PIN>` → ENTER (the bouncer screenshots black; retry if needed).
|
||||
- **fs-verity:** files under `shortcut_service` are verity-protected — `rm` then write, never edit.
|
||||
- **Restored caches can kill an app:** Ice Box OOM-crashed on a restored MMKV package cache. Drop
|
||||
`cache`, `code_cache`, WebView caches and per-app package-list caches; keep the real state.
|
||||
- **Re-map UID-keyed state** with the old/new `packages.list` (SSAID `name="<uid>"`, keystore
|
||||
`namespace`, Magisk `policies.uid`, doze lists are by package and need no remap).
|
||||
- **SD card:** if the partition had to be recreated, set the old UUID back
|
||||
(`tune2fs -U <old>` for ext4, `fsck` before and after) rather than fixing every app.
|
||||
- **Wi-Fi:** restored networks auto-join in their own order; re-add the ones that must stay
|
||||
manual with `cmd wifi add-network <ssid> wpa2 <psk> -d`.
|
||||
- **QS tiles** on A17: `settings put secure sysui_qs_tiles` is ignored; use
|
||||
`cmd statusbar set-tiles "<list>"`. Developer tiles must be enabled first
|
||||
(`pm enable 'com.android.settings/.development.qstile.DevelopmentTiles$SensorsOff'`).
|
||||
- **Launcher switch** (Nova had no build for the new Android): converting Nova's `favorites`
|
||||
table into Lawnchair's Launcher3 schema works (`nova-to-lawnchair.py`) — mark widgets
|
||||
`restored=1` so Launcher3 binds fresh widget IDs, and re-pin deep shortcuts for the new launcher.
|
||||
|
||||
**Banking-app root checks** are not a restore problem, but they show up right after one. Order
|
||||
that worked (2026-10, Magisk 31, A17): first the ROM's own spoofing settings, if it has any (PIF,
|
||||
TrickyStore target and keybox, hide developer status). If an app still crashes on purpose, look for
|
||||
an Xposed module that defuses that app's check. For PayPal 10.18 that was oslokrok on Vector, the
|
||||
LSPosed successor. Configure Vector headlessly with
|
||||
`/data/adb/modules/zygisk_vector/cli modules enable <pkg>` and `cli scope set <pkg> <app>/0`.
|
||||
**Take the app off the Magisk DenyList**: the DenyList strips every Zygisk module from the process,
|
||||
including the hook.
|
||||
|
||||
### 12.3 Keystore: transplanting app keys
|
||||
|
||||
Hardware-backed keys never leave the TEE, but keystore2 stores them **wrapped** in its SQLite DB.
|
||||
If the new ROM runs on the same TEE (compare `keyentry.km_uuid` old vs new — same device, same
|
||||
firmware ⇒ same), the old wrapped blobs still unwrap. Keys whose `blobmetadata` has tag 0
|
||||
(`EncryptedBy`) are additionally super-encrypted with the **old lock-screen credential's** key and
|
||||
cannot be moved. Domain 0 (APP) rows have `namespace = uid` → remap. Stop the apps, back up the live
|
||||
DB, insert `keyentry`/`blobentry`/`blobmetadata`/`keyparameter`/`keymetadata` with fresh ids, reboot.
|
||||
Apps that already regenerated keys on the new ROM also need their data re-restored to match.
|
||||
`scripts/android/keystore-transplant.py OLD_DB [--replace] OLD_UID:NEW_UID …` writes the SQL; dry-run
|
||||
it on a host copy of the live DB (`pragma integrity_check`) before applying with the device's own
|
||||
`sqlite3`. Tested: transplanted keys load with no keystore errors after a reboot. That does not
|
||||
bring back sessions the server ended, or apps that refuse to run because their build is too old.
|
||||
|
||||
### 12.4 Verify like a user would
|
||||
|
||||
- `test-all-apps.sh OUT` cold-starts every user app (unfreezing and re-freezing), screenshots it,
|
||||
and greps its log for crashes, ANRs and keystore/crypto errors → `results.tsv`. Look at every
|
||||
screenshot: logouts, licence checks, ToS/"what's new" dialogs and permission prompts don't crash.
|
||||
- Android 17 shell launches must match the activity's intent filter:
|
||||
`am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n 'pkg/act'`
|
||||
(single-quote it — component names contain `$`). `-n` alone gives "Access blocked".
|
||||
- `measure.sh OUT [pkg…]` snapshots memory, crashes/ANRs/lmk kills since boot and cold-launch
|
||||
times; run it on the old ROM before the wipe so there is a baseline.
|
||||
- Keep a running log of every security setting weakened during the restore (adb stay-awake, root
|
||||
grants, doze exemptions, …) with its revert command; the restore needs many of them and nobody
|
||||
remembers them afterwards.
|
||||
+36
-11
@@ -4,7 +4,7 @@ description: >-
|
||||
Use this skill when writing, reviewing, or debugging GSC (Game Script / Call of Duty script)
|
||||
or CSC — including entity-vs-level scope, threads and notify/wait semantics, endon/waittill
|
||||
discipline, undefined-value guards, the no-OOP/no-exceptions reality, asset precaching,
|
||||
per-frame cost, and the fact that GSC has no compiler, linter, or test harness worth the name.
|
||||
per-frame cost, block scoping, and compiling off-engine with gsc-tool.
|
||||
---
|
||||
|
||||
# GSC — Rules & Preferences
|
||||
@@ -13,10 +13,10 @@ Extends [`coding-dev.md`](skills-public/coding-dev.md). Everything there applies
|
||||
|
||||
GSC is the Call of Duty engine's own scripting language (`.gsc` server-side, `.csc`
|
||||
client-side). It is **not Lua, not JavaScript, and not C** despite resembling all three. It has
|
||||
no type system, no exceptions, no classes, no closures in most dialects, and — in a modded
|
||||
install — **no diagnostics beyond the in-game console**. That last fact drives most of what
|
||||
follows: the language will not tell you that you are wrong, so the code has to be written so
|
||||
that it cannot be.
|
||||
no type system, no exceptions, no classes, no closures in most dialects, and no test harness.
|
||||
It can be compiled off-engine (see §1), which catches syntax and scope errors — but nothing
|
||||
checks a type, a field that was never set, or a builtin that does not exist in the target title.
|
||||
So the code still has to be written so that it cannot be wrong.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Establish the dialect before writing anything.** IW3/IW4 (MW/MW2), IW5 (MW3), T5/T6 (Black
|
||||
@@ -27,18 +27,38 @@ that it cannot be.
|
||||
|
||||
---
|
||||
|
||||
## 1. There Is No Gate — Build One
|
||||
## 1. The Gate — `gsc-tool`, Then The Game
|
||||
|
||||
GSC has no `gsc -Wall`. What exists:
|
||||
**There is an off-engine GSC compiler: `gsc-tool`.** Reach for it first; it is
|
||||
easy to assume none exists and verify by loading a map instead.
|
||||
|
||||
```bash
|
||||
gsc-tool --mode comp --game <title> --system pc --dry path/to/script.gsc
|
||||
```
|
||||
|
||||
It exits non-zero and prints the engine's own diagnostic, e.g.
|
||||
`[ERROR]:compiler:script:97:25: local variable 'previous_count' not found`.
|
||||
|
||||
- **Use `--mode comp`, not `--mode parse`.** `parse` is syntax-only and happily
|
||||
accepts code the engine's compiler rejects, including the block-scope error
|
||||
above. `comp` with `--dry` writes nothing and does the real thing.
|
||||
- **`--game` must match the title**, and it is not the marketing name: MW2
|
||||
Campaign Remastered is `h2`, MWR is `h1`. The choices are `iw5 iw6 iw7 iw8
|
||||
iw9 s1 s2 s4 h1 h2 t6 t7 t8 t9 jup`. Wrong title, wrong builtin table.
|
||||
- Prebuilt binaries ship on its GitHub releases; there is no distro package.
|
||||
- Many mod trees already vendor it (Overlord has it under `deps/gsc-tool`).
|
||||
|
||||
What that still leaves:
|
||||
|
||||
| Check | How |
|
||||
| :--- | :--- |
|
||||
| **Syntax** | The engine's own compile at load. Read the console; a parse error names the file and line and then stops the whole script loading. |
|
||||
| **Syntax + semantics** | `gsc-tool --mode comp`. Put it in the build gate. |
|
||||
| **Builtin exists for this title** | `gsc-tool`'s own engine tables, e.g. `src/gsc/engine/h2_meth.cpp`. Grep before using a builtin you remember from another title. |
|
||||
| **Runtime errors** | In-game console / `developer 1`. Many mod loaders print a script stack trace. |
|
||||
| **Static lint** | Third-party only, and dialect-specific. Use one if it exists for the target title; otherwise there is none. |
|
||||
| **Tests** | None off-host. Behaviour is verified by playing the map. |
|
||||
|
||||
Therefore:
|
||||
So a compile is cheap and mandatory, and **behaviour is still unverified until
|
||||
the map runs.** Both halves matter:
|
||||
|
||||
- **`developer 1` (or the loader's equivalent) is mandatory during development.** Without it the
|
||||
engine suppresses script warnings and an undefined-value error looks like "the feature did
|
||||
@@ -48,7 +68,7 @@ Therefore:
|
||||
- **Instrument with `iprintln` / `iprintlnbold` / `println`** on the success *and* the failure
|
||||
path. A function that silently returns on a guard is indistinguishable from one that never ran.
|
||||
- **Say plainly in your report that GSC behaviour is unverified** unless you actually loaded the
|
||||
map and watched it. There is no test suite to hide behind.
|
||||
map and watched it. A clean `gsc-tool` compile proves the script loads, not that it works.
|
||||
|
||||
---
|
||||
|
||||
@@ -211,6 +231,11 @@ init()
|
||||
functions over reimplementing them. `maps\_utility` and `common_scripts\utility` already have
|
||||
flag, wait, and array helpers.
|
||||
- **`/* */` comments do not nest.** Commenting out a block that contains one breaks the file.
|
||||
- **A local declared inside a block is not visible after it.** Writing it in one `if` and reading
|
||||
it in a second does not compile — declare it unconditionally at function scope. Assigning
|
||||
`undefined` to an entity field unsets it, which is usually the right "restore nothing".
|
||||
- **No file-scope constants.** `MY_CONST = 0.2;` outside a function is not valid in most
|
||||
dialects; use a local at the point of use or a `level.` field set during init.
|
||||
- File and function size limits from the general rules apply, and matter more here than usual:
|
||||
there is no debugger worth the name, so a 300-line function is unanalysable.
|
||||
- **Delete dead code.** With no linter to flag it, commented-out GSC accumulates forever.
|
||||
|
||||
@@ -139,9 +139,9 @@ Attach an `AbortController` to every long-lived `fetch` and abort it in cleanup.
|
||||
- **Imports**: prefer the configured alias (`@/…`) over relative climbs.
|
||||
- Indentation and quote/semicolon style: **match the file you are editing**. Do not introduce
|
||||
a new formatter or lint config into a project that already has one.
|
||||
- **One file until it earns a split** — around 250 lines for a self-contained plugin/feature,
|
||||
or when a second file is genuinely reusable (`utils.ts`, `types.ts`, `components/`).
|
||||
Hard limits from the general rules still apply: ESLint's `max-lines` and
|
||||
- **One file until it earns a split** — when it nears the 1000-line limit, or when a second
|
||||
file is genuinely reusable (`utils.ts`, `types.ts`, `components/`).
|
||||
The limits from the general rules (1000 lines per file, 200 per function) apply: ESLint's `max-lines` and
|
||||
`max-lines-per-function` cover both natively, so prefer those over a separate test here.
|
||||
|
||||
---
|
||||
|
||||
@@ -225,9 +225,8 @@ When talking to a rate-limited API (a chat platform, a store, a tracker):
|
||||
function-length rule — `PLR0915` (`too-many-statements`, default 50) and `C901`
|
||||
(cyclomatic complexity) are the closest proxies and are worth enabling, but neither
|
||||
measures lines. Put **both** the function-length and file-length limits in a pytest
|
||||
case (see [`coding-dev.md` §4](skills-public/coding-dev.md)): file hard 1000 / soft 600 (soft
|
||||
prints a note rather than failing), function hard 100 measured by walking the `ast` rather than
|
||||
counting lines of text. **Guard the guard** with a test asserting the glob matched anything at
|
||||
case (see [`coding-dev.md` §4](skills-public/coding-dev.md)): file 1000 lines max, function
|
||||
200 lines max measured by walking the `ast` rather than counting lines of text. **Guard the guard** with a test asserting the glob matched anything at
|
||||
all — a pattern that matches nothing makes every limit vacuous and still passes.
|
||||
|
||||
---
|
||||
|
||||
@@ -73,8 +73,12 @@ workspace = true
|
||||
wrong for the *entire* codebase — not a shortcut for one stubborn call site. It silences
|
||||
the rule in code nobody has written yet.
|
||||
- **Re-test every `#[allow]` periodically.** If removing one produces no warning, delete it.
|
||||
- `too_many_lines` is the mechanical enforcement of the 100-line function limit. A function
|
||||
that trips it wants extracting, not an allow.
|
||||
- `too_many_lines` is the mechanical enforcement of the 200-line function limit. Clippy's
|
||||
default threshold is 100, so set it in `clippy.toml`:
|
||||
```toml
|
||||
too-many-lines-threshold = 200
|
||||
```
|
||||
A function that trips it wants extracting, not an allow.
|
||||
- Clippy has **no file-length lint**, so the 1000-line file limit belongs in a test — see
|
||||
[`coding-dev.md` §4](skills-public/coding-dev.md) for the pattern. Put it in the workspace
|
||||
root's `tests/` so one test covers every member crate, walking `src/` plus each
|
||||
|
||||
+26
-18
@@ -26,6 +26,14 @@ reasons attached, not style trivia — most exist because ignoring them caused a
|
||||
> When in doubt about which mode you are in, **ask** — it changes the answer on strictness,
|
||||
> file size, typing, and test policy all at once.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **A new software idea starts with research, not code.** When the user describes a tool, app,
|
||||
> service, or other project they want built, follow
|
||||
> [`software-vision.md`](skills-public/software-vision.md) first: find existing projects to use,
|
||||
> extend, or fork (web searches and user reviews first, then training-data recall, then
|
||||
> throwaway clones), and present the options before scaffolding anything — unless the user
|
||||
> explicitly opted out of research.
|
||||
|
||||
Per-language specifics live in [`code-languages/`](skills-public/code-languages/):
|
||||
|
||||
| Language | Rules |
|
||||
@@ -115,7 +123,9 @@ a fresh implementation will rediscover slowly and incompletely.
|
||||
|
||||
Before hand-rolling any subsystem — a protocol client, a parser, a date formatter, a
|
||||
debounce, a retry/backoff, a semver comparator — **search first and state the finding.**
|
||||
Prefer official/first-party SDKs over third-party wrappers.
|
||||
Prefer official/first-party SDKs over third-party wrappers. The same ordering applies to a
|
||||
whole project idea, not just a subsystem — see
|
||||
[`software-vision.md`](skills-public/software-vision.md).
|
||||
|
||||
### The corollary: new generic code becomes its own package
|
||||
|
||||
@@ -138,19 +148,23 @@ depends on those.
|
||||
|
||||
## 4. Size Limits — Files and Functions
|
||||
|
||||
| Unit | Soft target | Split at | Hard limit |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **Source file** | ~400 lines | 600 lines | **1000 lines** |
|
||||
| **Function** | under 50 lines | — | **100 lines** |
|
||||
| **Function parameters** | 3–4 | — | ~5, then pass a struct/object |
|
||||
| **Nesting depth** | 2 | — | 3, then extract |
|
||||
| Unit | Limit |
|
||||
| :--- | :--- |
|
||||
| **Source file** | **1000 lines** max |
|
||||
| **Function** | **200 lines** max |
|
||||
| **Function parameters** | ~5, then pass a struct/object |
|
||||
| **Nesting depth** | 3, then extract |
|
||||
|
||||
These are the limits, not targets to stay well under: there is no earlier "split at" point.
|
||||
A file of 700 lines with one coherent responsibility is fine and is not split for its size.
|
||||
|
||||
### Enforce both mechanically, never by eye
|
||||
|
||||
A limit nobody checks is a suggestion. Pick the **highest** applicable option:
|
||||
|
||||
1. **A linter rule**, where one exists — the function limit is usually covered natively
|
||||
(Clippy's `too_many_lines`, ESLint's `max-lines-per-function`, `golangci-lint`'s `funlen`).
|
||||
(Clippy's `too_many_lines` with `too-many-lines-threshold = 200`, ESLint's
|
||||
`max-lines-per-function`, `golangci-lint`'s `funlen`), configured to 200 lines.
|
||||
A function that trips it wants extracting, not a suppression. Note that some linters only
|
||||
offer *proxies* for length — Ruff's `PLR0915` counts statements, not lines — in which case
|
||||
enable the proxy and put the real limit in a test.
|
||||
@@ -160,24 +174,18 @@ A limit nobody checks is a suggestion. Pick the **highest** applicable option:
|
||||
repository rather than living in one machine's shell script.
|
||||
3. **The build script**, only as a fallback when the project has no test suite to hang it on.
|
||||
|
||||
Whichever you use, it should **fail over the hard limit** and separately *name* anything past
|
||||
the soft split point — so the seam gets found while splitting is still cheap, rather than at
|
||||
the moment the hard limit finally trips.
|
||||
Whichever you use, it should **fail over the limit** and name the offending files.
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn no_source_file_exceeds_limits() {
|
||||
const HARD: usize = 1000;
|
||||
const WARN: usize = 600;
|
||||
const LIMIT: usize = 1000;
|
||||
let mut over = Vec::new();
|
||||
|
||||
for path in source_files() {
|
||||
let lines = std::fs::read_to_string(&path).unwrap().lines().count();
|
||||
if lines > HARD {
|
||||
over.push(format!("{} — {lines} lines (limit {HARD})", path.display()));
|
||||
} else if lines > WARN {
|
||||
// Not a failure: surfaces the seam while splitting is still cheap.
|
||||
eprintln!("note: {} is {lines} lines, approaching the {HARD} limit", path.display());
|
||||
if lines > LIMIT {
|
||||
over.push(format!("{} — {lines} lines (limit {LIMIT})", path.display()));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Executable
+49
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Reinstall each restored app with its original installer (pm set-installer refuses root).
|
||||
|
||||
Same APKs, `install-multiple -r -i <installer>` keeps data. The device comes from ANDROID_SERIAL, as for adb. Usage: fix-installers.py BACKUP_DIR WORK_DIR
|
||||
"""
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
backup, work = Path(sys.argv[1]), Path(sys.argv[2])
|
||||
log_file = (work / "restore.log").open("a")
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
line = f"{time.strftime('%H:%M:%S')} {msg}"
|
||||
print(line, flush=True)
|
||||
log_file.write(line + "\n")
|
||||
log_file.flush()
|
||||
|
||||
|
||||
def adb(*args: str, timeout: int = 900) -> str:
|
||||
r = subprocess.run(["adb", *args], capture_output=True, text=True, timeout=timeout)
|
||||
return (r.stdout + r.stderr).strip()
|
||||
|
||||
|
||||
def installers(text: str) -> dict:
|
||||
# `pm list packages -i` prints "package:PKG", with -f "package:PATH=PKG".
|
||||
return dict(re.findall(r"[:=]([\w.]+)\s+installer=(\S+)", text))
|
||||
|
||||
|
||||
want = installers((backup / "rooted-fresh/packages-3.txt").read_text())
|
||||
now = installers(adb("shell", "pm list packages -3 -i"))
|
||||
disabled = set(re.findall(r"package:([\w.]+)", adb("shell", "pm list packages -d")))
|
||||
log("=== installer fix-up start")
|
||||
for pkg, inst in sorted(want.items()):
|
||||
if inst == "null" or pkg not in now or now[pkg] == inst or inst not in now | {"com.android.vending": 1}:
|
||||
continue
|
||||
apks = sorted(str(p) for p in (backup / "apks" / pkg).glob("*.apk"))
|
||||
if not apks:
|
||||
continue
|
||||
# Reinstalling re-enables a frozen app; disable it again afterwards.
|
||||
out = adb("install-multiple", "-r", "-d", "-i", inst, "--bypass-low-target-sdk-block", *apks)
|
||||
if pkg in disabled:
|
||||
adb("shell", f"su -c 'pm disable-user --user 0 {pkg}'")
|
||||
log(f"{pkg}: installer {now[pkg]} -> {inst}: {'OK' if 'Success' in out else 'FAILED ' + out[-200:]}")
|
||||
log("=== installer fix-up done")
|
||||
Executable
+74
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Emit SQL that copies apps' keystore2 keys from an old ROM's DB into the live one, under the new UIDs.
|
||||
|
||||
Only works when both ROMs use the same TEE (compare keyentry.km_uuid). Keys super-encrypted with the
|
||||
old lock-screen credential (blobmetadata tag 0, EncryptedBy) cannot be moved and are skipped.
|
||||
|
||||
Usage: keystore-transplant.py OLD_DB [--replace] OLD_UID:NEW_UID [...] > transplant.sql
|
||||
--replace first delete the app's keys that already exist on the new ROM (needed when it regenerated them)
|
||||
Apply with the apps force-stopped: `su -c 'sqlite3 /data/misc/keystore/persistent.sqlite < transplant.sql'`, reboot.
|
||||
"""
|
||||
|
||||
import secrets
|
||||
import sqlite3
|
||||
import sys
|
||||
|
||||
DOMAIN_APP = 0
|
||||
TAG_ENCRYPTED_BY = 0
|
||||
|
||||
|
||||
def lit(v: object) -> str:
|
||||
if v is None:
|
||||
return "NULL"
|
||||
if isinstance(v, bytes):
|
||||
return f"X'{v.hex()}'"
|
||||
if isinstance(v, str):
|
||||
return "'" + v.replace("'", "''") + "'"
|
||||
return str(v)
|
||||
|
||||
|
||||
def delete_keys(where: str) -> list[str]:
|
||||
ids = f"(SELECT id FROM keyentry WHERE {where})"
|
||||
return [f"DELETE FROM blobmetadata WHERE blobentryid IN (SELECT id FROM blobentry WHERE keyentryid IN {ids});",
|
||||
f"DELETE FROM blobentry WHERE keyentryid IN {ids};",
|
||||
f"DELETE FROM keyparameter WHERE keyentryid IN {ids};",
|
||||
f"DELETE FROM keymetadata WHERE keyentryid IN {ids};",
|
||||
f"DELETE FROM grant WHERE keyentryid IN {ids};",
|
||||
f"DELETE FROM keyentry WHERE {where};"]
|
||||
|
||||
|
||||
args = sys.argv[1:]
|
||||
old = sqlite3.connect(f"file:{args.pop(0)}?mode=ro", uri=True)
|
||||
replace = "--replace" in args
|
||||
pairs = [tuple(map(int, a.split(":"))) for a in args if a != "--replace"]
|
||||
# Older keystore2 schemas have no blobentry.state; every blob there is current (0).
|
||||
blob_state = "state" if any(c[1] == "state" for c in old.execute("PRAGMA table_info(blobentry)")) else "0"
|
||||
|
||||
out = ["BEGIN;"]
|
||||
for old_ns, new_ns in pairs:
|
||||
if replace:
|
||||
out += delete_keys(f"domain={DOMAIN_APP} AND namespace={new_ns}")
|
||||
keys = old.execute("SELECT id, key_type, alias, state, km_uuid FROM keyentry WHERE domain=? AND namespace=?",
|
||||
(DOMAIN_APP, old_ns)).fetchall()
|
||||
for kid, key_type, alias, state, km_uuid in keys:
|
||||
blobs = old.execute(f"SELECT id, subcomponent_type, blob, {blob_state} FROM blobentry WHERE keyentryid=?", (kid,)).fetchall()
|
||||
if any(old.execute("SELECT 1 FROM blobmetadata WHERE blobentryid=? AND tag=?", (b[0], TAG_ENCRYPTED_BY)).fetchone()
|
||||
for b in blobs):
|
||||
print(f"-- skip {old_ns} {alias!r}: super-encrypted with the old lock-screen credential", file=sys.stderr)
|
||||
continue
|
||||
new_id = secrets.randbits(63) * (1 if secrets.randbits(1) else -1)
|
||||
# The old key wins over one the app generated under the same alias on the new ROM.
|
||||
out += delete_keys(f"domain={DOMAIN_APP} AND namespace={new_ns} AND alias={lit(alias)}")
|
||||
out.append(f"INSERT INTO keyentry (id,key_type,domain,namespace,alias,state,km_uuid) VALUES "
|
||||
f"({new_id},{key_type},{DOMAIN_APP},{new_ns},{lit(alias)},{state},{lit(km_uuid)});")
|
||||
for bid, sub, blob, bstate in blobs:
|
||||
out.append(f"INSERT INTO blobentry (subcomponent_type,keyentryid,blob,state) VALUES ({sub},{new_id},{lit(blob)},{bstate});")
|
||||
for tag, data in old.execute("SELECT tag, data FROM blobmetadata WHERE blobentryid=?", (bid,)):
|
||||
out.append(f"INSERT INTO blobmetadata (blobentryid,tag,data) VALUES ((SELECT max(id) FROM blobentry WHERE keyentryid={new_id}),{tag},{lit(data)});")
|
||||
for tag, data, level in old.execute("SELECT tag, data, security_level FROM keyparameter WHERE keyentryid=?", (kid,)):
|
||||
out.append(f"INSERT INTO keyparameter (keyentryid,tag,data,security_level) VALUES ({new_id},{tag},{lit(data)},{level});")
|
||||
for tag, data in old.execute("SELECT tag, data FROM keymetadata WHERE keyentryid=?", (kid,)):
|
||||
out.append(f"INSERT INTO keymetadata (keyentryid,tag,data) VALUES ({new_id},{tag},{lit(data)});")
|
||||
print(f"-- {old_ns}->{new_ns} {alias!r}: {len(blobs)} blobs", file=sys.stderr)
|
||||
out.append("COMMIT;")
|
||||
print("\n".join(out))
|
||||
Executable
+8
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# Cold-start an app's launcher activity the way Android 17 accepts from the shell. Usage: launch.sh PKG [WAIT] [SHOT.png]
|
||||
p=$1 w=${2:-6} shot=${3:-}
|
||||
act=$(adb shell "cmd package resolve-activity --brief -c android.intent.category.LAUNCHER $p" </dev/null | tr -d '\r' | tail -1)
|
||||
adb shell "am start -S -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n '$act'" </dev/null >/dev/null
|
||||
sleep "$w"
|
||||
[ -n "$shot" ] && adb exec-out screencap -p > "$shot" </dev/null
|
||||
true
|
||||
Executable
+42
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env bash
|
||||
# Snapshot phone health: memory, load, crashes/ANRs since boot, low-memory kills, log errors, cold launches.
|
||||
# Usage: measure.sh OUT_FILE [launch-package ...]
|
||||
set -uo pipefail
|
||||
out=$1; shift
|
||||
a() { adb shell "$@" </dev/null; }
|
||||
|
||||
{
|
||||
echo "# $(date '+%F %T') $(a getprop ro.build.display.id) / Android $(a getprop ro.build.version.release)"
|
||||
echo "## system"
|
||||
a 'uptime; cat /proc/loadavg'
|
||||
a 'grep -E "MemTotal|MemAvailable|Cached:|SwapTotal|SwapFree" /proc/meminfo'
|
||||
a 'su -c "cat /proc/sys/vm/swappiness; cat /sys/block/zram0/mm_stat 2>/dev/null"'
|
||||
echo "procs: $(a 'ls -d /proc/[0-9]* | wc -l')"
|
||||
echo "## top cpu"
|
||||
a 'top -b -n 1 -m 12 -o PID,USER,%CPU,%MEM,NAME' | tail -13
|
||||
echo "## dropbox since boot"
|
||||
boot=$(a 'date -d @$(( $(date +%s) - $(cut -d. -f1 /proc/uptime) )) "+%Y-%m-%d %H:%M:%S"')
|
||||
a "dumpsys dropbox" | awk -v b="$boot" '$1" "$2 >= b && $3 ~ /anr|crash|watchdog|tombstone|wtf|lowmem|native/ {print $3}' | sort | uniq -c
|
||||
echo "## crash buffer (FATAL / ANR per process)"
|
||||
a 'logcat -d -b crash' | grep -oE "Process: [^,]+|FATAL EXCEPTION|>>> [^ ]+ <<<" | sort | uniq -c | sort -rn | head -20
|
||||
a 'logcat -d -b events' | grep -oE "am_anr: \[[^]]*" | awk -F, '{print "ANR " $3}' | sort | uniq -c
|
||||
echo "## lmkd / am_kill"
|
||||
a 'logcat -d -b events' | grep -cE "am_kill|lmk_kill" | sed 's/^/kills: /'
|
||||
a 'logcat -d -b events' | grep -oE "am_kill: \[[^]]*" | awk -F, '{print $5}' | sort | uniq -c | sort -rn | head -10
|
||||
echo "## logcat error tags (top 25)"
|
||||
a 'logcat -d *:E' | awk '{print $6}' | sort | uniq -c | sort -rn | head -25
|
||||
echo "## thermal / battery"
|
||||
a 'dumpsys battery | grep -E "level|temperature|status"; for z in /sys/class/thermal/thermal_zone*; do echo "$(cat $z/type) $(cat $z/temp)"; done 2>/dev/null | sort -k2 -n | tail -5'
|
||||
if [ $# -gt 0 ]; then
|
||||
echo "## cold launches (ms, am start -W -S, 3 runs)"
|
||||
for p in "$@"; do
|
||||
t=""
|
||||
for _ in 1 2 3; do
|
||||
act=$(a "cmd package resolve-activity --brief -c android.intent.category.LAUNCHER $p" | tr -d '\r' | tail -1)
|
||||
v=$(a "am start -W -S -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n '$act'" | awk '/TotalTime/{print $2}')
|
||||
t="$t ${v:-fail}"; a 'input keyevent KEYCODE_HOME'; sleep 2
|
||||
done
|
||||
echo "$p:$t"
|
||||
done
|
||||
fi
|
||||
} 2>&1 | tee "$out"
|
||||
Executable
+88
@@ -0,0 +1,88 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild the Nova home screen in Lawnchair, plus the Ice Box shortcuts Nova had pinned.
|
||||
|
||||
Usage: nova-to-lawnchair.py NOVA_DB LAWNCHAIR_DB OLD_ICEBOX_SHORTCUT_XML OUT_DIR
|
||||
|
||||
Writes OUT_DIR/launcher_5_5_5.db (Lawnchair schema, Nova's items), OUT_DIR/icebox.xml
|
||||
(shortcut_service package file with only the pinned shortcuts) and OUT_DIR/lawnchair-pins.xml
|
||||
(shortcut_service launcher file pinning them for Lawnchair).
|
||||
"""
|
||||
|
||||
import re
|
||||
import shutil
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
LAUNCHER = "app.lawnchair.nightly"
|
||||
ICEBOX = "com.catchingnow.icebox"
|
||||
# From /data/system/packages.xml on the new ROM: versionCode, lastUpdateTime, SHA-256 of the cert.
|
||||
LAUNCHER_INFO = ("1600020100", "1792700106881", "dHw2RbNXJYsuI+hR5TyWdH/grdAH5bos2X6MhVcuTcU=")
|
||||
ICEBOX_INFO = ("154821395", "1792699567277", "gvW+cC4DIBdGDKg3b22r1yWp5S3SSSDdXl17WF+/4bY=")
|
||||
# Nova-only containers (gesture/dock-swipe slots); Lawnchair has nothing to put them in.
|
||||
SKIP_CONTAINERS = {-207, -208}
|
||||
ITEM_APP, ITEM_FOLDER, ITEM_WIDGET, ITEM_DEEP_SHORTCUT = 0, 2, 4, 6
|
||||
FLAG_ID_NOT_VALID = 1 # Launcher3 restore flag: allocate and bind a fresh widget id on load
|
||||
MISSING_PACKAGES = {"com.dot.gallery"} # not installed on the new ROM
|
||||
# Google's smartspace widget on row 0 of the first page: Lawnchair reserves that row for its own.
|
||||
SKIP_IDS = {56}
|
||||
|
||||
nova_db, lc_db, icebox_xml, out = map(Path, sys.argv[1:5])
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def package_info(version: str, updated: str, sig: str) -> str:
|
||||
return (f'<package-info version="{version}" last_udpate_time="{updated}" shadow="0" allow-backup="1" '
|
||||
f'allow-backup-initialized="1" bk_src_version="-1" bk_src_backup-allowed="0">\n'
|
||||
f'<signature hash="{sig}" />\n</package-info>\n')
|
||||
|
||||
|
||||
nova = sqlite3.connect(nova_db)
|
||||
nova.row_factory = sqlite3.Row
|
||||
rows = [r for r in nova.execute("select * from favorites where _id >= 0 order by _id")
|
||||
if r["container"] not in SKIP_CONTAINERS and r["_id"] not in SKIP_IDS]
|
||||
folders = {r["_id"] for r in rows if r["itemType"] == ITEM_FOLDER}
|
||||
kept = []
|
||||
for r in rows:
|
||||
pkg = re.search(r"component=([\w.]+)/", r["intent"] or "")
|
||||
if pkg and pkg.group(1) in MISSING_PACKAGES:
|
||||
print(f"skip {r['title']}: {pkg.group(1)} not installed")
|
||||
continue
|
||||
if r["container"] >= 0 and r["container"] not in folders:
|
||||
continue
|
||||
kept.append(r)
|
||||
|
||||
db = out / "launcher_5_5_5.db"
|
||||
shutil.copy(lc_db, db)
|
||||
lc = sqlite3.connect(db)
|
||||
lc.execute("delete from favorites")
|
||||
pins = []
|
||||
for r in kept:
|
||||
widget = r["itemType"] == ITEM_WIDGET
|
||||
hotseat = r["container"] == -101
|
||||
in_folder = r["container"] >= 0
|
||||
x, y = int(max(r["cellX"] or 0, 0)), int(max(r["cellY"] or 0, 0))
|
||||
lc.execute(
|
||||
"insert into favorites (_id,title,intent,container,screen,cellX,cellY,spanX,spanY,itemType,"
|
||||
"appWidgetId,appWidgetProvider,restored,profileId,rank,options) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(r["_id"], r["title"], r["intent"], r["container"],
|
||||
x if hotseat else (0 if in_folder else r["screen"]),
|
||||
x, 0 if hotseat else y, int(r["spanX"]), int(r["spanY"]), r["itemType"],
|
||||
r["appWidgetId"] if widget else -1, r["appWidgetProvider"],
|
||||
FLAG_ID_NOT_VALID if widget else 0, 0, r["rank"], 0))
|
||||
if r["itemType"] == ITEM_DEEP_SHORTCUT:
|
||||
pins.append(re.search(r"S\.shortcut_id=([^;]+)", r["intent"]).group(1))
|
||||
lc.commit()
|
||||
print(f"{len(kept)} items written, pins: {pins}")
|
||||
|
||||
old = Path(icebox_xml).read_text()
|
||||
blocks = [m.group(0) for m in re.finditer(r'<shortcut id="([^"]+)".*?</shortcut>\n', old, re.S) if m.group(1) in pins]
|
||||
assert len(blocks) == len(pins), f"found {len(blocks)} of {len(pins)} shortcuts"
|
||||
head = re.search(r"<package name=[^>]*>\n", old).group(0)
|
||||
(out / "icebox.xml").write_text("<?xml version='1.0' encoding='utf-8' standalone='yes' ?>\n" + head
|
||||
+ package_info(*ICEBOX_INFO) + "".join(blocks) + "</package>\n")
|
||||
(out / "lawnchair-pins.xml").write_text(
|
||||
"<?xml version='1.0' encoding='utf-8' standalone='yes' ?>\n"
|
||||
f'<launcher-pins package-name="{LAUNCHER}" launcher-user="0">\n' + package_info(*LAUNCHER_INFO)
|
||||
+ f'<package package-name="{ICEBOX}" package-user="0">\n'
|
||||
+ "".join(f'<pin value="{p}" />\n' for p in pins) + "</package>\n</launcher-pins>\n")
|
||||
Executable
+40
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env bash
|
||||
# Replace an app with a differently signed build while keeping its data, permissions,
|
||||
# installer and frozen state. Data is also saved to BACKUP_DIR before the uninstall.
|
||||
# Usage: reinstall-keep-data.sh PACKAGE BACKUP_DIR APK [APK ...]
|
||||
set -euo pipefail
|
||||
pkg=$1 bak=$2; shift 2
|
||||
a() { adb shell "$@" </dev/null; }
|
||||
su_() { a "su -c '$1'"; }
|
||||
|
||||
mkdir -p "$bak"
|
||||
installer=$(a "pm list packages -3 -i $pkg" | tr -d '\r' | grep "package:$pkg " | sed 's/.*installer=//')
|
||||
disabled=$(a "pm list packages -d $pkg" | tr -d '\r' | grep -cx "package:$pkg" || true)
|
||||
perms=$(a "dumpsys package $pkg" | sed -n '/runtime permissions:/,/^ *[a-z]* .*:$/p' | grep -oE '[a-zA-Z0-9_.]+: granted=true' | cut -d: -f1 | sort -u | tr '\n' ' ')
|
||||
echo "$pkg: installer=$installer disabled=$disabled perms=[$perms]"
|
||||
|
||||
dirs="/data/data/$pkg /data/user_de/0/$pkg"
|
||||
[ "$(su_ "[ -d /data/media/0/Android/data/$pkg ] && echo y")" = y ] && dirs="$dirs /data/media/0/Android/data/$pkg"
|
||||
su_ "am force-stop $pkg"
|
||||
# The phone writes the tar to stdout; exec-out keeps it binary-safe.
|
||||
adb exec-out "su -c 'tar -cf - --exclude=$pkg/cache --exclude=$pkg/code_cache $dirs 2>/dev/null'" > "$bak/$pkg.tar"
|
||||
echo "saved $(stat -c %s "$bak/$pkg.tar") bytes to $bak/$pkg.tar"
|
||||
[ "$(stat -c %s "$bak/$pkg.tar")" -gt 10240 ] || { echo "backup suspiciously small, aborting"; exit 1; }
|
||||
|
||||
a "pm uninstall $pkg"
|
||||
iopt=(); [ -n "$installer" ] && [ "$installer" != null ] && iopt=(-i "$installer")
|
||||
adb install-multiple -r "${iopt[@]}" --bypass-low-target-sdk-block "$@"
|
||||
|
||||
uid=$(su_ "stat -c %u /data/data/$pkg")
|
||||
cache_gid=$((uid - 10000 + 20000))
|
||||
adb exec-in "su -c 'tar -xf - -C /'" < "$bak/$pkg.tar"
|
||||
su_ "while pgrep -x tar >/dev/null; do sleep 0.3; done"
|
||||
for d in /data/data/$pkg /data/user_de/0/$pkg; do
|
||||
su_ "chown -R $uid:$uid $d; for c in cache code_cache; do mkdir -p $d/\$c; chown $uid:$cache_gid $d/\$c; chmod 2771 $d/\$c; done; chcon -hR \"\$(stat -c %C $d)\" $d"
|
||||
done
|
||||
case " $dirs " in *Android/data*)
|
||||
su_ "d=/data/media/0/Android/data/$pkg; chown -R $uid:1078 \$d; find \$d -type d -exec chmod 2770 {} +; find \$d -type f -exec chmod 660 {} +; chcon -R u:object_r:media_rw_data_file:s0 \$d" ;;
|
||||
esac
|
||||
for p in $perms; do a "pm grant $pkg $p" 2>/dev/null || true; done
|
||||
[ "$disabled" = 1 ] && su_ "pm disable-user --user 0 $pkg"
|
||||
echo "$pkg: reinstalled as uid $uid, $(a "dumpsys package $pkg" | grep -m1 versionName | xargs)"
|
||||
Executable
+144
@@ -0,0 +1,144 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Restore user apps, app data and permissions onto a freshly flashed, rooted phone.
|
||||
|
||||
Sources (from the backup folder):
|
||||
apks/<pkg>/*.apk APK + splits
|
||||
rooted-fresh/dumpsys-package.txt installer, granted runtime permissions
|
||||
rooted-fresh/packages-3.txt which packages are user apps
|
||||
rooted-fresh/packages-disabled.txt frozen/disabled state
|
||||
rooted/apps-manifest.json special app-ops (overlay, all-files, ...)
|
||||
<work>/data/<pkg>, <work>/user_de/0/<pkg> unpacked data-fresh.tar
|
||||
|
||||
The device comes from ANDROID_SERIAL, as for adb. Usage: restore.py BACKUP_DIR WORK_DIR [--data-only] [pkg ...] (no pkgs = all user apps)
|
||||
--data-only only put the data back (apps already installed and updated): keeps the current APK,
|
||||
installer, permissions and frozen state. Never re-run a full restore over updated apps.
|
||||
Every action and its result is appended to WORK_DIR/restore.log.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
SKIP = {"com.topjohnwu.magisk"} # installed fresh, its data must not be reused
|
||||
# Dropped on restore: regenerated by the app, and the bulk of most backups.
|
||||
VOLATILE = ["cache", "code_cache", "app_webview/Default/Cache", "app_webview/Default/Code Cache"]
|
||||
|
||||
backup, work = Path(sys.argv[1]), Path(sys.argv[2])
|
||||
data_only = "--data-only" in sys.argv[3:]
|
||||
only = set(sys.argv[3:]) - {"--data-only"}
|
||||
log_file = (work / "restore.log").open("a")
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
line = f"{time.strftime('%H:%M:%S')} {msg}"
|
||||
print(line, flush=True)
|
||||
log_file.write(line + "\n")
|
||||
log_file.flush()
|
||||
|
||||
|
||||
def adb(*args: str, timeout: int = 300, stdin=None) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(["adb", *args], capture_output=True, text=stdin is None,
|
||||
timeout=timeout, stdin=stdin)
|
||||
|
||||
|
||||
def su(cmd: str, timeout: int = 300) -> str:
|
||||
r = adb("shell", f"su -c {shlex.quote(cmd)}", timeout=timeout)
|
||||
return (r.stdout + r.stderr).strip()
|
||||
|
||||
|
||||
def parse_dumpsys(text: str) -> dict:
|
||||
"""installer + granted runtime permissions per package."""
|
||||
out = {}
|
||||
for block in re.split(r"\n Package \[", text)[1:]:
|
||||
name = block.split("]", 1)[0]
|
||||
installer = re.search(r"\n installerPackageName=(\S+)", block)
|
||||
runtime = block.split("runtime permissions:", 1)
|
||||
perms = re.findall(r"\n ([\w.]+): granted=true", runtime[1]) if len(runtime) > 1 else []
|
||||
out[name] = {"installer": installer.group(1) if installer else None, "perms": sorted(set(perms))}
|
||||
return out
|
||||
|
||||
|
||||
fresh = backup / "rooted-fresh"
|
||||
meta = parse_dumpsys((fresh / "dumpsys-package.txt").read_text(errors="replace"))
|
||||
manifest = json.loads((backup / "rooted/apps-manifest.json").read_text())["apps"]
|
||||
user_pkgs = sorted(set(re.findall(r"=([\w.]+)\s+installer=", (fresh / "packages-3.txt").read_text())) - SKIP)
|
||||
disabled = set(re.findall(r"package:([\w.]+)", (fresh / "packages-disabled.txt").read_text()))
|
||||
targets = [p for p in user_pkgs if not only or p in only]
|
||||
log(f"=== restore start: {len(targets)} packages")
|
||||
|
||||
# Pass 1: install every APK set, so installers like Obtainium exist before pass 2 names them.
|
||||
for pkg in [] if data_only else targets:
|
||||
apks = sorted(str(p) for p in (backup / "apks" / pkg).glob("*.apk"))
|
||||
if not apks:
|
||||
log(f"{pkg}: NO APK, skipped")
|
||||
continue
|
||||
# --bypass-low-target-sdk-block: Android 14+ refuses targetSdk < 24 (Automagic workarounds).
|
||||
r = subprocess.run(["adb", "install-multiple", "-r", "-d", "-g",
|
||||
"--bypass-low-target-sdk-block", *apks],
|
||||
capture_output=True, text=True, timeout=900)
|
||||
log(f"{pkg}: install {'OK' if 'Success' in r.stdout else 'FAILED ' + (r.stdout + r.stderr).strip()[-200:]}")
|
||||
|
||||
installed = set(re.findall(r"package:([\w.]+)", adb("shell", "pm list packages").stdout))
|
||||
|
||||
# Pass 2: data, ownership, SELinux labels, installer, permissions, disabled state.
|
||||
for pkg in targets:
|
||||
if pkg not in installed:
|
||||
continue
|
||||
info = meta.get(pkg, {})
|
||||
su(f"am force-stop {pkg}")
|
||||
uid = su(f"stat -c %u /data/data/{pkg}")
|
||||
if not uid.isdigit():
|
||||
log(f"{pkg}: no data dir after install ({uid}), data skipped")
|
||||
continue
|
||||
cache_gid = int(uid) - 10000 + 20000 # u0_aNNN_cache
|
||||
|
||||
members = [m for m in (f"data/{pkg}", f"user_de/0/{pkg}") if (work / m).is_dir()]
|
||||
if members:
|
||||
for m in members:
|
||||
su(f"find /data/{m} -mindepth 1 -maxdepth 1 ! -name lib -exec rm -rf {{}} +")
|
||||
excludes = [f"--exclude={m}/{v}" for m in members for v in VOLATILE]
|
||||
tar = subprocess.Popen(["tar", "-cf", "-", "-C", str(work), *excludes, *members],
|
||||
stdout=subprocess.PIPE)
|
||||
r = adb("exec-in", "su -c 'tar -xf - -C /data'", stdin=tar.stdout, timeout=1800)
|
||||
tar.wait()
|
||||
# exec-in returns once our side is sent; the phone's tar may still be writing.
|
||||
su("while pgrep -x tar >/dev/null; do sleep 0.3; done", timeout=600)
|
||||
for m in members:
|
||||
su(f"chown -R {uid}:{uid} /data/{m}; "
|
||||
f"for d in cache code_cache; do mkdir -p /data/{m}/$d; chown {uid}:{cache_gid} /data/{m}/$d; "
|
||||
# restorecon gives only app_data_file:s0; the per-app MLS categories that
|
||||
# installd put on the top directory must be copied onto its contents.
|
||||
f"chmod 2771 /data/{m}/$d; done; chcon -hR \"$(stat -c %C /data/{m})\" /data/{m}",
|
||||
timeout=900)
|
||||
wrong = su(f"find {' '.join('/data/' + m for m in members)} ! -user {uid} | wc -l")
|
||||
log(f"{pkg}: data {'+'.join(members)} rc={r.returncode} uid={uid} wrong_owner={wrong}")
|
||||
else:
|
||||
log(f"{pkg}: no data in backup")
|
||||
if data_only:
|
||||
continue
|
||||
|
||||
inst = info.get("installer")
|
||||
if inst and inst != "null" and inst in installed | {"com.android.vending"}:
|
||||
log(f"{pkg}: installer {inst} -> {su(f'pm set-installer {pkg} {inst}') or 'OK'}")
|
||||
|
||||
# -g granted everything at install; revoke what was not granted before.
|
||||
wanted = set(info.get("perms", []))
|
||||
now = re.findall(r"\n\s+([\w.]+): granted=true", su(f"dumpsys package {pkg} | sed -n '/runtime permissions:/,$p'"))
|
||||
revoked = [p for p in set(now) - wanted]
|
||||
for p in revoked:
|
||||
su(f"pm revoke {pkg} {p}")
|
||||
for p in wanted - set(now):
|
||||
su(f"pm grant {pkg} {p}")
|
||||
ops = manifest.get(pkg, {}).get("specialAppOps", {})
|
||||
for op, mode in ops.items():
|
||||
su(f"appops set {pkg} {op} {mode}")
|
||||
log(f"{pkg}: perms granted={len(wanted)} revoked={len(revoked)} appops={','.join(ops) or '-'}")
|
||||
|
||||
if pkg in disabled:
|
||||
log(f"{pkg}: disable-user -> {su(f'pm disable-user --user 0 {pkg}')}")
|
||||
|
||||
log("=== restore done")
|
||||
Executable
+40
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env bash
|
||||
# Launch every user app with a launcher activity, one by one: unfreeze if frozen, cold start,
|
||||
# wait, screenshot, collect its crash/ANR/keystore log lines, check it is alive and in front,
|
||||
# then stop it and re-freeze. Writes OUT_DIR/results.tsv, OUT_DIR/<pkg>.png, OUT_DIR/<pkg>.log.
|
||||
# Usage: test-all-apps.sh OUT_DIR [pkg ...]
|
||||
set -uo pipefail
|
||||
out=$1; shift
|
||||
mkdir -p "$out"
|
||||
a() { adb shell "$@" </dev/null; }
|
||||
WAIT=${WAIT:-8}
|
||||
|
||||
if [ $# -gt 0 ]; then pkgs=("$@"); else
|
||||
mapfile -t pkgs < <(a 'pm list packages -3' | tr -d '\r' | sed 's/package://' | sort)
|
||||
fi
|
||||
frozen=$(a 'pm list packages -d' | tr -d '\r' | sed 's/package://')
|
||||
printf 'pkg\twas_frozen\tlaunch_ms\talive\tforeground\tcrash\tanr\tkeystore\tnote\n' > "$out/results.tsv"
|
||||
|
||||
for p in "${pkgs[@]}"; do
|
||||
wf=0; grep -qx "$p" <<<"$frozen" && wf=1
|
||||
[ $wf = 1 ] && a "su -c 'pm enable --user 0 $p'" >/dev/null
|
||||
act=$(a "cmd package resolve-activity --brief -c android.intent.category.LAUNCHER $p" | tr -d '\r' | tail -1)
|
||||
if [[ "$act" != */* ]]; then
|
||||
[ $wf = 1 ] && a "su -c 'pm disable-user --user 0 $p'" >/dev/null
|
||||
printf '%s\t%s\t-\t-\t-\t-\t-\t-\tno launcher activity\n' "$p" "$wf" >> "$out/results.tsv"; continue
|
||||
fi
|
||||
a 'logcat -c'
|
||||
ms=$(a "am start -W -S -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n '$act'" | awk '/TotalTime/{print $2}')
|
||||
sleep "$WAIT"
|
||||
adb exec-out screencap -p > "$out/$p.png" </dev/null
|
||||
pid=$(a "pidof $p" | tr -d '\r')
|
||||
fg=$(a 'dumpsys activity activities' | grep -m1 -E 'topResumedActivity|mResumedActivity' | grep -c "$p/")
|
||||
a 'logcat -d' > "$out/$p.log"
|
||||
crash=$(grep -cE "FATAL EXCEPTION|Process: $p|Fatal signal" "$out/$p.log")
|
||||
anr=$(grep -cE "ANR in $p" "$out/$p.log")
|
||||
ks=$(grep -ciE "AEADBadTag|KeyStoreException|VERIFICATION_FAILED|KeyPermanentlyInvalidated|UnrecoverableKey|BadPaddingException|file is not a database|SQLiteDatabaseCorrupt" "$out/$p.log")
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t\n' "$p" "$wf" "${ms:-?}" "${pid:+yes}" "$fg" "$crash" "$anr" "$ks" >> "$out/results.tsv"
|
||||
echo "$p frozen=$wf ms=${ms:-?} alive=${pid:+yes} fg=$fg crash=$crash anr=$anr ks=$ks"
|
||||
a "am force-stop $p; input keyevent KEYCODE_HOME"
|
||||
[ $wf = 1 ] && a "su -c 'pm disable-user --user 0 $p'" >/dev/null
|
||||
done
|
||||
@@ -0,0 +1,201 @@
|
||||
---
|
||||
name: software-vision
|
||||
description: >-
|
||||
Use this skill whenever the user describes an idea for a new piece of software, a tool, an app,
|
||||
a service, a bot, a plugin, a mod, or an automation — "I want something that …", "build me a …",
|
||||
"could we make a tool for …", "I need a service that …" — before choosing a language, scaffolding
|
||||
a repository, or writing any code. Requires prior-art research first: web searches for existing
|
||||
projects, forks, skeletons, libraries and what users say about them (reviews, complaints,
|
||||
comparisons), only then merged with training-data recall as a secondary source, a
|
||||
throwaway shallow clone of promising candidates to check them against the requirements, and a
|
||||
set of options (fork, depend on, extend, start fresh) for the user to choose from.
|
||||
---
|
||||
|
||||
# Software Vision — Find What Exists Before Building
|
||||
|
||||
When the user has an idea for a piece of software, the first job is **not** to design it. It is
|
||||
to find out what already exists that does all or most of it. Building a full stack from nothing
|
||||
when a project covering 80% of the requirements is one `git clone` away wastes days, and the
|
||||
result is usually worse than the thing that already had years of bug reports behind it.
|
||||
|
||||
This is the project-level form of the dependency ordering in
|
||||
[`coding-dev.md` §3](skills-public/coding-dev.md): use what exists, fork what is close, write
|
||||
your own only when nothing fits. That section governs individual subsystems; this one governs
|
||||
the whole idea.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Never assume nothing exists.** However niche the idea sounds, someone may have built it,
|
||||
> half-built it, abandoned it, drafted a skeleton, written a blog post about the protocol, or
|
||||
> published research on the hard part. "I couldn't think of anything" is not a finding — a list
|
||||
> of the searches you ran and what they turned up is.
|
||||
|
||||
---
|
||||
|
||||
## 1. When This Applies — and the Only Opt-Out
|
||||
|
||||
Run this whole process whenever the user describes software they want to exist: a new app,
|
||||
CLI, service, bot, plugin, mod, integration, library, or automation — whether it is a fresh
|
||||
repository or a large new component inside an existing one.
|
||||
|
||||
**The only way to skip it is an explicit opt-out from the user**, such as "no online research",
|
||||
"don't look at other projects", or "I want to write this from scratch myself". Then:
|
||||
|
||||
- Skip the web searches and the clones (§3–§4).
|
||||
- Still use the libraries rule in [`coding-dev.md` §3](skills-public/coding-dev.md) for
|
||||
individual subsystems unless the user excluded that too.
|
||||
- Do not argue for research afterwards; the user has already decided.
|
||||
|
||||
Urgency, a small-sounding scope, or "it's just a quick script" are **not** opt-outs. A few
|
||||
searches take a minute; reinventing a maintained tool takes far longer.
|
||||
|
||||
---
|
||||
|
||||
## 2. Pin Down the Requirements First
|
||||
|
||||
Before searching, turn the user's description into a short checklist, because that list is what
|
||||
every candidate gets measured against:
|
||||
|
||||
- **Must-haves** — what the software has to do to be useful at all.
|
||||
- **Nice-to-haves** — features the user mentioned that are not deal-breakers.
|
||||
- **Constraints** — platform (Linux, Windows, Wine, Android, a game engine), runtime, language
|
||||
preferences, self-hosted vs cloud, license needs, offline use, performance.
|
||||
|
||||
Show the checklist to the user with the results (§6) so they can correct it. If a requirement
|
||||
is genuinely ambiguous *and* would change which candidates qualify, ask — otherwise state the
|
||||
assumption and carry on.
|
||||
|
||||
---
|
||||
|
||||
## 3. Search First, Then Merge in Recall
|
||||
|
||||
The searches happen anyway, so they come **first**, and the decision rests on what they show.
|
||||
Training data can be months or years out of date: projects get abandoned, renamed, relicensed,
|
||||
or overtaken, and newer ones may never have been seen at all. Starting from recall anchors the
|
||||
whole evaluation on a stale picture, so recall comes second and only fills gaps.
|
||||
|
||||
### 3.1 Web searches first
|
||||
|
||||
**Run at least three to five searches**, varied in phrasing and source, before settling on any
|
||||
candidate. Cover:
|
||||
|
||||
| Where | What it finds |
|
||||
| :--- | :--- |
|
||||
| General web search with the problem phrased several ways ("self-hosted X", "open source alternative to Y", "X for Linux") | Products, blog posts, comparisons |
|
||||
| GitHub / GitLab / Codeberg search, topics, and `awesome-*` lists | Projects, forks, skeletons, abandoned prototypes |
|
||||
| The package registry for the likely language (crates.io, npm, PyPI, NuGet, pkg.go.dev, …) | Libraries that do the hard part |
|
||||
| Forums and communities (Reddit, Hacker News, the domain's own Discord/forum, issue trackers of adjacent projects) | "Does anything do X?" threads, half-finished attempts, known dead ends |
|
||||
| Papers, specs, reverse-engineering notes for niche domains | The research behind the hard part, even if nobody shipped an app |
|
||||
|
||||
Search for **parts** as well as the whole: an idea with no complete match often splits into
|
||||
pieces that each have a mature library or tool.
|
||||
|
||||
For each promising hit, note what can be checked without cloning: last commit or release date,
|
||||
license, language, open issue count and how maintainers respond, stars or downloads as a weak
|
||||
signal, and whether its README claims the must-haves.
|
||||
|
||||
### 3.2 What people say about it
|
||||
|
||||
A project's own README describes what it *claims* to do. What users say shows how it actually
|
||||
holds up, so search for opinions on every serious candidate too: reviews, "X vs Y" comparisons,
|
||||
Reddit and Hacker News threads, forum posts, and the complaints in its issue tracker. Look for:
|
||||
|
||||
- Recurring pain points (crashes, data loss, missing features people keep requesting)
|
||||
- Whether people are moving to it or away from it, and to what
|
||||
- How the maintainers respond to bugs and pull requests
|
||||
- Workarounds people rely on, which often point at the gaps a fork would need to fill
|
||||
|
||||
Weight recent opinions over old ones, and many independent voices over a single loud one. Put
|
||||
this in the comparison (§6) next to the technical facts.
|
||||
|
||||
### 3.3 Then merge in recall
|
||||
|
||||
Only after searching, add what you already know from training data: projects, products,
|
||||
libraries, and standards that the searches missed. Every recalled item must be checked with a
|
||||
quick search before it goes into the comparison (does it still exist, is it maintained, has the
|
||||
license changed?). Where recall and current sources disagree, **current sources win.**
|
||||
|
||||
---
|
||||
|
||||
## 4. Clone the Promising Candidates — Then Delete Them
|
||||
|
||||
When a candidate looks close but it is unclear whether it ticks every must-have, **look inside
|
||||
it** rather than guessing from the README.
|
||||
|
||||
1. **Shallow-clone into a throwaway directory** — the session's scratchpad or a fresh `mktemp -d`,
|
||||
never inside a project tree:
|
||||
```bash
|
||||
dir=$(mktemp -d) && git clone --depth 1 <url> "$dir/<name>"
|
||||
```
|
||||
2. **Read, don't run.** A clone is untrusted code. Inspect the README, directory layout, entry
|
||||
points, dependency manifest, build files, tests, and open issues. Do not run its install
|
||||
scripts, build, or test suite as part of evaluation — that is a decision for after the user
|
||||
picks it.
|
||||
3. **Map it against the checklist from §2**: which must-haves it covers, which are partial, which
|
||||
are missing, and roughly how much work each gap is. Note the architecture (does it have a
|
||||
plugin/extension point where the gaps could go?), code quality, and the license.
|
||||
4. **Delete the clone when done.** Nothing from evaluation stays on disk or in a project. If the
|
||||
user later chooses to fork it, that is a fresh, deliberate clone or fork in the right place.
|
||||
|
||||
Clone only the few candidates that are genuinely in contention, not every search hit.
|
||||
|
||||
---
|
||||
|
||||
## 5. Pick Likely Libraries
|
||||
|
||||
Whether the plan ends up as a fork or a fresh start, identify the libraries the work would lean
|
||||
on, for the language that fits best (the existing candidate's language if forking, otherwise the
|
||||
best fit for the constraints and the user's preferences). For each major subsystem — protocol,
|
||||
file format, UI toolkit, storage, networking, parsing — name one or two maintained candidates
|
||||
and why. The per-language rules under
|
||||
[`code-languages/`](skills-public/code-languages/) apply to whatever is chosen.
|
||||
|
||||
---
|
||||
|
||||
## 6. Present Options — the User Chooses
|
||||
|
||||
Report back **before** scaffolding or writing code, with:
|
||||
|
||||
1. **The requirements checklist** (§2), so the user can correct it.
|
||||
2. **What was searched** — the queries and sources, briefly, so an empty result is credible.
|
||||
3. **A comparison of the real candidates**:
|
||||
|
||||
| Candidate | Covers | Missing / partial | License | Health | User opinion | Fit |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
|
||||
4. **Options**, typically some of:
|
||||
- **Use it as is** — it already does the job; maybe configure or script around it.
|
||||
- **Extend it** — via its plugin/extension API, with no fork.
|
||||
- **Fork and modify** — close, but the gaps need changes to its core.
|
||||
- **Contribute upstream** — the gaps are useful to everyone; send them as PRs (then the
|
||||
project's own rules win, per the precedence table in
|
||||
[`coding-dev.md`](skills-public/coding-dev.md)).
|
||||
- **Combine** — several existing tools or libraries glued together.
|
||||
- **Start fresh** — on top of the libraries from §5, with a short, explicit reason why none of
|
||||
the candidates were a usable base.
|
||||
5. **A recommendation**, with the reason — but the decision is the user's. Do not quietly start
|
||||
on the recommended option. When the user has said to work autonomously and decide without
|
||||
them, take the recommendation and record the choice and the alternatives where they will see
|
||||
it.
|
||||
|
||||
If nothing usable exists, say so plainly, list what was checked, and still mention near misses
|
||||
worth borrowing ideas, formats, or code from (license permitting).
|
||||
|
||||
**Keep hosting out of the proposal.** The options are about *what software* to run, not how to
|
||||
deploy it. Containers, proxies, tunnels, extra helper services, and the database layout belong to
|
||||
the environment's own infrastructure skill, which gets read once an option is chosen and before
|
||||
anything is deployed. Note a hard requirement that affects the choice (it needs Postgres, it
|
||||
needs a GPU) and stop there.
|
||||
|
||||
---
|
||||
|
||||
## 7. Checklist
|
||||
|
||||
- [ ] Requirements turned into must-haves / nice-to-haves / constraints
|
||||
- [ ] At least three to five varied web searches first, covering the whole idea and its parts
|
||||
- [ ] User reviews, comparisons, and complaints gathered for each serious candidate
|
||||
- [ ] Training-data recall merged in afterwards, each item checked against current sources
|
||||
- [ ] Promising candidates shallow-cloned to a temp directory, read without running, mapped
|
||||
against the checklist, then deleted
|
||||
- [ ] Libraries identified for the likely language
|
||||
- [ ] Options and a recommendation presented; nothing built before the user chose (or before
|
||||
an autonomous decision was recorded)
|
||||
Reference in new issue
Block a user