From 5e6abadb95cdee4533fa31c324e3c001422e343d Mon Sep 17 00:00:00 2001 From: Bluscream Date: Fri, 30 Oct 2026 13:00:44 +0100 Subject: [PATCH] docs: add software-vision and android-backup-restore skills with android scripts Co-Authored-By: Claude Opus 5.5 --- README.md | 2 + android-backup-restore.md | 562 +++++++++++++++++++++++++ coding-dev.md | 12 +- scripts/android/fix-installers.py | 49 +++ scripts/android/keystore-transplant.py | 74 ++++ scripts/android/launch.sh | 8 + scripts/android/measure.sh | 42 ++ scripts/android/nova-to-lawnchair.py | 88 ++++ scripts/android/reinstall-keep-data.sh | 40 ++ scripts/android/restore.py | 144 +++++++ scripts/android/test-all-apps.sh | 40 ++ software-vision.md | 201 +++++++++ 12 files changed, 1261 insertions(+), 1 deletion(-) create mode 100644 android-backup-restore.md create mode 100755 scripts/android/fix-installers.py create mode 100755 scripts/android/keystore-transplant.py create mode 100755 scripts/android/launch.sh create mode 100755 scripts/android/measure.sh create mode 100755 scripts/android/nova-to-lawnchair.py create mode 100755 scripts/android/reinstall-keep-data.sh create mode 100755 scripts/android/restore.py create mode 100755 scripts/android/test-all-apps.sh create mode 100644 software-vision.md diff --git a/README.md b/README.md index c370509..e4ea68f 100644 --- a/README.md +++ b/README.md @@ -9,8 +9,10 @@ A collection of public, reusable agent skills, cheatsheets, and operational work | Skill File | Scope / Topics Covered | | :--- | :--- | | **[`coding-dev.md`](skills-public/coding-dev.md)** | Standing rules for all code work: maximum linter/compiler strictness, never filtering build output, reuse-before-reinvent dependency ordering, file/function size limits, test & regression discipline, verification gates, scope control, commit hygiene, and honest reporting. | +| **[`software-vision.md`](skills-public/software-vision.md)** | Starting from a software idea: turn it into a requirements checklist, search the web first (projects, forks, skeletons, libraries, user reviews), then merge in training-data recall checked against current sources, shallow-clone promising candidates to a temp dir to check them, then present use / extend / fork / start-fresh options before building anything. | | **[`cli-tooling.md`](skills-public/cli-tooling.md)** | Using command-line tools correctly: reading the help tree instead of guessing flags, preferring an existing CLI over an ad-hoc script, non-interactive invocation, machine-readable output formats, orienting on prior work before starting, script placement & symlinking, and care around destructive commands. | | **[`afk-sleep-work.md`](skills-public/afk-sleep-work.md)** | Operating unattended and autonomously: continuous forward momentum, avoiding polling traps and deadlocks, host preservation and resource budgeting (disk, packages, processes), isolated tool provisioning, avoiding "last breath" traps on token/quota exhaustion, bypassing or driving GUI launchers, and wake-up checkpoints. | +| **[`android-backup-restore.md`](skills-public/android-backup-restore.md)** | Extracting/backing up/restoring Android data by access tier before a wipe or migration: EDL/firehose & bootloader, custom recovery (TWRP/OrangeFox/CWM), root, device-owner, Shizuku, plain ADB, and on-device apps; FBE decryption limits, per-partition dumps (userdata, EFS/modem/IMEI, persist), SELinux/UID-correct restores, the death of `adb backup`, and app-specific data (WhatsApp, Signal, TOTP/Keystore) that does not survive. | ### šŸ”¤ Per-Language Rules ([`code-languages/`](skills-public/code-languages/)) diff --git a/android-backup-restore.md b/android-backup-restore.md new file mode 100644 index 0000000..a6d9cf1 --- /dev/null +++ b/android-backup-restore.md @@ -0,0 +1,562 @@ +--- +name: android-backup-restore +description: >- + Use this skill when extracting, backing up, migrating, or restoring data from an Android + device ā€” especially before a wipe, ROM flash, factory reset, or device-to-device move, and + when deciding what is salvageable at a given access level. Covers the full ladder of access + tiers (EDL/firehose & bootloader, custom recovery CWM/TWRP/OrangeFox, root, device-owner, + Shizuku, plain ADB, and on-device apps), file-based-encryption decryption limits, per-partition + dumps (userdata, EFS/modem/IMEI, persist), SELinux/UID-correct restores, the death of + `adb backup`, and app-specific data (WhatsApp, Signal, TOTP/Keystore) that does not survive. +--- + +# Android Backup & Restore ā€” Extraction by Access Level + +How to pull as much as possible off an Android device before wiping, flashing, or migrating ā€” +and how to put it back. Organised as a **ladder of access tiers**: the more privileged the +access, the more you can extract. Work out which tier you have, pull from the highest tier +available, and fall back down the ladder for whatever that tier cannot reach. + +This is a *data-recovery and migration* skill for devices you own or are authorised to service. +Shell orchestration here follows [`bash.md`](skills-public/code-languages/bash.md) (strict mode, +hard `timeout`s, no wait-loops); CLI invocation follows +[`cli-tooling.md`](skills-public/cli-tooling.md) (read ` --help` before trusting any flag +below ā€” tool surfaces drift, and the exact partition/block paths are **per-device**). + +> [!IMPORTANT] +> **The single biggest variable is encryption.** Every modern Android device uses File-Based +> Encryption (FBE); the `/data` partition is a ciphertext blob until something that holds the +> user credential derives the keys. That "something" is normally the booted OS. A custom +> recovery can only read plaintext `/data` if it was *built to decrypt your exact Android +> version* and you type the lockscreen PIN/password into it. See Ā§3. This is why "boot it +> normally and pull over ADB/root" beats "boot to recovery and dd" whenever the device still +> boots. + +--- + +## 1. The Access Ladder (decision table) + +Pick the highest row you can reach. Each tier can extract everything below it plus its own column. + +| Tier | Access you need | Reaches raw partitions? | Reaches decrypted `/data/data` app data? | Reaches EFS/IMEI/persist? | Notes | +| :--- | :--- | :--- | :--- | :--- | :--- | +| **EDL / firehose / BROM** (Ā§8) | Vendor edl/mtk loader, device in 9008/BROM | āœ… full flash, offline | āš ļø only as ciphertext (still FBE-locked) | āœ… | Last resort for dead/bricked devices; dumps are encrypted blobs unless you also have the keys. | +| **Bootloader / fastboot** (Ā§8) | Unlocked bootloader | āš ļø some (`fastboot fetch` on newer; mostly flash-only) | āŒ | partial | Mostly a *flashing* interface, not a dumping one. | +| **Custom recovery** TWRP/OrangeFox/CWM (Ā§3) | Unlocked bootloader + matching recovery | āœ… every partition via `dd`/built-in backup | āš ļø **only if recovery decrypts your FBE** + you enter the PIN | āœ… | Classic "nandroid". Decryption is the catch on Android 10+. | +| **Root (booted OS)** (Ā§4) | Magisk/KernelSU/APatch on a booting device | āœ… (block devices readable) | āœ… **fully decrypted** (OS is running) | āœ… | The best tier for a *working* device ā€” plaintext data + raw partitions together. | +| **Device Owner / DPC** (Ā§5) | `dpm set-device-owner` on a fresh/unprovisioned device | āŒ | āŒ (own managed scope only) | āŒ | Niche; grants/denies permissions, reads some providers; not a bulk dumper. | +| **Shizuku / ADB-shell privilege** (Ā§6) | Shizuku (wireless-debug or one-time ADB) | āŒ | āš ļø APKs, `Android/data`, `Android/obb`, media ā€” **not** `/data/data` of release apps | āŒ | `adb shell` UID (2000) reach, on-device, no PC tether. | +| **Plain ADB** (Ā§6) | USB debugging authorised | āŒ | āš ļø same as Shizuku + `run-as` for *debuggable* apps only | āŒ | `adb backup` is effectively dead (Ā§7). Pull storage, providers, bugreport. | +| **On-device app, no root** (Ā§6.4) | Just the unlocked phone | āŒ | āŒ (own data only) | āŒ | Vendor cloud/D2D, Seedvault, per-app export. Last-mile for the non-technical case. | + +**Rule of thumb for a pre-wipe sweep:** if it boots and you can unlock it ā†’ get **root** (Ā§4) if +possible, else **ADB** (Ā§6). If it does *not* boot ā†’ **custom recovery** (Ā§3), else **EDL** (Ā§8). +Always grab **EFS/IMEI/persist** (Ā§4.3) before any flash regardless of tier. + +--- + +## 2. First: inventory what is actually worth saving + +Before dumping 128 GB of userdata, enumerate. Most of the value is small and specific. + +- **Media & documents** ā€” `/sdcard` (a.k.a. `/storage/emulated/0`): DCIM, Pictures, Download, + Documents, Music, ringtones. Reachable at *every* tier including plain ADB. Pull this first; it + is the easy win and the bulk of what users actually miss. +- **App data** ā€” `/data/data/` and `/data/user/0/` (credential-encrypted, "CE") and + `/data/user_de/0/` (device-encrypted, "DE"). Needs root or recovery-decrypt. This is where + game saves, app logins, notes, 2FA seeds live. +- **Scoped external app data** ā€” `Android/data/` and `Android/obb/` on `/sdcard`. + Blocked from ordinary file managers since Android 11/13 but reachable via ADB/Shizuku/root. +- **Telephony/PIM** ā€” SMS/MMS, call log, contacts, calendar. See Ā§6.2 (providers) and Ā§9. +- **Identity partitions** ā€” EFS/modem (`modemst1`, `modemst2`, `fsg`), `persist`, `efs`. IMEI, + Wi-Fi/BT MAC, sensor calibration. Ā§4.3. **Irreplaceable ā€” back up before any flash.** +- **The things that will NOT survive** no matter the tier ā€” hardware-backed Keystore keys, and + by extension many 2FA/banking apps. Ā§10. Plan migrations for these *before* wiping. + +--- + +## 3. Custom Recovery (TWRP / OrangeFox / CWM) ā€” the offline nandroid + +Use when the device will not boot normally, or you want a block-level image independent of the OS. +Requires an **unlocked bootloader** (itself a wipe on most devices ā€” so unlock *before* you have +data worth keeping, or accept the loss). Boot without flashing when you can: +`fastboot boot twrp.img` leaves the installed recovery untouched. + +> [!TIP] +> **Try the *installed* recovery before downloading or temp-booting one.** `adb reboot recovery`, +> then `adb shell id` ā€” many custom ROMs ship a current OrangeFox/TWRP whose `adbd` already runs as +> **root** and, crucially, is new enough to **decrypt FBE on the installed Android version** (so +> `/data/data` shows real package names and readable file contents). This beats `fastboot boot`-ing +> an old recovery image: a recovery built from a much older branch often won't boot on a new ROM at +> all (its kernel panics and the device falls back to the OS ā€” you'll land back in Android as +> `uid=2000(shell)`, the tell-tale that it didn't take). Confirm decryption with +> `ls /data/data` (real names, not hex) and by reading one `*/shared_prefs/*.xml`. + +### 3.1 The decryption problem (read this before trusting a backup) + +TWRP/OrangeFox can only back up **plaintext** `/data` if its decryption code matches your device's +FBE implementation *and* you enter the lockscreen credential in recovery (Keymaster derives the +keys from it). Reality on Android 10+: + +- Use a recovery **built from the branch matching your Android version** (e.g. an `android-12.1`+ + build for Android 12+). A recovery built from an old branch shows `/data` as garbage folder names + or fails to mount. +- Devices that **shipped on Android ā‰¤11 and were upgraded** to 12+ often cannot be decrypted by any + TWRP build ā€” the keystore/Keymaster path changed. OrangeFox sometimes succeeds where TWRP fails; + try both. If neither decrypts, you can still image the partition but only as ciphertext (useless + without the keys). +- If decryption fails, **do not factory-reset to "fix" it** ā€” that destroys the keys and the data. + +Verify decryption worked before relying on it: in the TWRP terminal, `ls -la /data/data` should +show real package names (`com.whatsapp`, ā€¦), not hex blobs. + +### 3.2 Built-in backup vs. raw `dd` + +- **Built-in backup** (TWRP "Backup" / OrangeFox): tick System, Data, Boot, EFS, etc. Produces a + nandroid set under `/sdcard/TWRP/BACKUPS/`. If `/data` decrypted, the Data archive holds plaintext + app data; if not, it backs up the raw (encrypted) partition. Pull the whole folder off afterwards. +- **Raw partition image** when you want a specific partition or the built-in backup won't cover it. + From the recovery `adb shell` or terminal ā€” **block paths are per-device**, find them first: + + ```bash + # from recovery shell ā€” enumerate by name, DO NOT guess the numbers + ls -l /dev/block/by-name/ # symlinks: userdata, boot_a, modemst1, efs, persist, ā€¦ + cat /proc/partitions + + # image a named partition to portable storage (not back onto /data) + timeout 1800 dd if=/dev/block/by-name/userdata of=/sdcard/userdata.img bs=8M + ``` + + Pull with `adb pull /sdcard/userdata.img` (or write to an SD card / OTG drive). A raw `userdata` + image is only restorable/mountable if you also have the FBE keys ā€” prefer the decrypted built-in + Data backup for app data, and reserve raw images for small partitions (Ā§4.3). + +### 3.3 Restore + +Restore the built-in nandroid through the recovery UI, or `dd` a raw image back **only onto the +exact same partition of the same device** (never another device's EFS ā€” Ā§4.3). After a decrypted +Data restore across Android versions, expect some apps to misbehave; a clean reinstall + selective +data restore (Ā§4.2) is often more reliable than a whole-partition nandroid across versions. + +--- + +## 4. Root on a booting device ā€” the best tier for a live phone + +If the device boots and you can unlock it, root (Magisk / KernelSU / APatch) is the strongest +position: the OS has already derived the FBE keys, so **all app data is plaintext**, and you still +have raw block-device access. (For installing root on this workstation's own test devices, Magisk +APKs/modules may already be on the NAS ā€” see +[check-nas-before-downloading](check-nas-before-downloading.md) before fetching.) + +### 4.1 Bulk app-data pull + +```bash +# as root, on device or via `adb shell su -c` +# tar preserves owners/modes/symlinks ā€” plain `cp`/`adb pull` loses them (see Ā§4.4) +su -c 'tar -cpf /sdcard/appdata-whatsapp.tar \ + /data/data/com.whatsapp /data/user_de/0/com.whatsapp' +# then from the PC: +adb pull /sdcard/appdata-whatsapp.tar +``` + +Pull `/data/data/` (CE) **and** `/data/user_de/0/` (DE) for each app ā€” some apps split +state across both. For a whole-device grab, `tar` `/data/data`, `/data/user/0`, `/data/user_de/0` +and `/data/misc` (wifi configs, etc.), but mind disk budget on the pulling host. + +> [!TIP] +> **Stream straight off the device with `adb exec-out` ā€” no on-device staging.** When `/data` has +> little free space (you can't write a multi-GB tar back onto the partition you're reading), pipe +> the archive directly to the host, and `dd` images the same way (`exec-out` is binary-safe; plain +> `adb shell` mangles bytes): +> ```bash +> adb exec-out 'cd /data && tar -cp -f - data user_de system misc vendor adb' > data-full.tar +> adb exec-out 'dd if=/dev/block/by-name/boot status=none' > boot.img +> ``` +> Toybox/busybox `tar` can't store SELinux contexts and exits non-zero when it skips sockets +> (`unknown file type '140000'` ā€” harmless). Capture contexts separately as a restore sidecar: +> `for d in /data/data/* /data/user_de/0/*; do stat -c '%a %U %G %C %n' "$d"; done`. The per-app +> SELinux *categories* (`:c123,c456`) encode the app sandbox and are **not** regenerable by +> `restorecon` alone ā€” this sidecar is how you put them back. +> +> **Record restore metadata while you have the device.** A `dumpsys package` capture gives, per +> app: `codePath` (ā†’ system/vendor/product/`/data/app`=user), `pkgFlags` (SYSTEM/UPDATED_SYSTEM_APP), +> `installerPackageName` (Play `com.android.vending` vs sideloaded `null`), versionCode, and granted +> runtime permissions; `cmd appops dump` adds special ops (SYSTEM_ALERT_WINDOW, +> MANAGE_EXTERNAL_STORAGE, ā€¦). Parse these into a manifest so you can later restore *only* user apps +> and re-grant their exact permissions. + +Prefer dedicated tools over hand-rolled tar when restoring to a *different* device or Android +version, because they record ownership/SELinux/permissions that a naive tar restore drops (Ā§4.4): + +- **Swift Backup** (root; Shizuku for a weaker subset) ā€” app + data, permissions, SSAID. +- **Neo Backup** (FOSS, OAndBackupX fork; root incl. KernelSU/APatch) ā€” app + data, schedules. +- **OAndBackupX / Migrate** ā€” FOSS nandroid-style app+data sets. + +### 4.2 Raw partitions while rooted + +Same `dd` from `/dev/block/by-name/` as Ā§3.2, but from the running OS (`su -c dd ā€¦`). Good for +grabbing `boot`/`vendor_boot` to patch, or EFS/persist (Ā§4.3) without rebooting to recovery. + +### 4.3 EFS / modem / IMEI / persist ā€” back these up before ANY flash + +These hold IMEI, baseband calibration, Wi-Fi/BT MAC, and sensor calibration. Corrupting them bricks +the radio permanently and there is no download that restores *your* unique values. **Partition names +and block numbers are per-device ā€” look them up, never copy numbers from a guide.** + +```bash +su -c 'ls -l /dev/block/by-name/' | grep -Ei 'efs|modemst|fsg|fsc|persist|nvram|nvdata|protect' +# Qualcomm: commonly modemst1, modemst2, fsg, fsc, persist +# MediaTek: commonly nvram, nvdata, protect_f, protect_s, persist +for p in modemst1 modemst2 fsg persist; do + su -c "dd if=/dev/block/by-name/$p of=/sdcard/efs-$p.img bs=4M" +done +``` + +Pull the images **off the device and keep copies on a host/NAS** ā€” a backup that lives only on the +partition you are about to wipe is no backup. Restore the same way (`dd ā€¦ of=/dev/block/by-name/$p`) +**only to the identical partition on the identical device**. Never flash one phone's EFS onto +another ā€” you would overwrite its unique identity with useless values. + +### 4.4 Restoring raw `/data/data` tars correctly (UID + SELinux) + +A tar of `/data/data/` restored by hand will *not* launch unless you fix two things Android +assigns at install time: + +1. **Install the app first** (same package, ideally same version) so Android creates the data dir + with the correct per-app UID and SELinux context. +2. Restore the files, then **fix ownership and labels**: + + ```bash + # find the app's real UID from the freshly-created dir, don't assume 100xx + su -c 'ls -ld /data/data/com.example' # e.g. u0_a142 u0_a142 + su -c 'tar -xpf /sdcard/appdata-example.tar -C /' + su -c 'chown -R u0_a142:u0_a142 /data/data/com.example' + su -c 'restorecon -RFD /data/data/com.example' # argument order matters; -v to see work + ``` + +Then reboot before launching. Skipping the UID chown or `restorecon` is the usual reason a +"successful" restore produces an app that crashes on open. Dedicated backup apps (Ā§4.1) do this +bookkeeping for you ā€” hand-tar only when you must. + +### 4.5 Getting root headlessly (no GUI taps) when the device boots + +If the bootloader is unlocked and you need root to reach Ā§4 but can't tap through the Magisk app, +patch and flash boot entirely from the shell: + +1. **Get the *exact* stock boot.img.** The authoritative copy is the live partition ā€” dump it from + a root recovery (Ā§3) with `dd`/`exec-out`. Don't flash a boot.img of unknown origin: check its + cmdline first (`androidboot.hardware=qcom` = Qualcomm vs MediaTek's `bootopt=ā€¦` ā€” a wrong-SoC + image bricks the device). +2. **Patch it** with Magisk's own tooling. The Magisk APK is a zip: its `assets/boot_patch.sh` needs + the arm64 libs renamed (`lib/arm64-v8a/libmagiskboot.so`ā†’`magiskboot`, `libmagiskinit.so`ā†’ + `magiskinit`, `libmagisk.so`ā†’`magisk`, `libinit-ld.so`ā†’`init-ld`) plus `stub.apk` and + `util_functions.sh` in the same dir, then `sh boot_patch.sh boot.img` ā†’ `new-boot.img`. + - Set the flags **explicitly** (standalone `boot_patch.sh` treats unset as false, which can + bootloop): for a modern FBE device with a vbmeta partition and dynamic partitions use + `KEEPVERITY=true KEEPFORCEENCRYPT=true PATCHVBMETAFLAG=false RECOVERYMODE=false LEGACYSAR=false`. + `Failed to patch` on the kernel DTB step is benign (nothing to patch). + - The recovery installer (`META-INF/com/google/android/update-binary`, which auto-detects these + flags via `get_flags()`) is cleaner **but** fails outside a real sideload: it tries to mount + `/system`, which on super/dynamic-partition ROMs is `/dev/mapper/system` and won't mount in a + plain `adb shell`. `boot_patch.sh` with explicit flags sidesteps that. +3. **Flash** `new-boot.img` (verify it unpacks: `magiskboot unpack new-boot.img`) with + `fastboot flash boot` or, from a root recovery, `dd ā€¦ of=/dev/block/by-name/boot`. Keep the stock + image for instant rollback. Reboot: `magiskd` in `ps -A` + `magisk -V` confirms root took and + survives reboot. +4. **Grant su to the ADB shell** (Magisk denies it by default, silently): Magisk app ā†’ Superuser + access = "Apps and ADB" + Automatic response = "Grant", or seed a `policies` row + (`uid=2000 policy=2`) in `/data/adb/magisk.db`. Until then `adb shell su -c ā€¦` returns + `Permission denied` even though root is installed and working. + +--- + +## 5. Device Owner (DPC) ā€” narrow, not a bulk extractor + +A device-owner app (set with `adb shell dpm set-device-owner /` on a device with +**no accounts added yet**) gets elevated *policy* powers, not filesystem powers. It is relevant to +extraction only in that it can silently grant its own runtime permissions and read some content +providers that would otherwise prompt ā€” useful for a managed enrolment flow, not for cleaning out a +phone before a wipe. Hard limits: + +- `setPermissionGrantState()` for **sensor** permissions (location, camera, mic, body sensors, + activity recognition) is restricted/deprecated for *profile* owners on Android 12+ (silently + returns false); device owners retain more but the surface is shrinking each release. +- It cannot read other apps' `/data/data`, cannot image partitions, and cannot reach EFS. + +If you find yourself reaching for device-owner to extract data, the real answer is almost always +root (Ā§4) or ADB providers (Ā§6.2) instead. + +--- + +## 6. ADB & Shizuku ā€” no root, device boots + +Plain ADB runs as the `shell` user (UID 2000). Shizuku grants that **same** privilege to on-device +apps (started via wireless debugging or a one-time ADB command), so their reach is identical to ADB +ā€” handy when you have no PC. + +### 6.1 External storage & scoped app dirs + +```bash +adb pull /sdcard/ ./sdcard-backup/ # DCIM, Download, Documents, etc. ā€” works everywhere +# Android/data and Android/obb are blocked from on-device file managers (11/13+) but +# the adb shell UID can still read them: +adb pull /sdcard/Android/data/ ./android-data/ +adb pull /sdcard/Android/obb/ ./android-obb/ +``` + +### 6.2 Content providers (telephony/PIM without root) + +`adb shell content query` reaches providers the `shell` UID is allowed to read. What actually works +on a stock device is uneven: + +```bash +adb shell content query --uri content://contacts/phones # contacts ā€” usually readable +adb shell content query --uri content://com.android.calendar/events +# SMS/MMS and call_log generally FAIL for the shell UID (need READ_SMS / be default app): +adb shell content query --uri content://sms # typically permission denied +``` + +For SMS/MMS, call logs, and contacts as restorable files, use an on-device FOSS app rather than +fighting providers over ADB: + +- **SMS Import / Export** (`com.github.tmo1.sms_ie`, F-Droid) ā€” SMS+MMS+call log+contacts to + JSON/ZIP, no root, and it *restores* too (set it as default SMS app during restore). +- **Open Android Backup** (mrrfv) ā€” PC-driven, rootless, uses a companion app (not `adb backup`): + restores apps-without-data, internal storage, and contacts (vCard); exports SMS (CSV) and call + log (txt) as **view-only** (no restore, no MMS attachments). +- Contacts also export cleanly from the stock Contacts app ("Export to .vcf") or via Google account + sync ā€” often the least-effort path for the non-technical case. + +### 6.3 `run-as` ā€” app data for *debuggable* apps only + +```bash +adb shell run-as com.example.debug tar -cpf - ./ > example-data.tar +``` + +`run-as` only works for apps with `android:debuggable="true"` (i.e. your own dev builds, not Play +Store releases). Shizuku-based file managers use the same `run-as` path and inherit the same limit. +Release-app `/data/data` is **not** reachable at this tier ā€” that needs root (Ā§4) or a +recovery-decrypt (Ā§3). + +### 6.4 Bugreport & app lists (cheap, always worth grabbing) + +```bash +adb bugreport ./bugreport.zip # logs, settings dump, installed-package metadata +adb shell pm list packages -3 -f # third-party packages + APK paths, to reinstall later +adb shell cmd package list packages --show-versioncode +``` + +To re-pull the APKs themselves for reinstall: + +```bash +adb shell pm path com.example # -> package:/data/app/.../base.apk + splits +adb pull /data/app/~~.../com.example-.../base.apk # repeat for each split; reinstall with +adb install-multiple base.apk split_config.*.apk # install-multiple +``` + +### 6.5 Vendor D2D and cloud (no-tech last mile) + +- **Google** cloud backup + Pixel/Android cable "device-to-device" restore at setup. App data rides + along *only* if the Google **account** is transferred (Block Store is account-bound); untick + nothing under "Accounts". `allowBackup="false"` stops cloud backup but D2D still transfers app data + unless the app sets `dataExtractionRules`/`android:fullBackupOnly` to opt out. +- **Samsung Smart Switch** (PC or cable): backs up most app data, but **encrypted app data (e.g. + WhatsApp), banking logins, and sideloaded APKs do not transfer** ā€” handle those per Ā§9/Ā§10. +- **Seedvault** (CalyxOS/GrapheneOS/LineageOS): the FOSS backup transport. On Android 15 builds its + **device-to-device** mode (Expert settings) is default-on and captures far more app data than the + old cloud-style transport, because it presents as a D2D migration to each app. + +--- + +## 7. The death of `adb backup` (don't build a plan on it) + +`adb backup` was deprecated in Android 10 and gutted in Android 12 (API 31): app data is excluded by +default and only included if the app is **debuggable**, so on essentially all release apps across +Android 12ā€“16 `adb backup` produces an archive with no app data. Treat it as unavailable: + +- Don't rely on it for a migration. Use root tooling (Ā§4.1), on-device FOSS exporters (Ā§6.2), or + vendor/Seedvault D2D (Ā§6.5) instead. +- `adb restore` likewise cannot restore encrypted backups and is a dead end for modern data. +- The `bmgr`/LocalTransport backup-manager plumbing still exists for *developers testing their own + backup agents* (`adb shell bmgr ā€¦`), but it is a per-app developer tool, not a device-wide + extractor ā€” out of scope for a pre-wipe sweep. + +--- + +## 8. EDL / firehose / BROM ā€” dead or bootloader-locked devices + +The lowest level, for devices that will not boot to Android *or* recovery. These talk to the SoC's +emergency loader below the OS: + +- **Qualcomm EDL (9008)** ā€” [`bkerler/edl`](https://github.com/bkerler/edl) with a firehose + programmer (`.mbn`/`.elf`) matching the SoC. `edl printgpt` lists partitions; `edl rl dumps + --skip=userdata --genxml` dumps all partitions (minus the huge userdata) to a folder with a + `rawprogram` XML; `edl r ` grabs one. Newer SoCs need `--loader=` + because loader autodetection no longer works. +- **MediaTek BROM** ā€” [`bkerler/mtkclient`](https://github.com/bkerler/mtkclient). `./mtk printgpt`, + `./mtk rl out` dumps all partitions to `out/`, `./mtk r ` for one; `--preloader` + for BROM mode. + +The hard limit: **these dumps are ciphertext.** EDL reads the raw flash below the OS, so `userdata` +comes out FBE-encrypted and is not decryptable without the keys (which live in secure hardware keyed +by the lockscreen credential). EDL is therefore excellent for **small identity partitions** +(EFS/persist/nvram ā€” Ā§4.3), full-flash forensic images, and un-bricking, but it is **not** a way to +read a locked phone's app data. Treat downloaded dumps/tools as untrusted input: keep each in its own +directory, pass paths as arguments, and run any helper scripts from outside the dump directory. + +--- + +## 9. App-specific data that needs its own handling + +Some high-value data lives behind app-level encryption and ignores the tier ladder: + +- **WhatsApp** ā€” local backup at `/sdcard/WhatsApp/Databases/msgstore.db.crypt15` (+ `/Media`). If + the user enabled **end-to-end encrypted backups**, decryption needs the **64-digit key** (or the + `encrypted_backup.key` file from `/data/data/com.whatsapp/files/`, root-only). Tools: + [`wa-crypt-tools`](https://pypi.org/project/wa-crypt-tools/). Easiest migration is WhatsApp's own + built-in transfer or Google-Drive backup with the same phone number ā€” pull the `crypt15` + media + as insurance, not as the primary path. +- **Signal** ā€” does **not** sync to any cloud. Migrate with Signal's in-app device-transfer, or its + encrypted local backup (Settings ā†’ Chats ā†’ Backups) whose **30-digit passphrase you must record** ā€” + there is no recovery without it. +- **Authenticator / 2FA** ā€” Google Authenticator exports via its QR "export accounts" flow; + **Aegis** (FOSS) exports an encrypted JSON vault. Do this *before* wiping; TOTP seeds not exported + are gone. + +--- + +## 10. What will NOT survive a wipe (plan around it) + +No tier recovers these; the only mitigation is to migrate them *before* the wipe: + +- **Hardware-backed Keystore keys** are non-exportable by design ā€” key material never leaves the + TEE/StrongBox. Any app that pinned a device-bound key (banking apps, some password managers, + FIDO/passkeys stored in hardware) will treat the restored/new device as untrusted and re-enrol. +- **Play Integrity / SafetyNet attestation** fails on rooted or custom-ROM devices independently of + your data ā€” so a perfectly-restored banking app may still refuse to run. That is an attestation + problem, not a backup problem; don't chase it as data loss. +- **SSAID / `ANDROID_ID`** changes on wipe (per app-signing-key + user), which can log apps out or + reset entitlements even when their `/data/data` is restored byte-perfect. + +Before wiping: export 2FA (Ā§9), confirm password-manager vaults sync to their cloud, record Signal +(Ā§9) and WhatsApp keys, and re-enrol passkeys/banking on the new device rather than expecting a +restore to carry them. + +--- + +## 11. Pre-wipe checklist (the one-screen version) + +1. **Decide the tier** (Ā§1). Booting + unlockable ā†’ root (Ā§4) or ADB (Ā§6). Not booting ā†’ recovery + (Ā§3) ā†’ EDL (Ā§8). +2. **EFS/modem/persist first** (Ā§4.3) ā€” before any unlock or flash, off-device. +3. **Media & docs** ā€” `adb pull /sdcard` (Ā§6.1). +4. **App data** ā€” root tar + dedicated tool (Ā§4.1), or recovery-decrypted Data backup (Ā§3). +5. **Scoped dirs** ā€” `Android/data`, `Android/obb` (Ā§6.1). +6. **PIM** ā€” SMS/call-log/contacts via SMS Import-Export or vCard (Ā§6.2). +7. **App-specific** ā€” WhatsApp/Signal/2FA via their own flows (Ā§9). +8. **App list + APKs** ā€” `pm list packages -3`, pull splits (Ā§6.4) to rebuild the app set. +9. **System state the app tars miss** (Ā§12.1) ā€” keystore DB, SSAIDs, URI grants, doze whitelist, + app-ops, launcher/shortcuts/QS/controls, Wi-Fi, SD-card UUIDs, root grants, old `packages.list`. +10. **Baseline + app-sweep** the old ROM (`measure.sh`, `test-all-apps.sh`, Ā§12.4) so the new one + has something to be compared against. +11. **Passwords apps keep themselves** ā€” App Manager's KeyStore password, backup-app passphrases, + Signal/WhatsApp keys (Ā§9). Write them down *before* the wipe; none are recoverable after. +12. **Verify the backups off-device** (open the archive, check `ls /data/data` really decrypted) + *before* you type `fastboot -w` or confirm the reset. A backup you have not read back is not a + backup. + +--- + +## 12. Restoring onto a new ROM (root) ā€” what a data tar does not carry + +Learned from a full LineageOS-based A16 ā†’ Infinity-X A17 switch (Redmi Note 7, Magisk). Restoring +`/data/data` + `user_de` + `Android/data` per Ā§4.4 brings back most apps, but **every app gets a new +UID** on the new ROM, and a lot of state is keyed by UID or lives in system files. The scripts in +[`scripts/android/`](skills-public/scripts/android/) automate the parts below; they take the +device from `ANDROID_SERIAL` like `adb` itself. + +### 12.1 Grab these too, before the wipe (all root, all small) + +| What | Where | Why it matters on restore | +| :--- | :--- | :--- | +| Old UID map | `/data/system/packages.list` | Remap everything keyed by UID (SSAID, keystore, grants) | +| Keystore2 DB | `/data/misc/keystore/persistent.sqlite` | Apps whose keys vanish show "Keystore error"/log out (Ā§12.3) | +| SSAIDs | `/data/system/users/0/settings_ssaid.xml` | `ANDROID_ID` change logs apps out (GitHub, Bolt, Lovense, ā€¦) | +| Persistent SAF grants | `/data/system/urigrants.xml` | Folder pickers (music players, camera, sync apps) lose their folders | +| Doze whitelist | `/data/system/deviceidle.xml` | Notification/automation apps silently stop in background | +| App-ops & special access | `cmd appops dump`, `cmd notification` policy access, `dumpsys package` | `MANAGE_EXTERNAL_STORAGE`, usage stats, DND, `READ_LOGS`, install-packages, notification listeners are **not** runtime permissions | +| Launcher + pinned shortcuts | launcher DB, `/data/system_ce/0/shortcut_service/` | Home screen; pinned shortcuts (e.g. Ice Box) are owned by the system, not the launcher | +| QS tiles, device controls | `settings get secure sysui_qs_tiles`; SystemUI `user_de` `files/controls_favorites.xml` + `shared_prefs/controls_prefs.xml` | Home Assistant controls, dev tiles | +| Wi-Fi | `/data/misc/apex/com.android.wifi/` or `cmd wifi list-networks` + PSKs | Also note which SSIDs must **not** auto-join | +| Root grants + Magisk settings | `magisk --sqlite "SELECT * FROM policies"`, `ā€¦ FROM settings`, `ā€¦ FROM denylist`, `/data/adb/modules` (note which are disabled) | UIDs change, grants don't follow; Zygisk off ā‡’ DenyList silently does nothing | +| Accounts | `dumpsys account`; `/data/system_ce/0/accounts_ce.db` | Shows which logins (2nd Google account, app-registered accounts) must be re-added by hand ā€” they can't be transplanted | +| SD-card volume IDs | `blkid` of each SD partition | Public volume ID = filesystem UUID; reformat ā‡’ new ID ā‡’ broken paths and grants | +| Freezer state | `pm list packages -d` + the freezer app's own list | Updates/reinstalls unfreeze | + +### 12.2 Restore order and traps + +- **Update first, restore data second.** Installing the old APKs and then the store's update is + fine; restoring data *into* an older APK than the data was written by is a downgrade ā€” apps wipe + or crash on it. Never re-run a full restore pass over updated apps; restore data only. +- **Installer of record:** `pm set-installer` refuses even root. Reinstall the same APKs with + `adb install-multiple -r -i ` (keeps data) so Play/Aurora/F-Droid/Obtainium see their + apps as updatable (`fix-installers.py`). +- **New signature, same package** (forks such as Shizuku ā†’ Shizuku-Next for a newer Android): tar the + data dirs, uninstall, install, untar, chown/chcon, re-grant, re-freeze ā€” `reinstall-keep-data.sh`. +- **System files are read only at `system_server` start.** Write `urigrants.xml`, + `settings_ssaid.xml`, `shortcut_service/*` etc. (owner `system:system`, mode 600, + `u:object_r:system_data_file:s0`), then `kill -9 $(pidof system_server)` ā€” a soft restart that + does **not** save over them first, which a normal reboot does. It locks the user: unlock with + wake ā†’ swipe ā†’ `input text ` ā†’ ENTER (the bouncer screenshots black; retry if needed). +- **fs-verity:** files under `shortcut_service` are verity-protected ā€” `rm` then write, never edit. +- **Restored caches can kill an app:** Ice Box OOM-crashed on a restored MMKV package cache. Drop + `cache`, `code_cache`, WebView caches and per-app package-list caches; keep the real state. +- **Re-map UID-keyed state** with the old/new `packages.list` (SSAID `name=""`, keystore + `namespace`, Magisk `policies.uid`, doze lists are by package and need no remap). +- **SD card:** if the partition had to be recreated, set the old UUID back + (`tune2fs -U ` for ext4, `fsck` before and after) rather than fixing every app. +- **Wi-Fi:** restored networks auto-join in their own order; re-add the ones that must stay + manual with `cmd wifi add-network wpa2 -d`. +- **QS tiles** on A17: `settings put secure sysui_qs_tiles` is ignored; use + `cmd statusbar set-tiles ""`. Developer tiles must be enabled first + (`pm enable 'com.android.settings/.development.qstile.DevelopmentTiles$SensorsOff'`). +- **Launcher switch** (Nova had no build for the new Android): converting Nova's `favorites` + table into Lawnchair's Launcher3 schema works (`nova-to-lawnchair.py`) ā€” mark widgets + `restored=1` so Launcher3 binds fresh widget IDs, and re-pin deep shortcuts for the new launcher. + +**Banking-app root checks** are not a restore problem, but they show up right after one. Order +that worked (2026-10, Magisk 31, A17): first the ROM's own spoofing settings, if it has any (PIF, +TrickyStore target and keybox, hide developer status). If an app still crashes on purpose, look for +an Xposed module that defuses that app's check. For PayPal 10.18 that was oslokrok on Vector, the +LSPosed successor. Configure Vector headlessly with +`/data/adb/modules/zygisk_vector/cli modules enable ` and `cli scope set /0`. +**Take the app off the Magisk DenyList**: the DenyList strips every Zygisk module from the process, +including the hook. + +### 12.3 Keystore: transplanting app keys + +Hardware-backed keys never leave the TEE, but keystore2 stores them **wrapped** in its SQLite DB. +If the new ROM runs on the same TEE (compare `keyentry.km_uuid` old vs new ā€” same device, same +firmware ā‡’ same), the old wrapped blobs still unwrap. Keys whose `blobmetadata` has tag 0 +(`EncryptedBy`) are additionally super-encrypted with the **old lock-screen credential's** key and +cannot be moved. Domain 0 (APP) rows have `namespace = uid` ā†’ remap. Stop the apps, back up the live +DB, insert `keyentry`/`blobentry`/`blobmetadata`/`keyparameter`/`keymetadata` with fresh ids, reboot. +Apps that already regenerated keys on the new ROM also need their data re-restored to match. +`scripts/android/keystore-transplant.py OLD_DB [--replace] OLD_UID:NEW_UID ā€¦` writes the SQL; dry-run +it on a host copy of the live DB (`pragma integrity_check`) before applying with the device's own +`sqlite3`. Tested: transplanted keys load with no keystore errors after a reboot. That does not +bring back sessions the server ended, or apps that refuse to run because their build is too old. + +### 12.4 Verify like a user would + +- `test-all-apps.sh OUT` cold-starts every user app (unfreezing and re-freezing), screenshots it, + and greps its log for crashes, ANRs and keystore/crypto errors ā†’ `results.tsv`. Look at every + screenshot: logouts, licence checks, ToS/"what's new" dialogs and permission prompts don't crash. +- Android 17 shell launches must match the activity's intent filter: + `am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n 'pkg/act'` + (single-quote it ā€” component names contain `$`). `-n` alone gives "Access blocked". +- `measure.sh OUT [pkgā€¦]` snapshots memory, crashes/ANRs/lmk kills since boot and cold-launch + times; run it on the old ROM before the wipe so there is a baseline. +- Keep a running log of every security setting weakened during the restore (adb stay-awake, root + grants, doze exemptions, ā€¦) with its revert command; the restore needs many of them and nobody + remembers them afterwards. diff --git a/coding-dev.md b/coding-dev.md index 2d44d79..2f47864 100644 --- a/coding-dev.md +++ b/coding-dev.md @@ -26,6 +26,14 @@ reasons attached, not style trivia ā€” most exist because ignoring them caused a > When in doubt about which mode you are in, **ask** ā€” it changes the answer on strictness, > file size, typing, and test policy all at once. +> [!IMPORTANT] +> **A new software idea starts with research, not code.** When the user describes a tool, app, +> service, or other project they want built, follow +> [`software-vision.md`](skills-public/software-vision.md) first: find existing projects to use, +> extend, or fork (web searches and user reviews first, then training-data recall, then +> throwaway clones), and present the options before scaffolding anything ā€” unless the user +> explicitly opted out of research. + Per-language specifics live in [`code-languages/`](skills-public/code-languages/): | Language | Rules | @@ -115,7 +123,9 @@ a fresh implementation will rediscover slowly and incompletely. Before hand-rolling any subsystem ā€” a protocol client, a parser, a date formatter, a debounce, a retry/backoff, a semver comparator ā€” **search first and state the finding.** -Prefer official/first-party SDKs over third-party wrappers. +Prefer official/first-party SDKs over third-party wrappers. The same ordering applies to a +whole project idea, not just a subsystem ā€” see +[`software-vision.md`](skills-public/software-vision.md). ### The corollary: new generic code becomes its own package diff --git a/scripts/android/fix-installers.py b/scripts/android/fix-installers.py new file mode 100755 index 0000000..b7257a0 --- /dev/null +++ b/scripts/android/fix-installers.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""Reinstall each restored app with its original installer (pm set-installer refuses root). + +Same APKs, `install-multiple -r -i ` keeps data. The device comes from ANDROID_SERIAL, as for adb. Usage: fix-installers.py BACKUP_DIR WORK_DIR +""" + +import re +import subprocess +import sys +import time +from pathlib import Path + +backup, work = Path(sys.argv[1]), Path(sys.argv[2]) +log_file = (work / "restore.log").open("a") + + +def log(msg: str) -> None: + line = f"{time.strftime('%H:%M:%S')} {msg}" + print(line, flush=True) + log_file.write(line + "\n") + log_file.flush() + + +def adb(*args: str, timeout: int = 900) -> str: + r = subprocess.run(["adb", *args], capture_output=True, text=True, timeout=timeout) + return (r.stdout + r.stderr).strip() + + +def installers(text: str) -> dict: + # `pm list packages -i` prints "package:PKG", with -f "package:PATH=PKG". + return dict(re.findall(r"[:=]([\w.]+)\s+installer=(\S+)", text)) + + +want = installers((backup / "rooted-fresh/packages-3.txt").read_text()) +now = installers(adb("shell", "pm list packages -3 -i")) +disabled = set(re.findall(r"package:([\w.]+)", adb("shell", "pm list packages -d"))) +log("=== installer fix-up start") +for pkg, inst in sorted(want.items()): + if inst == "null" or pkg not in now or now[pkg] == inst or inst not in now | {"com.android.vending": 1}: + continue + apks = sorted(str(p) for p in (backup / "apks" / pkg).glob("*.apk")) + if not apks: + continue + # Reinstalling re-enables a frozen app; disable it again afterwards. + out = adb("install-multiple", "-r", "-d", "-i", inst, "--bypass-low-target-sdk-block", *apks) + if pkg in disabled: + adb("shell", f"su -c 'pm disable-user --user 0 {pkg}'") + log(f"{pkg}: installer {now[pkg]} -> {inst}: {'OK' if 'Success' in out else 'FAILED ' + out[-200:]}") +log("=== installer fix-up done") diff --git a/scripts/android/keystore-transplant.py b/scripts/android/keystore-transplant.py new file mode 100755 index 0000000..1bba666 --- /dev/null +++ b/scripts/android/keystore-transplant.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Emit SQL that copies apps' keystore2 keys from an old ROM's DB into the live one, under the new UIDs. + +Only works when both ROMs use the same TEE (compare keyentry.km_uuid). Keys super-encrypted with the +old lock-screen credential (blobmetadata tag 0, EncryptedBy) cannot be moved and are skipped. + +Usage: keystore-transplant.py OLD_DB [--replace] OLD_UID:NEW_UID [...] > transplant.sql + --replace first delete the app's keys that already exist on the new ROM (needed when it regenerated them) +Apply with the apps force-stopped: `su -c 'sqlite3 /data/misc/keystore/persistent.sqlite < transplant.sql'`, reboot. +""" + +import secrets +import sqlite3 +import sys + +DOMAIN_APP = 0 +TAG_ENCRYPTED_BY = 0 + + +def lit(v: object) -> str: + if v is None: + return "NULL" + if isinstance(v, bytes): + return f"X'{v.hex()}'" + if isinstance(v, str): + return "'" + v.replace("'", "''") + "'" + return str(v) + + +def delete_keys(where: str) -> list[str]: + ids = f"(SELECT id FROM keyentry WHERE {where})" + return [f"DELETE FROM blobmetadata WHERE blobentryid IN (SELECT id FROM blobentry WHERE keyentryid IN {ids});", + f"DELETE FROM blobentry WHERE keyentryid IN {ids};", + f"DELETE FROM keyparameter WHERE keyentryid IN {ids};", + f"DELETE FROM keymetadata WHERE keyentryid IN {ids};", + f"DELETE FROM grant WHERE keyentryid IN {ids};", + f"DELETE FROM keyentry WHERE {where};"] + + +args = sys.argv[1:] +old = sqlite3.connect(f"file:{args.pop(0)}?mode=ro", uri=True) +replace = "--replace" in args +pairs = [tuple(map(int, a.split(":"))) for a in args if a != "--replace"] +# Older keystore2 schemas have no blobentry.state; every blob there is current (0). +blob_state = "state" if any(c[1] == "state" for c in old.execute("PRAGMA table_info(blobentry)")) else "0" + +out = ["BEGIN;"] +for old_ns, new_ns in pairs: + if replace: + out += delete_keys(f"domain={DOMAIN_APP} AND namespace={new_ns}") + keys = old.execute("SELECT id, key_type, alias, state, km_uuid FROM keyentry WHERE domain=? AND namespace=?", + (DOMAIN_APP, old_ns)).fetchall() + for kid, key_type, alias, state, km_uuid in keys: + blobs = old.execute(f"SELECT id, subcomponent_type, blob, {blob_state} FROM blobentry WHERE keyentryid=?", (kid,)).fetchall() + if any(old.execute("SELECT 1 FROM blobmetadata WHERE blobentryid=? AND tag=?", (b[0], TAG_ENCRYPTED_BY)).fetchone() + for b in blobs): + print(f"-- skip {old_ns} {alias!r}: super-encrypted with the old lock-screen credential", file=sys.stderr) + continue + new_id = secrets.randbits(63) * (1 if secrets.randbits(1) else -1) + # The old key wins over one the app generated under the same alias on the new ROM. + out += delete_keys(f"domain={DOMAIN_APP} AND namespace={new_ns} AND alias={lit(alias)}") + out.append(f"INSERT INTO keyentry (id,key_type,domain,namespace,alias,state,km_uuid) VALUES " + f"({new_id},{key_type},{DOMAIN_APP},{new_ns},{lit(alias)},{state},{lit(km_uuid)});") + for bid, sub, blob, bstate in blobs: + out.append(f"INSERT INTO blobentry (subcomponent_type,keyentryid,blob,state) VALUES ({sub},{new_id},{lit(blob)},{bstate});") + for tag, data in old.execute("SELECT tag, data FROM blobmetadata WHERE blobentryid=?", (bid,)): + out.append(f"INSERT INTO blobmetadata (blobentryid,tag,data) VALUES ((SELECT max(id) FROM blobentry WHERE keyentryid={new_id}),{tag},{lit(data)});") + for tag, data, level in old.execute("SELECT tag, data, security_level FROM keyparameter WHERE keyentryid=?", (kid,)): + out.append(f"INSERT INTO keyparameter (keyentryid,tag,data,security_level) VALUES ({new_id},{tag},{lit(data)},{level});") + for tag, data in old.execute("SELECT tag, data FROM keymetadata WHERE keyentryid=?", (kid,)): + out.append(f"INSERT INTO keymetadata (keyentryid,tag,data) VALUES ({new_id},{tag},{lit(data)});") + print(f"-- {old_ns}->{new_ns} {alias!r}: {len(blobs)} blobs", file=sys.stderr) +out.append("COMMIT;") +print("\n".join(out)) diff --git a/scripts/android/launch.sh b/scripts/android/launch.sh new file mode 100755 index 0000000..0440153 --- /dev/null +++ b/scripts/android/launch.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +# Cold-start an app's launcher activity the way Android 17 accepts from the shell. Usage: launch.sh PKG [WAIT] [SHOT.png] +p=$1 w=${2:-6} shot=${3:-} +act=$(adb shell "cmd package resolve-activity --brief -c android.intent.category.LAUNCHER $p" /dev/null +sleep "$w" +[ -n "$shot" ] && adb exec-out screencap -p > "$shot" /dev/null"' +echo "procs: $(a 'ls -d /proc/[0-9]* | wc -l')" +echo "## top cpu" +a 'top -b -n 1 -m 12 -o PID,USER,%CPU,%MEM,NAME' | tail -13 +echo "## dropbox since boot" +boot=$(a 'date -d @$(( $(date +%s) - $(cut -d. -f1 /proc/uptime) )) "+%Y-%m-%d %H:%M:%S"') +a "dumpsys dropbox" | awk -v b="$boot" '$1" "$2 >= b && $3 ~ /anr|crash|watchdog|tombstone|wtf|lowmem|native/ {print $3}' | sort | uniq -c +echo "## crash buffer (FATAL / ANR per process)" +a 'logcat -d -b crash' | grep -oE "Process: [^,]+|FATAL EXCEPTION|>>> [^ ]+ <<<" | sort | uniq -c | sort -rn | head -20 +a 'logcat -d -b events' | grep -oE "am_anr: \[[^]]*" | awk -F, '{print "ANR " $3}' | sort | uniq -c +echo "## lmkd / am_kill" +a 'logcat -d -b events' | grep -cE "am_kill|lmk_kill" | sed 's/^/kills: /' +a 'logcat -d -b events' | grep -oE "am_kill: \[[^]]*" | awk -F, '{print $5}' | sort | uniq -c | sort -rn | head -10 +echo "## logcat error tags (top 25)" +a 'logcat -d *:E' | awk '{print $6}' | sort | uniq -c | sort -rn | head -25 +echo "## thermal / battery" +a 'dumpsys battery | grep -E "level|temperature|status"; for z in /sys/class/thermal/thermal_zone*; do echo "$(cat $z/type) $(cat $z/temp)"; done 2>/dev/null | sort -k2 -n | tail -5' +if [ $# -gt 0 ]; then + echo "## cold launches (ms, am start -W -S, 3 runs)" + for p in "$@"; do + t="" + for _ in 1 2 3; do + act=$(a "cmd package resolve-activity --brief -c android.intent.category.LAUNCHER $p" | tr -d '\r' | tail -1) + v=$(a "am start -W -S -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n '$act'" | awk '/TotalTime/{print $2}') + t="$t ${v:-fail}"; a 'input keyevent KEYCODE_HOME'; sleep 2 + done + echo "$p:$t" + done +fi +} 2>&1 | tee "$out" diff --git a/scripts/android/nova-to-lawnchair.py b/scripts/android/nova-to-lawnchair.py new file mode 100755 index 0000000..5b0239a --- /dev/null +++ b/scripts/android/nova-to-lawnchair.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +"""Rebuild the Nova home screen in Lawnchair, plus the Ice Box shortcuts Nova had pinned. + +Usage: nova-to-lawnchair.py NOVA_DB LAWNCHAIR_DB OLD_ICEBOX_SHORTCUT_XML OUT_DIR + +Writes OUT_DIR/launcher_5_5_5.db (Lawnchair schema, Nova's items), OUT_DIR/icebox.xml +(shortcut_service package file with only the pinned shortcuts) and OUT_DIR/lawnchair-pins.xml +(shortcut_service launcher file pinning them for Lawnchair). +""" + +import re +import shutil +import sqlite3 +import sys +from pathlib import Path + +LAUNCHER = "app.lawnchair.nightly" +ICEBOX = "com.catchingnow.icebox" +# From /data/system/packages.xml on the new ROM: versionCode, lastUpdateTime, SHA-256 of the cert. +LAUNCHER_INFO = ("1600020100", "1792700106881", "dHw2RbNXJYsuI+hR5TyWdH/grdAH5bos2X6MhVcuTcU=") +ICEBOX_INFO = ("154821395", "1792699567277", "gvW+cC4DIBdGDKg3b22r1yWp5S3SSSDdXl17WF+/4bY=") +# Nova-only containers (gesture/dock-swipe slots); Lawnchair has nothing to put them in. +SKIP_CONTAINERS = {-207, -208} +ITEM_APP, ITEM_FOLDER, ITEM_WIDGET, ITEM_DEEP_SHORTCUT = 0, 2, 4, 6 +FLAG_ID_NOT_VALID = 1 # Launcher3 restore flag: allocate and bind a fresh widget id on load +MISSING_PACKAGES = {"com.dot.gallery"} # not installed on the new ROM +# Google's smartspace widget on row 0 of the first page: Lawnchair reserves that row for its own. +SKIP_IDS = {56} + +nova_db, lc_db, icebox_xml, out = map(Path, sys.argv[1:5]) +out.mkdir(parents=True, exist_ok=True) + + +def package_info(version: str, updated: str, sig: str) -> str: + return (f'\n' + f'\n\n') + + +nova = sqlite3.connect(nova_db) +nova.row_factory = sqlite3.Row +rows = [r for r in nova.execute("select * from favorites where _id >= 0 order by _id") + if r["container"] not in SKIP_CONTAINERS and r["_id"] not in SKIP_IDS] +folders = {r["_id"] for r in rows if r["itemType"] == ITEM_FOLDER} +kept = [] +for r in rows: + pkg = re.search(r"component=([\w.]+)/", r["intent"] or "") + if pkg and pkg.group(1) in MISSING_PACKAGES: + print(f"skip {r['title']}: {pkg.group(1)} not installed") + continue + if r["container"] >= 0 and r["container"] not in folders: + continue + kept.append(r) + +db = out / "launcher_5_5_5.db" +shutil.copy(lc_db, db) +lc = sqlite3.connect(db) +lc.execute("delete from favorites") +pins = [] +for r in kept: + widget = r["itemType"] == ITEM_WIDGET + hotseat = r["container"] == -101 + in_folder = r["container"] >= 0 + x, y = int(max(r["cellX"] or 0, 0)), int(max(r["cellY"] or 0, 0)) + lc.execute( + "insert into favorites (_id,title,intent,container,screen,cellX,cellY,spanX,spanY,itemType," + "appWidgetId,appWidgetProvider,restored,profileId,rank,options) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", + (r["_id"], r["title"], r["intent"], r["container"], + x if hotseat else (0 if in_folder else r["screen"]), + x, 0 if hotseat else y, int(r["spanX"]), int(r["spanY"]), r["itemType"], + r["appWidgetId"] if widget else -1, r["appWidgetProvider"], + FLAG_ID_NOT_VALID if widget else 0, 0, r["rank"], 0)) + if r["itemType"] == ITEM_DEEP_SHORTCUT: + pins.append(re.search(r"S\.shortcut_id=([^;]+)", r["intent"]).group(1)) +lc.commit() +print(f"{len(kept)} items written, pins: {pins}") + +old = Path(icebox_xml).read_text() +blocks = [m.group(0) for m in re.finditer(r'\n', old, re.S) if m.group(1) in pins] +assert len(blocks) == len(pins), f"found {len(blocks)} of {len(pins)} shortcuts" +head = re.search(r"]*>\n", old).group(0) +(out / "icebox.xml").write_text("\n" + head + + package_info(*ICEBOX_INFO) + "".join(blocks) + "\n") +(out / "lawnchair-pins.xml").write_text( + "\n" + f'\n' + package_info(*LAUNCHER_INFO) + + f'\n' + + "".join(f'\n' for p in pins) + "\n\n") diff --git a/scripts/android/reinstall-keep-data.sh b/scripts/android/reinstall-keep-data.sh new file mode 100755 index 0000000..37c0fea --- /dev/null +++ b/scripts/android/reinstall-keep-data.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# Replace an app with a differently signed build while keeping its data, permissions, +# installer and frozen state. Data is also saved to BACKUP_DIR before the uninstall. +# Usage: reinstall-keep-data.sh PACKAGE BACKUP_DIR APK [APK ...] +set -euo pipefail +pkg=$1 bak=$2; shift 2 +a() { adb shell "$@" /dev/null'" > "$bak/$pkg.tar" +echo "saved $(stat -c %s "$bak/$pkg.tar") bytes to $bak/$pkg.tar" +[ "$(stat -c %s "$bak/$pkg.tar")" -gt 10240 ] || { echo "backup suspiciously small, aborting"; exit 1; } + +a "pm uninstall $pkg" +iopt=(); [ -n "$installer" ] && [ "$installer" != null ] && iopt=(-i "$installer") +adb install-multiple -r "${iopt[@]}" --bypass-low-target-sdk-block "$@" + +uid=$(su_ "stat -c %u /data/data/$pkg") +cache_gid=$((uid - 10000 + 20000)) +adb exec-in "su -c 'tar -xf - -C /'" < "$bak/$pkg.tar" +su_ "while pgrep -x tar >/dev/null; do sleep 0.3; done" +for d in /data/data/$pkg /data/user_de/0/$pkg; do + su_ "chown -R $uid:$uid $d; for c in cache code_cache; do mkdir -p $d/\$c; chown $uid:$cache_gid $d/\$c; chmod 2771 $d/\$c; done; chcon -hR \"\$(stat -c %C $d)\" $d" +done +case " $dirs " in *Android/data*) + su_ "d=/data/media/0/Android/data/$pkg; chown -R $uid:1078 \$d; find \$d -type d -exec chmod 2770 {} +; find \$d -type f -exec chmod 660 {} +; chcon -R u:object_r:media_rw_data_file:s0 \$d" ;; +esac +for p in $perms; do a "pm grant $pkg $p" 2>/dev/null || true; done +[ "$disabled" = 1 ] && su_ "pm disable-user --user 0 $pkg" +echo "$pkg: reinstalled as uid $uid, $(a "dumpsys package $pkg" | grep -m1 versionName | xargs)" diff --git a/scripts/android/restore.py b/scripts/android/restore.py new file mode 100755 index 0000000..a3c6e20 --- /dev/null +++ b/scripts/android/restore.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +"""Restore user apps, app data and permissions onto a freshly flashed, rooted phone. + +Sources (from the backup folder): + apks//*.apk APK + splits + rooted-fresh/dumpsys-package.txt installer, granted runtime permissions + rooted-fresh/packages-3.txt which packages are user apps + rooted-fresh/packages-disabled.txt frozen/disabled state + rooted/apps-manifest.json special app-ops (overlay, all-files, ...) + /data/, /user_de/0/ unpacked data-fresh.tar + +The device comes from ANDROID_SERIAL, as for adb. Usage: restore.py BACKUP_DIR WORK_DIR [--data-only] [pkg ...] (no pkgs = all user apps) + --data-only only put the data back (apps already installed and updated): keeps the current APK, + installer, permissions and frozen state. Never re-run a full restore over updated apps. +Every action and its result is appended to WORK_DIR/restore.log. +""" + +import json +import re +import shlex +import subprocess +import sys +import time +from pathlib import Path + +SKIP = {"com.topjohnwu.magisk"} # installed fresh, its data must not be reused +# Dropped on restore: regenerated by the app, and the bulk of most backups. +VOLATILE = ["cache", "code_cache", "app_webview/Default/Cache", "app_webview/Default/Code Cache"] + +backup, work = Path(sys.argv[1]), Path(sys.argv[2]) +data_only = "--data-only" in sys.argv[3:] +only = set(sys.argv[3:]) - {"--data-only"} +log_file = (work / "restore.log").open("a") + + +def log(msg: str) -> None: + line = f"{time.strftime('%H:%M:%S')} {msg}" + print(line, flush=True) + log_file.write(line + "\n") + log_file.flush() + + +def adb(*args: str, timeout: int = 300, stdin=None) -> subprocess.CompletedProcess: + return subprocess.run(["adb", *args], capture_output=True, text=stdin is None, + timeout=timeout, stdin=stdin) + + +def su(cmd: str, timeout: int = 300) -> str: + r = adb("shell", f"su -c {shlex.quote(cmd)}", timeout=timeout) + return (r.stdout + r.stderr).strip() + + +def parse_dumpsys(text: str) -> dict: + """installer + granted runtime permissions per package.""" + out = {} + for block in re.split(r"\n Package \[", text)[1:]: + name = block.split("]", 1)[0] + installer = re.search(r"\n installerPackageName=(\S+)", block) + runtime = block.split("runtime permissions:", 1) + perms = re.findall(r"\n ([\w.]+): granted=true", runtime[1]) if len(runtime) > 1 else [] + out[name] = {"installer": installer.group(1) if installer else None, "perms": sorted(set(perms))} + return out + + +fresh = backup / "rooted-fresh" +meta = parse_dumpsys((fresh / "dumpsys-package.txt").read_text(errors="replace")) +manifest = json.loads((backup / "rooted/apps-manifest.json").read_text())["apps"] +user_pkgs = sorted(set(re.findall(r"=([\w.]+)\s+installer=", (fresh / "packages-3.txt").read_text())) - SKIP) +disabled = set(re.findall(r"package:([\w.]+)", (fresh / "packages-disabled.txt").read_text())) +targets = [p for p in user_pkgs if not only or p in only] +log(f"=== restore start: {len(targets)} packages") + +# Pass 1: install every APK set, so installers like Obtainium exist before pass 2 names them. +for pkg in [] if data_only else targets: + apks = sorted(str(p) for p in (backup / "apks" / pkg).glob("*.apk")) + if not apks: + log(f"{pkg}: NO APK, skipped") + continue + # --bypass-low-target-sdk-block: Android 14+ refuses targetSdk < 24 (Automagic workarounds). + r = subprocess.run(["adb", "install-multiple", "-r", "-d", "-g", + "--bypass-low-target-sdk-block", *apks], + capture_output=True, text=True, timeout=900) + log(f"{pkg}: install {'OK' if 'Success' in r.stdout else 'FAILED ' + (r.stdout + r.stderr).strip()[-200:]}") + +installed = set(re.findall(r"package:([\w.]+)", adb("shell", "pm list packages").stdout)) + +# Pass 2: data, ownership, SELinux labels, installer, permissions, disabled state. +for pkg in targets: + if pkg not in installed: + continue + info = meta.get(pkg, {}) + su(f"am force-stop {pkg}") + uid = su(f"stat -c %u /data/data/{pkg}") + if not uid.isdigit(): + log(f"{pkg}: no data dir after install ({uid}), data skipped") + continue + cache_gid = int(uid) - 10000 + 20000 # u0_aNNN_cache + + members = [m for m in (f"data/{pkg}", f"user_de/0/{pkg}") if (work / m).is_dir()] + if members: + for m in members: + su(f"find /data/{m} -mindepth 1 -maxdepth 1 ! -name lib -exec rm -rf {{}} +") + excludes = [f"--exclude={m}/{v}" for m in members for v in VOLATILE] + tar = subprocess.Popen(["tar", "-cf", "-", "-C", str(work), *excludes, *members], + stdout=subprocess.PIPE) + r = adb("exec-in", "su -c 'tar -xf - -C /data'", stdin=tar.stdout, timeout=1800) + tar.wait() + # exec-in returns once our side is sent; the phone's tar may still be writing. + su("while pgrep -x tar >/dev/null; do sleep 0.3; done", timeout=600) + for m in members: + su(f"chown -R {uid}:{uid} /data/{m}; " + f"for d in cache code_cache; do mkdir -p /data/{m}/$d; chown {uid}:{cache_gid} /data/{m}/$d; " + # restorecon gives only app_data_file:s0; the per-app MLS categories that + # installd put on the top directory must be copied onto its contents. + f"chmod 2771 /data/{m}/$d; done; chcon -hR \"$(stat -c %C /data/{m})\" /data/{m}", + timeout=900) + wrong = su(f"find {' '.join('/data/' + m for m in members)} ! -user {uid} | wc -l") + log(f"{pkg}: data {'+'.join(members)} rc={r.returncode} uid={uid} wrong_owner={wrong}") + else: + log(f"{pkg}: no data in backup") + if data_only: + continue + + inst = info.get("installer") + if inst and inst != "null" and inst in installed | {"com.android.vending"}: + log(f"{pkg}: installer {inst} -> {su(f'pm set-installer {pkg} {inst}') or 'OK'}") + + # -g granted everything at install; revoke what was not granted before. + wanted = set(info.get("perms", [])) + now = re.findall(r"\n\s+([\w.]+): granted=true", su(f"dumpsys package {pkg} | sed -n '/runtime permissions:/,$p'")) + revoked = [p for p in set(now) - wanted] + for p in revoked: + su(f"pm revoke {pkg} {p}") + for p in wanted - set(now): + su(f"pm grant {pkg} {p}") + ops = manifest.get(pkg, {}).get("specialAppOps", {}) + for op, mode in ops.items(): + su(f"appops set {pkg} {op} {mode}") + log(f"{pkg}: perms granted={len(wanted)} revoked={len(revoked)} appops={','.join(ops) or '-'}") + + if pkg in disabled: + log(f"{pkg}: disable-user -> {su(f'pm disable-user --user 0 {pkg}')}") + +log("=== restore done") diff --git a/scripts/android/test-all-apps.sh b/scripts/android/test-all-apps.sh new file mode 100755 index 0000000..9213a3a --- /dev/null +++ b/scripts/android/test-all-apps.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# Launch every user app with a launcher activity, one by one: unfreeze if frozen, cold start, +# wait, screenshot, collect its crash/ANR/keystore log lines, check it is alive and in front, +# then stop it and re-freeze. Writes OUT_DIR/results.tsv, OUT_DIR/.png, OUT_DIR/.log. +# Usage: test-all-apps.sh OUT_DIR [pkg ...] +set -uo pipefail +out=$1; shift +mkdir -p "$out" +a() { adb shell "$@" "$out/results.tsv" + +for p in "${pkgs[@]}"; do + wf=0; grep -qx "$p" <<<"$frozen" && wf=1 + [ $wf = 1 ] && a "su -c 'pm enable --user 0 $p'" >/dev/null + act=$(a "cmd package resolve-activity --brief -c android.intent.category.LAUNCHER $p" | tr -d '\r' | tail -1) + if [[ "$act" != */* ]]; then + [ $wf = 1 ] && a "su -c 'pm disable-user --user 0 $p'" >/dev/null + printf '%s\t%s\t-\t-\t-\t-\t-\t-\tno launcher activity\n' "$p" "$wf" >> "$out/results.tsv"; continue + fi + a 'logcat -c' + ms=$(a "am start -W -S -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n '$act'" | awk '/TotalTime/{print $2}') + sleep "$WAIT" + adb exec-out screencap -p > "$out/$p.png" "$out/$p.log" + crash=$(grep -cE "FATAL EXCEPTION|Process: $p|Fatal signal" "$out/$p.log") + anr=$(grep -cE "ANR in $p" "$out/$p.log") + ks=$(grep -ciE "AEADBadTag|KeyStoreException|VERIFICATION_FAILED|KeyPermanentlyInvalidated|UnrecoverableKey|BadPaddingException|file is not a database|SQLiteDatabaseCorrupt" "$out/$p.log") + printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t\n' "$p" "$wf" "${ms:-?}" "${pid:+yes}" "$fg" "$crash" "$anr" "$ks" >> "$out/results.tsv" + echo "$p frozen=$wf ms=${ms:-?} alive=${pid:+yes} fg=$fg crash=$crash anr=$anr ks=$ks" + a "am force-stop $p; input keyevent KEYCODE_HOME" + [ $wf = 1 ] && a "su -c 'pm disable-user --user 0 $p'" >/dev/null +done diff --git a/software-vision.md b/software-vision.md new file mode 100644 index 0000000..ad48f74 --- /dev/null +++ b/software-vision.md @@ -0,0 +1,201 @@ +--- +name: software-vision +description: >- + Use this skill whenever the user describes an idea for a new piece of software, a tool, an app, + a service, a bot, a plugin, a mod, or an automation ā€” "I want something that ā€¦", "build me a ā€¦", + "could we make a tool for ā€¦", "I need a service that ā€¦" ā€” before choosing a language, scaffolding + a repository, or writing any code. Requires prior-art research first: web searches for existing + projects, forks, skeletons, libraries and what users say about them (reviews, complaints, + comparisons), only then merged with training-data recall as a secondary source, a + throwaway shallow clone of promising candidates to check them against the requirements, and a + set of options (fork, depend on, extend, start fresh) for the user to choose from. +--- + +# Software Vision ā€” Find What Exists Before Building + +When the user has an idea for a piece of software, the first job is **not** to design it. It is +to find out what already exists that does all or most of it. Building a full stack from nothing +when a project covering 80% of the requirements is one `git clone` away wastes days, and the +result is usually worse than the thing that already had years of bug reports behind it. + +This is the project-level form of the dependency ordering in +[`coding-dev.md` Ā§3](skills-public/coding-dev.md): use what exists, fork what is close, write +your own only when nothing fits. That section governs individual subsystems; this one governs +the whole idea. + +> [!IMPORTANT] +> **Never assume nothing exists.** However niche the idea sounds, someone may have built it, +> half-built it, abandoned it, drafted a skeleton, written a blog post about the protocol, or +> published research on the hard part. "I couldn't think of anything" is not a finding ā€” a list +> of the searches you ran and what they turned up is. + +--- + +## 1. When This Applies ā€” and the Only Opt-Out + +Run this whole process whenever the user describes software they want to exist: a new app, +CLI, service, bot, plugin, mod, integration, library, or automation ā€” whether it is a fresh +repository or a large new component inside an existing one. + +**The only way to skip it is an explicit opt-out from the user**, such as "no online research", +"don't look at other projects", or "I want to write this from scratch myself". Then: + +- Skip the web searches and the clones (Ā§3ā€“Ā§4). +- Still use the libraries rule in [`coding-dev.md` Ā§3](skills-public/coding-dev.md) for + individual subsystems unless the user excluded that too. +- Do not argue for research afterwards; the user has already decided. + +Urgency, a small-sounding scope, or "it's just a quick script" are **not** opt-outs. A few +searches take a minute; reinventing a maintained tool takes far longer. + +--- + +## 2. Pin Down the Requirements First + +Before searching, turn the user's description into a short checklist, because that list is what +every candidate gets measured against: + +- **Must-haves** ā€” what the software has to do to be useful at all. +- **Nice-to-haves** ā€” features the user mentioned that are not deal-breakers. +- **Constraints** ā€” platform (Linux, Windows, Wine, Android, a game engine), runtime, language + preferences, self-hosted vs cloud, license needs, offline use, performance. + +Show the checklist to the user with the results (Ā§6) so they can correct it. If a requirement +is genuinely ambiguous *and* would change which candidates qualify, ask ā€” otherwise state the +assumption and carry on. + +--- + +## 3. Search First, Then Merge in Recall + +The searches happen anyway, so they come **first**, and the decision rests on what they show. +Training data can be months or years out of date: projects get abandoned, renamed, relicensed, +or overtaken, and newer ones may never have been seen at all. Starting from recall anchors the +whole evaluation on a stale picture, so recall comes second and only fills gaps. + +### 3.1 Web searches first + +**Run at least three to five searches**, varied in phrasing and source, before settling on any +candidate. Cover: + +| Where | What it finds | +| :--- | :--- | +| General web search with the problem phrased several ways ("self-hosted X", "open source alternative to Y", "X for Linux") | Products, blog posts, comparisons | +| GitHub / GitLab / Codeberg search, topics, and `awesome-*` lists | Projects, forks, skeletons, abandoned prototypes | +| The package registry for the likely language (crates.io, npm, PyPI, NuGet, pkg.go.dev, ā€¦) | Libraries that do the hard part | +| Forums and communities (Reddit, Hacker News, the domain's own Discord/forum, issue trackers of adjacent projects) | "Does anything do X?" threads, half-finished attempts, known dead ends | +| Papers, specs, reverse-engineering notes for niche domains | The research behind the hard part, even if nobody shipped an app | + +Search for **parts** as well as the whole: an idea with no complete match often splits into +pieces that each have a mature library or tool. + +For each promising hit, note what can be checked without cloning: last commit or release date, +license, language, open issue count and how maintainers respond, stars or downloads as a weak +signal, and whether its README claims the must-haves. + +### 3.2 What people say about it + +A project's own README describes what it *claims* to do. What users say shows how it actually +holds up, so search for opinions on every serious candidate too: reviews, "X vs Y" comparisons, +Reddit and Hacker News threads, forum posts, and the complaints in its issue tracker. Look for: + +- Recurring pain points (crashes, data loss, missing features people keep requesting) +- Whether people are moving to it or away from it, and to what +- How the maintainers respond to bugs and pull requests +- Workarounds people rely on, which often point at the gaps a fork would need to fill + +Weight recent opinions over old ones, and many independent voices over a single loud one. Put +this in the comparison (Ā§6) next to the technical facts. + +### 3.3 Then merge in recall + +Only after searching, add what you already know from training data: projects, products, +libraries, and standards that the searches missed. Every recalled item must be checked with a +quick search before it goes into the comparison (does it still exist, is it maintained, has the +license changed?). Where recall and current sources disagree, **current sources win.** + +--- + +## 4. Clone the Promising Candidates ā€” Then Delete Them + +When a candidate looks close but it is unclear whether it ticks every must-have, **look inside +it** rather than guessing from the README. + +1. **Shallow-clone into a throwaway directory** ā€” the session's scratchpad or a fresh `mktemp -d`, + never inside a project tree: + ```bash + dir=$(mktemp -d) && git clone --depth 1 "$dir/" + ``` +2. **Read, don't run.** A clone is untrusted code. Inspect the README, directory layout, entry + points, dependency manifest, build files, tests, and open issues. Do not run its install + scripts, build, or test suite as part of evaluation ā€” that is a decision for after the user + picks it. +3. **Map it against the checklist from Ā§2**: which must-haves it covers, which are partial, which + are missing, and roughly how much work each gap is. Note the architecture (does it have a + plugin/extension point where the gaps could go?), code quality, and the license. +4. **Delete the clone when done.** Nothing from evaluation stays on disk or in a project. If the + user later chooses to fork it, that is a fresh, deliberate clone or fork in the right place. + +Clone only the few candidates that are genuinely in contention, not every search hit. + +--- + +## 5. Pick Likely Libraries + +Whether the plan ends up as a fork or a fresh start, identify the libraries the work would lean +on, for the language that fits best (the existing candidate's language if forking, otherwise the +best fit for the constraints and the user's preferences). For each major subsystem ā€” protocol, +file format, UI toolkit, storage, networking, parsing ā€” name one or two maintained candidates +and why. The per-language rules under +[`code-languages/`](skills-public/code-languages/) apply to whatever is chosen. + +--- + +## 6. Present Options ā€” the User Chooses + +Report back **before** scaffolding or writing code, with: + +1. **The requirements checklist** (Ā§2), so the user can correct it. +2. **What was searched** ā€” the queries and sources, briefly, so an empty result is credible. +3. **A comparison of the real candidates**: + + | Candidate | Covers | Missing / partial | License | Health | User opinion | Fit | + | :--- | :--- | :--- | :--- | :--- | :--- | :--- | + +4. **Options**, typically some of: + - **Use it as is** ā€” it already does the job; maybe configure or script around it. + - **Extend it** ā€” via its plugin/extension API, with no fork. + - **Fork and modify** ā€” close, but the gaps need changes to its core. + - **Contribute upstream** ā€” the gaps are useful to everyone; send them as PRs (then the + project's own rules win, per the precedence table in + [`coding-dev.md`](skills-public/coding-dev.md)). + - **Combine** ā€” several existing tools or libraries glued together. + - **Start fresh** ā€” on top of the libraries from Ā§5, with a short, explicit reason why none of + the candidates were a usable base. +5. **A recommendation**, with the reason ā€” but the decision is the user's. Do not quietly start + on the recommended option. When the user has said to work autonomously and decide without + them, take the recommendation and record the choice and the alternatives where they will see + it. + +If nothing usable exists, say so plainly, list what was checked, and still mention near misses +worth borrowing ideas, formats, or code from (license permitting). + +**Keep hosting out of the proposal.** The options are about *what software* to run, not how to +deploy it. Containers, proxies, tunnels, extra helper services, and the database layout belong to +the environment's own infrastructure skill, which gets read once an option is chosen and before +anything is deployed. Note a hard requirement that affects the choice (it needs Postgres, it +needs a GPU) and stop there. + +--- + +## 7. Checklist + +- [ ] Requirements turned into must-haves / nice-to-haves / constraints +- [ ] At least three to five varied web searches first, covering the whole idea and its parts +- [ ] User reviews, comparisons, and complaints gathered for each serious candidate +- [ ] Training-data recall merged in afterwards, each item checked against current sources +- [ ] Promising candidates shallow-cloned to a temp directory, read without running, mapped + against the checklist, then deleted +- [ ] Libraries identified for the likely language +- [ ] Options and a recommendation presented; nothing built before the user chose (or before + an autonomous decision was recorded)