0 Fork 0

Files

T

BluscreamandClaude Opus 5.5 5e6abadb95 docs: add software-vision and android-backup-restore skills with android scripts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-30 13:00:44 +01:00

35 KiB

Raw Permalink Blame History

name, description

name	description
android-backup-restore	Use this skill when extracting, backing up, migrating, or restoring data from an Android device — especially before a wipe, ROM flash, factory reset, or device-to-device move, and when deciding what is salvageable at a given access level. Covers the full ladder of access tiers (EDL/firehose & bootloader, custom recovery CWM/TWRP/OrangeFox, root, device-owner, Shizuku, plain ADB, and on-device apps), file-based-encryption decryption limits, per-partition dumps (userdata, EFS/modem/IMEI, persist), SELinux/UID-correct restores, the death of `adb backup`, and app-specific data (WhatsApp, Signal, TOTP/Keystore) that does not survive.

name

description

android-backup-restore

Use this skill when extracting, backing up, migrating, or restoring data from an Android device — especially before a wipe, ROM flash, factory reset, or device-to-device move, and when deciding what is salvageable at a given access level. Covers the full ladder of access tiers (EDL/firehose & bootloader, custom recovery CWM/TWRP/OrangeFox, root, device-owner, Shizuku, plain ADB, and on-device apps), file-based-encryption decryption limits, per-partition dumps (userdata, EFS/modem/IMEI, persist), SELinux/UID-correct restores, the death of `adb backup`, and app-specific data (WhatsApp, Signal, TOTP/Keystore) that does not survive.

Android Backup & Restore — Extraction by Access Level

How to pull as much as possible off an Android device before wiping, flashing, or migrating — and how to put it back. Organised as a ladder of access tiers: the more privileged the access, the more you can extract. Work out which tier you have, pull from the highest tier available, and fall back down the ladder for whatever that tier cannot reach.

This is a data-recovery and migration skill for devices you own or are authorised to service. Shell orchestration here follows bash.md (strict mode, hard timeouts, no wait-loops); CLI invocation follows cli-tooling.md (read <tool> --help before trusting any flag below — tool surfaces drift, and the exact partition/block paths are per-device).

Important

The single biggest variable is encryption. Every modern Android device uses File-Based Encryption (FBE); the /data partition is a ciphertext blob until something that holds the user credential derives the keys. That "something" is normally the booted OS. A custom recovery can only read plaintext /data if it was built to decrypt your exact Android version and you type the lockscreen PIN/password into it. See §3. This is why "boot it normally and pull over ADB/root" beats "boot to recovery and dd" whenever the device still boots.

1. The Access Ladder (decision table)

Pick the highest row you can reach. Each tier can extract everything below it plus its own column.

Tier	Access you need	Reaches raw partitions?	Reaches decrypted `/data/data` app data?	Reaches EFS/IMEI/persist?	Notes
EDL / firehose / BROM (§8)	Vendor edl/mtk loader, device in 9008/BROM	✅ full flash, offline	⚠️ only as ciphertext (still FBE-locked)	✅	Last resort for dead/bricked devices; dumps are encrypted blobs unless you also have the keys.
Bootloader / fastboot (§8)	Unlocked bootloader	⚠️ some (`fastboot fetch` on newer; mostly flash-only)	❌	partial	Mostly a flashing interface, not a dumping one.
Custom recovery TWRP/OrangeFox/CWM (§3)	Unlocked bootloader + matching recovery	✅ every partition via `dd`/built-in backup	⚠️ only if recovery decrypts your FBE + you enter the PIN	✅	Classic "nandroid". Decryption is the catch on Android 10+.
Root (booted OS) (§4)	Magisk/KernelSU/APatch on a booting device	✅ (block devices readable)	✅ fully decrypted (OS is running)	✅	The best tier for a working device — plaintext data + raw partitions together.
Device Owner / DPC (§5)	`dpm set-device-owner` on a fresh/unprovisioned device	❌	❌ (own managed scope only)	❌	Niche; grants/denies permissions, reads some providers; not a bulk dumper.
Shizuku / ADB-shell privilege (§6)	Shizuku (wireless-debug or one-time ADB)	❌	⚠️ APKs, `Android/data`, `Android/obb`, media — not `/data/data` of release apps	❌	`adb shell` UID (2000) reach, on-device, no PC tether.
Plain ADB (§6)	USB debugging authorised	❌	⚠️ same as Shizuku + `run-as` for debuggable apps only	❌	`adb backup` is effectively dead (§7). Pull storage, providers, bugreport.
On-device app, no root (§6.4)	Just the unlocked phone	❌	❌ (own data only)	❌	Vendor cloud/D2D, Seedvault, per-app export. Last-mile for the non-technical case.

Rule of thumb for a pre-wipe sweep: if it boots and you can unlock it → get root (§4) if possible, else ADB (§6). If it does not boot → custom recovery (§3), else EDL (§8). Always grab EFS/IMEI/persist (§4.3) before any flash regardless of tier.

2. First: inventory what is actually worth saving

Before dumping 128 GB of userdata, enumerate. Most of the value is small and specific.

Media & documents — /sdcard (a.k.a. /storage/emulated/0): DCIM, Pictures, Download, Documents, Music, ringtones. Reachable at every tier including plain ADB. Pull this first; it is the easy win and the bulk of what users actually miss.
App data — /data/data/<pkg> and /data/user/0/<pkg> (credential-encrypted, "CE") and /data/user_de/0/<pkg> (device-encrypted, "DE"). Needs root or recovery-decrypt. This is where game saves, app logins, notes, 2FA seeds live.
Scoped external app data — Android/data/<pkg> and Android/obb/<pkg> on /sdcard. Blocked from ordinary file managers since Android 11/13 but reachable via ADB/Shizuku/root.
Telephony/PIM — SMS/MMS, call log, contacts, calendar. See §6.2 (providers) and §9.
Identity partitions — EFS/modem (modemst1, modemst2, fsg), persist, efs. IMEI, Wi-Fi/BT MAC, sensor calibration. §4.3. Irreplaceable — back up before any flash.
The things that will NOT survive no matter the tier — hardware-backed Keystore keys, and by extension many 2FA/banking apps. §10. Plan migrations for these before wiping.

3. Custom Recovery (TWRP / OrangeFox / CWM) — the offline nandroid

Use when the device will not boot normally, or you want a block-level image independent of the OS. Requires an unlocked bootloader (itself a wipe on most devices — so unlock before you have data worth keeping, or accept the loss). Boot without flashing when you can: fastboot boot twrp.img leaves the installed recovery untouched.

Tip

Try the installed recovery before downloading or temp-booting one. adb reboot recovery, then adb shell id — many custom ROMs ship a current OrangeFox/TWRP whose adbd already runs as root and, crucially, is new enough to decrypt FBE on the installed Android version (so /data/data shows real package names and readable file contents). This beats fastboot boot-ing an old recovery image: a recovery built from a much older branch often won't boot on a new ROM at all (its kernel panics and the device falls back to the OS — you'll land back in Android as uid=2000(shell), the tell-tale that it didn't take). Confirm decryption with ls /data/data (real names, not hex) and by reading one */shared_prefs/*.xml.

3.1 The decryption problem (read this before trusting a backup)

TWRP/OrangeFox can only back up plaintext /data if its decryption code matches your device's FBE implementation and you enter the lockscreen credential in recovery (Keymaster derives the keys from it). Reality on Android 10+:

Use a recovery built from the branch matching your Android version (e.g. an android-12.1+ build for Android 12+). A recovery built from an old branch shows /data as garbage folder names or fails to mount.
Devices that shipped on Android ≤11 and were upgraded to 12+ often cannot be decrypted by any TWRP build — the keystore/Keymaster path changed. OrangeFox sometimes succeeds where TWRP fails; try both. If neither decrypts, you can still image the partition but only as ciphertext (useless without the keys).
If decryption fails, do not factory-reset to "fix" it — that destroys the keys and the data.

Verify decryption worked before relying on it: in the TWRP terminal, ls -la /data/data should show real package names (com.whatsapp, …), not hex blobs.

3.2 Built-in backup vs. raw `dd`

Built-in backup (TWRP "Backup" / OrangeFox): tick System, Data, Boot, EFS, etc. Produces a nandroid set under /sdcard/TWRP/BACKUPS/. If /data decrypted, the Data archive holds plaintext app data; if not, it backs up the raw (encrypted) partition. Pull the whole folder off afterwards.
Raw partition image when you want a specific partition or the built-in backup won't cover it. From the recovery adb shell or terminal — block paths are per-device, find them first:
```
# from recovery shell — enumerate by name, DO NOT guess the numbers
ls -l /dev/block/by-name/            # symlinks: userdata, boot_a, modemst1, efs, persist, …
cat /proc/partitions

# image a named partition to portable storage (not back onto /data)
timeout 1800 dd if=/dev/block/by-name/userdata of=/sdcard/userdata.img bs=8M
```
Pull with adb pull /sdcard/userdata.img (or write to an SD card / OTG drive). A raw userdata image is only restorable/mountable if you also have the FBE keys — prefer the decrypted built-in Data backup for app data, and reserve raw images for small partitions (§4.3).

3.3 Restore

Restore the built-in nandroid through the recovery UI, or dd a raw image back only onto the exact same partition of the same device (never another device's EFS — §4.3). After a decrypted Data restore across Android versions, expect some apps to misbehave; a clean reinstall + selective data restore (§4.2) is often more reliable than a whole-partition nandroid across versions.

4. Root on a booting device — the best tier for a live phone

If the device boots and you can unlock it, root (Magisk / KernelSU / APatch) is the strongest position: the OS has already derived the FBE keys, so all app data is plaintext, and you still have raw block-device access. (For installing root on this workstation's own test devices, Magisk APKs/modules may already be on the NAS — see check-nas-before-downloading before fetching.)

4.1 Bulk app-data pull

# as root, on device or via `adb shell su -c`
# tar preserves owners/modes/symlinks — plain `cp`/`adb pull` loses them (see §4.4)
su -c 'tar -cpf /sdcard/appdata-whatsapp.tar \
        /data/data/com.whatsapp /data/user_de/0/com.whatsapp'
# then from the PC:
adb pull /sdcard/appdata-whatsapp.tar

Pull /data/data/<pkg> (CE) and /data/user_de/0/<pkg> (DE) for each app — some apps split state across both. For a whole-device grab, tar /data/data, /data/user/0, /data/user_de/0 and /data/misc (wifi configs, etc.), but mind disk budget on the pulling host.

Tip

Stream straight off the device with adb exec-out — no on-device staging. When /data has little free space (you can't write a multi-GB tar back onto the partition you're reading), pipe the archive directly to the host, and dd images the same way (exec-out is binary-safe; plain adb shell mangles bytes):
adb exec-out 'cd /data && tar -cp -f - data user_de system misc vendor adb' > data-full.tar
adb exec-out 'dd if=/dev/block/by-name/boot status=none'                     > boot.img
Toybox/busybox tar can't store SELinux contexts and exits non-zero when it skips sockets (unknown file type '140000' — harmless). Capture contexts separately as a restore sidecar: for d in /data/data/* /data/user_de/0/*; do stat -c '%a %U %G %C %n' "$d"; done. The per-app SELinux categories (:c123,c456) encode the app sandbox and are not regenerable by restorecon alone — this sidecar is how you put them back.

Record restore metadata while you have the device. A dumpsys package capture gives, per app: codePath (→ system/vendor/product//data/app=user), pkgFlags (SYSTEM/UPDATED_SYSTEM_APP), installerPackageName (Play com.android.vending vs sideloaded null), versionCode, and granted runtime permissions; cmd appops dump adds special ops (SYSTEM_ALERT_WINDOW, MANAGE_EXTERNAL_STORAGE, …). Parse these into a manifest so you can later restore only user apps and re-grant their exact permissions.

Prefer dedicated tools over hand-rolled tar when restoring to a different device or Android version, because they record ownership/SELinux/permissions that a naive tar restore drops (§4.4):

Swift Backup (root; Shizuku for a weaker subset) — app + data, permissions, SSAID.
Neo Backup (FOSS, OAndBackupX fork; root incl. KernelSU/APatch) — app + data, schedules.
OAndBackupX / Migrate — FOSS nandroid-style app+data sets.

4.2 Raw partitions while rooted

Same dd from /dev/block/by-name/ as §3.2, but from the running OS (su -c dd …). Good for grabbing boot/vendor_boot to patch, or EFS/persist (§4.3) without rebooting to recovery.

4.3 EFS / modem / IMEI / persist — back these up before ANY flash

These hold IMEI, baseband calibration, Wi-Fi/BT MAC, and sensor calibration. Corrupting them bricks the radio permanently and there is no download that restores your unique values. Partition names and block numbers are per-device — look them up, never copy numbers from a guide.

su -c 'ls -l /dev/block/by-name/' | grep -Ei 'efs|modemst|fsg|fsc|persist|nvram|nvdata|protect'
# Qualcomm: commonly modemst1, modemst2, fsg, fsc, persist
# MediaTek: commonly nvram, nvdata, protect_f, protect_s, persist
for p in modemst1 modemst2 fsg persist; do
  su -c "dd if=/dev/block/by-name/$p of=/sdcard/efs-$p.img bs=4M"
done

Pull the images off the device and keep copies on a host/NAS — a backup that lives only on the partition you are about to wipe is no backup. Restore the same way (dd … of=/dev/block/by-name/$p) only to the identical partition on the identical device. Never flash one phone's EFS onto another — you would overwrite its unique identity with useless values.

4.4 Restoring raw `/data/data` tars correctly (UID + SELinux)

A tar of /data/data/<pkg> restored by hand will not launch unless you fix two things Android assigns at install time:

Install the app first (same package, ideally same version) so Android creates the data dir with the correct per-app UID and SELinux context.

Restore the files, then fix ownership and labels:

# find the app's real UID from the freshly-created dir, don't assume 100xx
su -c 'ls -ld /data/data/com.example'          # e.g. u0_a142  u0_a142
su -c 'tar -xpf /sdcard/appdata-example.tar -C /'
su -c 'chown -R u0_a142:u0_a142 /data/data/com.example'
su -c 'restorecon -RFD /data/data/com.example' # argument order matters; -v to see work

Then reboot before launching. Skipping the UID chown or restorecon is the usual reason a "successful" restore produces an app that crashes on open. Dedicated backup apps (§4.1) do this bookkeeping for you — hand-tar only when you must.

4.5 Getting root headlessly (no GUI taps) when the device boots

If the bootloader is unlocked and you need root to reach §4 but can't tap through the Magisk app, patch and flash boot entirely from the shell:

Get the exact stock boot.img. The authoritative copy is the live partition — dump it from a root recovery (§3) with dd/exec-out. Don't flash a boot.img of unknown origin: check its cmdline first (androidboot.hardware=qcom = Qualcomm vs MediaTek's bootopt=… — a wrong-SoC image bricks the device).
Patch it with Magisk's own tooling. The Magisk APK is a zip: its assets/boot_patch.sh needs the arm64 libs renamed (lib/arm64-v8a/libmagiskboot.so→magiskboot, libmagiskinit.so→ magiskinit, libmagisk.so→magisk, libinit-ld.so→init-ld) plus stub.apk and util_functions.sh in the same dir, then sh boot_patch.sh boot.img → new-boot.img.
- Set the flags explicitly (standalone boot_patch.sh treats unset as false, which can bootloop): for a modern FBE device with a vbmeta partition and dynamic partitions use KEEPVERITY=true KEEPFORCEENCRYPT=true PATCHVBMETAFLAG=false RECOVERYMODE=false LEGACYSAR=false. Failed to patch on the kernel DTB step is benign (nothing to patch).
- The recovery installer (META-INF/com/google/android/update-binary, which auto-detects these flags via get_flags()) is cleaner but fails outside a real sideload: it tries to mount /system, which on super/dynamic-partition ROMs is /dev/mapper/system and won't mount in a plain adb shell. boot_patch.sh with explicit flags sidesteps that.
Flash new-boot.img (verify it unpacks: magiskboot unpack new-boot.img) with fastboot flash boot or, from a root recovery, dd … of=/dev/block/by-name/boot. Keep the stock image for instant rollback. Reboot: magiskd in ps -A + magisk -V confirms root took and survives reboot.
Grant su to the ADB shell (Magisk denies it by default, silently): Magisk app → Superuser access = "Apps and ADB" + Automatic response = "Grant", or seed a policies row (uid=2000 policy=2) in /data/adb/magisk.db. Until then adb shell su -c … returns Permission denied even though root is installed and working.

5. Device Owner (DPC) — narrow, not a bulk extractor

A device-owner app (set with adb shell dpm set-device-owner <pkg>/<receiver> on a device with no accounts added yet) gets elevated policy powers, not filesystem powers. It is relevant to extraction only in that it can silently grant its own runtime permissions and read some content providers that would otherwise prompt — useful for a managed enrolment flow, not for cleaning out a phone before a wipe. Hard limits:

setPermissionGrantState() for sensor permissions (location, camera, mic, body sensors, activity recognition) is restricted/deprecated for profile owners on Android 12+ (silently returns false); device owners retain more but the surface is shrinking each release.
It cannot read other apps' /data/data, cannot image partitions, and cannot reach EFS.

If you find yourself reaching for device-owner to extract data, the real answer is almost always root (§4) or ADB providers (§6.2) instead.

6. ADB & Shizuku — no root, device boots

Plain ADB runs as the shell user (UID 2000). Shizuku grants that same privilege to on-device apps (started via wireless debugging or a one-time ADB command), so their reach is identical to ADB — handy when you have no PC.

6.1 External storage & scoped app dirs

adb pull /sdcard/ ./sdcard-backup/          # DCIM, Download, Documents, etc. — works everywhere
# Android/data and Android/obb are blocked from on-device file managers (11/13+) but
# the adb shell UID can still read them:
adb pull /sdcard/Android/data/ ./android-data/
adb pull /sdcard/Android/obb/  ./android-obb/

6.2 Content providers (telephony/PIM without root)

adb shell content query reaches providers the shell UID is allowed to read. What actually works on a stock device is uneven:

adb shell content query --uri content://contacts/phones          # contacts — usually readable
adb shell content query --uri content://com.android.calendar/events
# SMS/MMS and call_log generally FAIL for the shell UID (need READ_SMS / be default app):
adb shell content query --uri content://sms                      # typically permission denied

For SMS/MMS, call logs, and contacts as restorable files, use an on-device FOSS app rather than fighting providers over ADB:

SMS Import / Export (com.github.tmo1.sms_ie, F-Droid) — SMS+MMS+call log+contacts to JSON/ZIP, no root, and it restores too (set it as default SMS app during restore).
Open Android Backup (mrrfv) — PC-driven, rootless, uses a companion app (not adb backup): restores apps-without-data, internal storage, and contacts (vCard); exports SMS (CSV) and call log (txt) as view-only (no restore, no MMS attachments).
Contacts also export cleanly from the stock Contacts app ("Export to .vcf") or via Google account sync — often the least-effort path for the non-technical case.

6.3 `run-as` — app data for debuggable apps only

adb shell run-as com.example.debug tar -cpf - ./ > example-data.tar

run-as only works for apps with android:debuggable="true" (i.e. your own dev builds, not Play Store releases). Shizuku-based file managers use the same run-as path and inherit the same limit. Release-app /data/data is not reachable at this tier — that needs root (§4) or a recovery-decrypt (§3).

6.4 Bugreport & app lists (cheap, always worth grabbing)

adb bugreport ./bugreport.zip          # logs, settings dump, installed-package metadata
adb shell pm list packages -3 -f       # third-party packages + APK paths, to reinstall later
adb shell cmd package list packages --show-versioncode

To re-pull the APKs themselves for reinstall:

adb shell pm path com.example                       # -> package:/data/app/.../base.apk + splits
adb pull /data/app/~~.../com.example-.../base.apk    # repeat for each split; reinstall with
adb install-multiple base.apk split_config.*.apk     #   install-multiple

6.5 Vendor D2D and cloud (no-tech last mile)

Google cloud backup + Pixel/Android cable "device-to-device" restore at setup. App data rides along only if the Google account is transferred (Block Store is account-bound); untick nothing under "Accounts". allowBackup="false" stops cloud backup but D2D still transfers app data unless the app sets dataExtractionRules/android:fullBackupOnly to opt out.
Samsung Smart Switch (PC or cable): backs up most app data, but encrypted app data (e.g. WhatsApp), banking logins, and sideloaded APKs do not transfer — handle those per §9/§10.
Seedvault (CalyxOS/GrapheneOS/LineageOS): the FOSS backup transport. On Android 15 builds its device-to-device mode (Expert settings) is default-on and captures far more app data than the old cloud-style transport, because it presents as a D2D migration to each app.

7. The death of `adb backup` (don't build a plan on it)

adb backup was deprecated in Android 10 and gutted in Android 12 (API 31): app data is excluded by default and only included if the app is debuggable, so on essentially all release apps across Android 12–16 adb backup produces an archive with no app data. Treat it as unavailable:

Don't rely on it for a migration. Use root tooling (§4.1), on-device FOSS exporters (§6.2), or vendor/Seedvault D2D (§6.5) instead.
adb restore likewise cannot restore encrypted backups and is a dead end for modern data.
The bmgr/LocalTransport backup-manager plumbing still exists for developers testing their own backup agents (adb shell bmgr …), but it is a per-app developer tool, not a device-wide extractor — out of scope for a pre-wipe sweep.

8. EDL / firehose / BROM — dead or bootloader-locked devices

The lowest level, for devices that will not boot to Android or recovery. These talk to the SoC's emergency loader below the OS:

Qualcomm EDL (9008) — bkerler/edl with a firehose programmer (.mbn/.elf) matching the SoC. edl printgpt lists partitions; edl rl dumps --skip=userdata --genxml dumps all partitions (minus the huge userdata) to a folder with a rawprogram XML; edl r <partition> <out.img> grabs one. Newer SoCs need --loader=<file> because loader autodetection no longer works.
MediaTek BROM — bkerler/mtkclient. ./mtk printgpt, ./mtk rl out dumps all partitions to out/, ./mtk r <partition> <out.img> for one; --preloader for BROM mode.

The hard limit: these dumps are ciphertext. EDL reads the raw flash below the OS, so userdata comes out FBE-encrypted and is not decryptable without the keys (which live in secure hardware keyed by the lockscreen credential). EDL is therefore excellent for small identity partitions (EFS/persist/nvram — §4.3), full-flash forensic images, and un-bricking, but it is not a way to read a locked phone's app data. Treat downloaded dumps/tools as untrusted input: keep each in its own directory, pass paths as arguments, and run any helper scripts from outside the dump directory.

9. App-specific data that needs its own handling

Some high-value data lives behind app-level encryption and ignores the tier ladder:

WhatsApp — local backup at /sdcard/WhatsApp/Databases/msgstore.db.crypt15 (+ /Media). If the user enabled end-to-end encrypted backups, decryption needs the 64-digit key (or the encrypted_backup.key file from /data/data/com.whatsapp/files/, root-only). Tools: wa-crypt-tools. Easiest migration is WhatsApp's own built-in transfer or Google-Drive backup with the same phone number — pull the crypt15 + media as insurance, not as the primary path.
Signal — does not sync to any cloud. Migrate with Signal's in-app device-transfer, or its encrypted local backup (Settings → Chats → Backups) whose 30-digit passphrase you must record — there is no recovery without it.
Authenticator / 2FA — Google Authenticator exports via its QR "export accounts" flow; Aegis (FOSS) exports an encrypted JSON vault. Do this before wiping; TOTP seeds not exported are gone.

10. What will NOT survive a wipe (plan around it)

No tier recovers these; the only mitigation is to migrate them before the wipe:

Hardware-backed Keystore keys are non-exportable by design — key material never leaves the TEE/StrongBox. Any app that pinned a device-bound key (banking apps, some password managers, FIDO/passkeys stored in hardware) will treat the restored/new device as untrusted and re-enrol.
Play Integrity / SafetyNet attestation fails on rooted or custom-ROM devices independently of your data — so a perfectly-restored banking app may still refuse to run. That is an attestation problem, not a backup problem; don't chase it as data loss.
SSAID / ANDROID_ID changes on wipe (per app-signing-key + user), which can log apps out or reset entitlements even when their /data/data is restored byte-perfect.

Before wiping: export 2FA (§9), confirm password-manager vaults sync to their cloud, record Signal (§9) and WhatsApp keys, and re-enrol passkeys/banking on the new device rather than expecting a restore to carry them.

11. Pre-wipe checklist (the one-screen version)

Decide the tier (§1). Booting + unlockable → root (§4) or ADB (§6). Not booting → recovery (§3) → EDL (§8).
EFS/modem/persist first (§4.3) — before any unlock or flash, off-device.
Media & docs — adb pull /sdcard (§6.1).
App data — root tar + dedicated tool (§4.1), or recovery-decrypted Data backup (§3).
Scoped dirs — Android/data, Android/obb (§6.1).
PIM — SMS/call-log/contacts via SMS Import-Export or vCard (§6.2).
App-specific — WhatsApp/Signal/2FA via their own flows (§9).
App list + APKs — pm list packages -3, pull splits (§6.4) to rebuild the app set.
System state the app tars miss (§12.1) — keystore DB, SSAIDs, URI grants, doze whitelist, app-ops, launcher/shortcuts/QS/controls, Wi-Fi, SD-card UUIDs, root grants, old packages.list.
Baseline + app-sweep the old ROM (measure.sh, test-all-apps.sh, §12.4) so the new one has something to be compared against.
Passwords apps keep themselves — App Manager's KeyStore password, backup-app passphrases, Signal/WhatsApp keys (§9). Write them down before the wipe; none are recoverable after.
Verify the backups off-device (open the archive, check ls /data/data really decrypted) before you type fastboot -w or confirm the reset. A backup you have not read back is not a backup.

12. Restoring onto a new ROM (root) — what a data tar does not carry

Learned from a full LineageOS-based A16 → Infinity-X A17 switch (Redmi Note 7, Magisk). Restoring /data/data + user_de + Android/data per §4.4 brings back most apps, but every app gets a new UID on the new ROM, and a lot of state is keyed by UID or lives in system files. The scripts in scripts/android/ automate the parts below; they take the device from ANDROID_SERIAL like adb itself.

12.1 Grab these too, before the wipe (all root, all small)

What	Where	Why it matters on restore
Old UID map	`/data/system/packages.list`	Remap everything keyed by UID (SSAID, keystore, grants)
Keystore2 DB	`/data/misc/keystore/persistent.sqlite`	Apps whose keys vanish show "Keystore error"/log out (§12.3)
SSAIDs	`/data/system/users/0/settings_ssaid.xml`	`ANDROID_ID` change logs apps out (GitHub, Bolt, Lovense, …)
Persistent SAF grants	`/data/system/urigrants.xml`	Folder pickers (music players, camera, sync apps) lose their folders
Doze whitelist	`/data/system/deviceidle.xml`	Notification/automation apps silently stop in background
App-ops & special access	`cmd appops dump`, `cmd notification` policy access, `dumpsys package`	`MANAGE_EXTERNAL_STORAGE`, usage stats, DND, `READ_LOGS`, install-packages, notification listeners are not runtime permissions
Launcher + pinned shortcuts	launcher DB, `/data/system_ce/0/shortcut_service/`	Home screen; pinned shortcuts (e.g. Ice Box) are owned by the system, not the launcher
QS tiles, device controls	`settings get secure sysui_qs_tiles`; SystemUI `user_de` `files/controls_favorites.xml` + `shared_prefs/controls_prefs.xml`	Home Assistant controls, dev tiles
Wi-Fi	`/data/misc/apex/com.android.wifi/` or `cmd wifi list-networks` + PSKs	Also note which SSIDs must not auto-join
Root grants + Magisk settings	`magisk --sqlite "SELECT * FROM policies"`, `… FROM settings`, `… FROM denylist`, `/data/adb/modules` (note which are disabled)	UIDs change, grants don't follow; Zygisk off ⇒ DenyList silently does nothing
Accounts	`dumpsys account`; `/data/system_ce/0/accounts_ce.db`	Shows which logins (2nd Google account, app-registered accounts) must be re-added by hand — they can't be transplanted
SD-card volume IDs	`blkid` of each SD partition	Public volume ID = filesystem UUID; reformat ⇒ new ID ⇒ broken paths and grants
Freezer state	`pm list packages -d` + the freezer app's own list	Updates/reinstalls unfreeze

12.2 Restore order and traps

Update first, restore data second. Installing the old APKs and then the store's update is fine; restoring data into an older APK than the data was written by is a downgrade — apps wipe or crash on it. Never re-run a full restore pass over updated apps; restore data only.
Installer of record: pm set-installer refuses even root. Reinstall the same APKs with adb install-multiple -r -i <installer> (keeps data) so Play/Aurora/F-Droid/Obtainium see their apps as updatable (fix-installers.py).
New signature, same package (forks such as Shizuku → Shizuku-Next for a newer Android): tar the data dirs, uninstall, install, untar, chown/chcon, re-grant, re-freeze — reinstall-keep-data.sh.
System files are read only at system_server start. Write urigrants.xml, settings_ssaid.xml, shortcut_service/* etc. (owner system:system, mode 600, u:object_r:system_data_file:s0), then kill -9 $(pidof system_server) — a soft restart that does not save over them first, which a normal reboot does. It locks the user: unlock with wake → swipe → input text <PIN> → ENTER (the bouncer screenshots black; retry if needed).
fs-verity: files under shortcut_service are verity-protected — rm then write, never edit.
Restored caches can kill an app: Ice Box OOM-crashed on a restored MMKV package cache. Drop cache, code_cache, WebView caches and per-app package-list caches; keep the real state.
Re-map UID-keyed state with the old/new packages.list (SSAID name="<uid>", keystore namespace, Magisk policies.uid, doze lists are by package and need no remap).
SD card: if the partition had to be recreated, set the old UUID back (tune2fs -U <old> for ext4, fsck before and after) rather than fixing every app.
Wi-Fi: restored networks auto-join in their own order; re-add the ones that must stay manual with cmd wifi add-network <ssid> wpa2 <psk> -d.
QS tiles on A17: settings put secure sysui_qs_tiles is ignored; use cmd statusbar set-tiles "<list>". Developer tiles must be enabled first (pm enable 'com.android.settings/.development.qstile.DevelopmentTiles$SensorsOff').
Launcher switch (Nova had no build for the new Android): converting Nova's favorites table into Lawnchair's Launcher3 schema works (nova-to-lawnchair.py) — mark widgets restored=1 so Launcher3 binds fresh widget IDs, and re-pin deep shortcuts for the new launcher.

Banking-app root checks are not a restore problem, but they show up right after one. Order that worked (2026-10, Magisk 31, A17): first the ROM's own spoofing settings, if it has any (PIF, TrickyStore target and keybox, hide developer status). If an app still crashes on purpose, look for an Xposed module that defuses that app's check. For PayPal 10.18 that was oslokrok on Vector, the LSPosed successor. Configure Vector headlessly with /data/adb/modules/zygisk_vector/cli modules enable <pkg> and cli scope set <pkg> <app>/0. Take the app off the Magisk DenyList: the DenyList strips every Zygisk module from the process, including the hook.

12.3 Keystore: transplanting app keys

Hardware-backed keys never leave the TEE, but keystore2 stores them wrapped in its SQLite DB. If the new ROM runs on the same TEE (compare keyentry.km_uuid old vs new — same device, same firmware ⇒ same), the old wrapped blobs still unwrap. Keys whose blobmetadata has tag 0 (EncryptedBy) are additionally super-encrypted with the old lock-screen credential's key and cannot be moved. Domain 0 (APP) rows have namespace = uid → remap. Stop the apps, back up the live DB, insert keyentry/blobentry/blobmetadata/keyparameter/keymetadata with fresh ids, reboot. Apps that already regenerated keys on the new ROM also need their data re-restored to match. scripts/android/keystore-transplant.py OLD_DB [--replace] OLD_UID:NEW_UID … writes the SQL; dry-run it on a host copy of the live DB (pragma integrity_check) before applying with the device's own sqlite3. Tested: transplanted keys load with no keystore errors after a reboot. That does not bring back sessions the server ended, or apps that refuse to run because their build is too old.

12.4 Verify like a user would

test-all-apps.sh OUT cold-starts every user app (unfreezing and re-freezing), screenshots it, and greps its log for crashes, ANRs and keystore/crypto errors → results.tsv. Look at every screenshot: logouts, licence checks, ToS/"what's new" dialogs and permission prompts don't crash.
Android 17 shell launches must match the activity's intent filter: am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -n 'pkg/act' (single-quote it — component names contain $). -n alone gives "Access blocked".
measure.sh OUT [pkg…] snapshots memory, crashes/ANRs/lmk kills since boot and cold-launch times; run it on the old ROM before the wipe so there is a baseline.
Keep a running log of every security setting weakened during the restore (adb stay-awake, root grants, doze exemptions, …) with its revert command; the restore needs many of them and nobody remembers them afterwards.

35 KiB Raw Permalink Blame History Unescape Escape