🗜 Git Bloat Lockdown

Headline numbers

How to recover your clone

Every other PC that has a clone of this repo must follow the recovery doc before the next git pull. The full guide is at:

→ docs/GIT_HISTORY_REWRITE_2026-05-25.md (also viewable via gh api repos/eltonaguiar/findtorontoevents_antigravity.ca/contents/docs/GIT_HISTORY_REWRITE_2026-05-25.md)

Validator script

After recovery, run scripts/validate-history-rewrite.sh from the repo root. It runs 8 checks and prints PASS/FAIL per check + overall verdict:

1. Recovery doc present
2. Local HEAD matches origin/main
3. git fsck clean (no missing blobs)
4. Post-rewrite gitignore commits in lineage
5. No scrubbed bloat paths tracked in index
6. Gitignore covers the scrubbed paths
7. No unresolved merge conflicts
8. git pull --ff-only origin main would succeed

All 8 must PASS. If any FAIL, the script shows the specific fix command.

The 11 gitignore rounds

Each round was driven by a different audit — local heuristics, mirror-clone history scan, Roo on paid-mode-fast, Roo on free-mode-fast, Cursor on the same PC, GitHub Copilot, multi-AI fan-out. Cumulative effect: a defense-in-depth pattern that catches every category of regenerable bloat the repo's CI/training pipelines emit.

The "parent-dir excludes block file-level negations" gotcha

This caused two production-deploy regressions during the session — `super_signals.json` then `competition-stocks.json` both went missing from CI fresh-checkouts. Each time the FTP put failed with "No such file or directory".

The fix pattern

Replace bare directory ignores with file-globs, then add file-level negations:

# DOES NOT WORK — negation is silently ignored
STOCKS/competition/
!STOCKS/competition/forward_picks.json

# WORKS — file-glob + negations
STOCKS/competition/*.json
!STOCKS/competition/competition-crypto.json
!STOCKS/competition/competition-forex.json
!STOCKS/competition/forward_picks.json
# ... etc for every deploy-required file in the dir

The "gitignore vs git add -f" dance for CI writers

Some workflows are the canonical writer for files that should NOT be tracked by devs locally. For example, .github/workflows/train_crypto_models.yml retrains and commits ML model weights every day — but if a dev does git add -A locally, we don't want their stale copies leaking into the index.

Pattern: gitignore the path AND have the canonical writer use git add -f to bypass gitignore for its specific commit. v9 fix in this session shows the template:

# In .gitignore (blocks local dev adds):
ml_crypto_predictor/models/*.pkl

# In the workflow (canonical writer keeps working):
git add -f ml_crypto_predictor/models/ \
           ml_crypto_predictor/production_models/ \
           ml_crypto_predictor/optim/ \
           backtest_results/ updates/data/ || true

Audit of all 20+ workflows that use git add against gitignored paths showed they already correctly use -f for paths that need it. quick-guess-ml.yml, now-scanner.yml, alpha-trend-catcher.yml etc. all follow the pattern.

Critical files preserved via !negations (don't break these)

Each of these is referenced by a deploy workflow (FTP put / scp / cp) and MUST stay tracked. The gitignore explicitly un-ignores them via ! after a broader pattern that would otherwise catch them:

If a deploy starts failing

Diagnostic flow when a deploy workflow fails with "No such file or directory":

1. Identify the missing file from the error log.
2. Run git check-ignore -v <file> — this prints the matching .gitignore line if it's being blocked.
3. If matched, add an explicit !negation right after the matching pattern in .gitignore.
4. git add -f <file>, commit, push.
5. The recovery doc has an "open PRs" section explaining how to rebase PRs onto new main if needed.

Open work post-session