

You’re right.
We’ve decided that IPFS is not yet ready for prime time. We’ll still link to files on IPFS from Anna’s Archive when possible, but we won’t host it ourselves anymore, nor do we recommend others to mirror using IPFS. Please see our Torrents page if you want to help preserve our collection.
I’ve had great success with restic. It will handle your 4TB just fine, here’s some stats of mine:
Total File Count: 78374 Total Size: 13.324 TiB
and another one, not as large but with lots of files
Total File Count: 1295210 Total Size: 2.717 TiB
Restic will automatically deduplicate your data so your duplicates won’t waste storage at your backup location.
I’ve recently learned about backrest which can serve as a restic UI if you’re not comfortable with the cli, but I haven’t used it myself.
To clean your duplicates at the source I would look into Czkawka as another lemming already suggested.