I have a problem with that last sentence. Because larger files do take longer to load from storage into main memory, and then longer to load into VRAM from main memory. Also, with larger files, you can't keep them cached and ready to be reused, because you have to free that memory for other large files.
Your argument might be true if computers generally had RAM and/or VRAM larger than the entire game. But when games are 200+GB and typical main memory is 16-32GB for most folks, and only 64-128GB at the higher end, you know data will have to be shuffled around. VRAM situation is more dire: typical is 6-8GB, high end is 12-16GB and absolute max is 20-24GB.
Yes, faster storage and faster RAM help, but all those loads and unloads of huge chunks of data do take up time, cause stutters, or absurdly long loading screens despite the high performance components.
OPTIMIZE YOUR GAMES. Lossy compression is fine, and uncompressed assets should be optional downloads.
Bigger Than Godzilla: Why Are Games Using So Many Gigabytes? (www.techspot.com)