A raspberry Pi is a very good emulation device using the RetroPie image. A Pi 4b can go up to PSX/N64 fairly easily.
On the handheld side of things, most of them that “come with” ROM sets will have them loaded on an SD card. These manufacturers often skimp on the cards though, so expect it to die quickly. You can usually just clone the whole SD to a new one and it’s fine.
Most of these devices use RetroArch and software emulation. However, there is another option. The Mister project and devices sold by Analogue use field programmable gate arrays - chips that can change their structure according to software. This means running an NES game on one of these devices is more literally like running it on original hardware. For accurate emulation, this is the best option by far. However, it comes with a significantly higher price tag.
In general the easiest and least expensive startup for emulation is on the PC. With fairly modest hardware, emulation of everything up to PS2 is possible with some newer platforms also being very emulatable (notably everything Nintendo puts out is easy to run because their architecture is largely straightforward, their systems are lower power, and there is significantly more demand for their games)
If you specifically want something hooked up to your TV, a first generation (launch window, before they increased the battery life) Switch can happily run a fair amount of stuff, including everything up to N64/PS1. The (new)3DS/2DS is also a great emulation device and can run basically everything up to SNES/Genesis handheld.
Oh and one more option. If you have Android, you can easily install a variety of emulators and use a Bluetooth or wired controller with them utilizing a controller phone mount.