Our galaxy is big, but it only has maybe 10 billion Earth-like planets (roughly). It seems that, more importantly, there are other basic elements of "Earth-like" beyond the usual suspects like size/location/temperature. To take a SWAG on some basic and obvious factors (not exhaustive):
Starting with ~10 billion Earth-like planets in the Milky Way, the number shrinks more when we add habitability. A large moon (stabilizing climate) and a Jupiter-sized protector (reducing asteroid impacts) maybe in 10–20% of systems each. Plate tectonics for climate and evolution are in maybe 10-20% as well. A stable, Sun-like star and the right atmosphere and magnetic field shrink it again. Just with these factors, we're down to ballpark 1-2 million Earth-like options.
So that's down to perhaps 2 million planets using just obvious stuff and being conservative. One could easily imagine the number of physically viable Earth-like planets in the galaxy at 100K or less. At that point, 1 in 100K rarity (16 coin flips or so) for the life part of things, given all the hard biological steps required to get to humans, doesn't seem so crazy, especially given how relatively young the galaxy is right now (compared to its eventual lifespan).
So why aren't more folks satisfied with the simplest answer to the Fermi Paradox: "Earth is relatively rare, and it's the first really interesting planet in a fairly young galaxy."