In my 30s, I successfully drove to and found the house that my family moved out of before I turned 2. I haven't been in it since we moved, but I could still draw you the floorplan today.
I've never considered myself to have a particularly good memory, so I don't doubt that anybody else would have an issue doing the same thing.
You may not have a good memory in general, but I think you have exceptionally good early memories. I can't remember a single room of the house I left when I was four.
I've never considered myself to have a particularly good memory, so I don't doubt that anybody else would have an issue doing the same thing.