A lot has to do with size as you said, since larger size means more water moving. More intensity obviously means more water moving as well, and the higher the intensity plus size means obviously a higher surge. The track of the storm matters, since a storm heading perpendicular into a shore will tend to maximize surge vs one tracking near parallel. But then you have to factor in local effects like where bays, inlets and harbors funnel water in and the slope of the shelf. There's a reason Tampa Bay, NY Harbor and Lake Pontchartrain are considered huge risk surge areas, because they are great at funneling the water in from a surge and maximizing it. Sandy was a high end Cat 1 but gigantic in size and about 10mb lower in pressure at landfall than Milton, which meant a huge amount of water moved, into a very bad area for surge due to the upside down L-shaped NY Harbor, lots of bays and inlets to funnel the water and a long sloping shelf. Sandy also tracked in a way that drove the surge NW directly into the harbor. Part of the impact is just the luck of what tide the surge hits during. Sandy's peak surge was during a full moon high tide which in my town added 4-5 feet on top to what it would've been at low tide (storm tide was 10 feet roughly where I lived then and even higher in places in NYC and certain spots in NJ around funnel shaped inlets). You pointed out Charley, and Andrew 1992 wasn't really known for surge other than a small part of Miami due to its size primarily and factors specific to SE FL.
The E coast of FL isn't as susceptible to surge because the shelf is very close to shore and you generally don't have all these bays and inlets to funnel the water in. But of course any low lying location is susceptible from the right storm at the wrong time.
Generally the IKE number as bdgwx posts often is a good indicator of the surge potential with the storm since it combines wind speed with overall storm size.
Here's a good article explaining the different factors. Storm Surge Overview (noaa.gov)