Yeah I certainly agree there is definitely more to it than that.
The biggest weight overall I think is structure and placement of these anomalies and synoptic features. This is that extra piece which makes the seasonal forecasting challenging...sure we can have an idea of how NAO/AO may evolve based on QBO projection and how the stratosphere may evolve but we really don't know how they will be structured...and that's the key piece...I think anyways.
Do you remember way back I think his name was Allan (Raleighwx) did something with breaking the NAO into regions (similar to ENSO) with east-based and west-based +/- NAO events...he even had an index to measure it...but never heard of anything since and there is very little about this online. but doing the correlations to patterns by breaking down (say a negative NAO) between east/west based increased the correlation significantly.