I don't have much to add other than repeating how good this thread is, and thank you for the information.
Coming from a software engineering background, I am curious what programming language most models are developed in? Also, what kind of computers are they using to process all of the algorithms?
The computing systems are really a system of systems with Supercomputers doing a lot of the heavy lifting and mainframes and client/server stuff taking a significant amount of local load. Here's a high-level diagram of the overall architecture, with some stuff redacted out:
Here's a slide deck that details most of the infrastructure and its tie in to numerical weather prediction modelling. There are some interesting computing systems links on slide 68.
The local office stuff is typical of a remote or regional office. Local, high-res models with smaller domains are run at these sites so that 1) the main computing systems are not impeded and 2) it offers the local offices more flexibility in ad-hoc runs for significant or hyper-local forecasting needs. Here's a pic of our local office:
Here's a draft roadmap for the NextGen exascale computing systems that will likely power all sorts of government-facing numerical and graphical predictive analysis systems, including those in NOAA, going forward. I only have the draft on this workstation. Ping me via PM if you want the latest and I'll dig it up when I can.