NWS Product Debate

[chris87]

Yeah, good point RE: the perturbed members.  

 

Given those members, most of your distribution is going to come from the SREFs in almost each scenario.  But, based on past performance, I'm going to give the Euro, GFS and GGEM much more weighting than any individual SREF member and even more than the SREF mean.  Not sure what the best solution is there, but giving most weight to the SREFs is going to create issues in most storms.  

 

well eventually the computing power to run 100s to 1000s member ensemble so you can better quantify uncertainty -- suppose its probably not as far off as we think but forecasting itself is going to undergo a big change in coming decades and products such as these min/max are only the beginning -- but a valid argument could be made that they are a bit ahead of time given our current capabilities. 

[WxUSAF]

which is why they are adding a bunch of euro ens members next year (according to wxman)

Yeah, I saw that and that will certainly help greatly.  

[Ian]

are you talking about from a verification standpoint? if so, i'd agree, i have never understood why WFO-based verification stats are kept so secret -- because performance within the office is heavily scrutinized internally by regional and national leadership. 

From that standpoint and others. I think it might just be putting a whole bunch of scientists together who are very strongly wedded to their own opinions -- as all people really into weather forecasting tend to be. NWS is already exploring a lot to better communicate with the public and private sector especially in the social science realm, but social science is only part of creating a "weather ready nation," yet it's the primary buzzword these days. Outside cabinet level offices most gov orgs deal with big outside policy groups to resolve issues they cannot internally. NWS is very "not created here don't care" in their thinking on the whole I believe.

[chris87]

And again I commend you for discussing things here as I'm sure you do in many places.. from what I've seen you are a rare commodity in the organization. And I know enough people in it at various levels that I'm not just talking out my arse or something here. 

 

NWS is moving in the direction that will probably eventually resolve a lot of the issues currentl seen... I am just not sure it will ever happen fast enough to keep up with the rest of the world unless there is more outside interference run.  Unfortunately, IMO, everyone is so heavily focused on climate change that it's hard to find people who are as interested in similar policy ideas for simple weather.  I do think that will change as well eventually.. but I don't have 90% confidence yet.

 

this is a big issue -- but there are other issues that I think play a bigger role that are probably more pertinent to improvements in NWP/forecasting in the US. 

 

There's a reason the Europeans are so good at what they do -- they centralized all their capabilities both from a NWP/DA standpoint and from a remote sensing standpoint, now they don't do everything better than us. But let's take DA for example in the US, we essentially have diluted our talent pool among two organizations, NOAA and NASA, each with their own perceived "mandate", each producing, at times, redundant capabilities rather than working towards one common goal, which in NOAA's case is improving short and mid-range weather forecasts. Not an easy to solve problem, NASA will proclaim that they need their own model development for their applications and won't want to rely on NOAA to produce those products -- we can debate if that's true or not, but unless we either see a dramatic increase in funding or a consolidation of our intellectual assets, we'll likely continue to hold back our potential for improvements. 

[ORH_wxman]

A highly automated output with only occasional/marginal manual augmentation is going to have its inherent flaws. Manual augmentation might be slightly higher at certain portions of the forecast cycle, but when the map is being generated, the goal is to have the output be as skillful as possible as to minimize the manual adjustments.

 

A good starting point would certianly be to weight the model guidance as needed....here is how the models have performed in the Northeast US the past 3 months at a 24 hour lead time (probably the period when products will be getting a lot of attention before a storm)...and its not an endorsement of the SREF, which isn't a surprise:

 

[Ocean Storm]

A highly automated output with only occasional/marginal manual augmentation is going to have its inherent flaws. Manual augmentation might be slightly higher at certain portions of the forecast cycle, but when the map is being generated, the goal is to have the output be as skillful as possible as to minimize the manual adjustments.

 

A good starting point would certianly be to weight the model guidance as needed....here is how the models have performed in the Northeast US the past 3 months at a 24 hour lead time (probably the period when products will be getting a lot of attention before a storm)...and its not an endorsement of the SREF, which isn't a surprise:

 

 

SREF Mean does nail precip forecast pretty well at shorter time range than 24? 

[ORH_wxman]

SREF Mean does nail precip forecast pretty well at shorter time range than 24? 

 

Yeah about 6-12 hours out it certainly does better...but then again, so do the other models. I don't have the 12 hour scores available unfortunately...maybe somebody else does.

[ers-wxman1]

And again I commend you for discussing things here as I'm sure you do in many places.. from what I've seen you are a rare commodity in the organization. And I know enough people in it at various levels that I'm not just talking out my arse or something here.

NWS is moving in the direction that will probably eventually resolve a lot of the issues currentl seen... I am just not sure it will ever happen fast enough to keep up with the rest of the world unless there is more outside interference run. Unfortunately, IMO, everyone is so heavily focused on climate change that it's hard to find people who are as interested in similar policy ideas for simple weather. I do think that will change as well eventually.. but I don't have 90% confidence yet.

I want to take this time to explain what I'm involved with and what it means for the agency. I think doing this will answer some questions and bring a bit more transparency. Here goes...

My position is new in the NWS. I'm an ERS Meteorologist, Emergency Response Meteorologist. LWX is currently 1 of 6 offices that was selected to take part in a 3 year pilot project basically to test concepts that are core to the mission of the agency and to test and develop initiatives that will guide us into the future. So, the offices are: Sterling, Tampa, New Orleans, Charleston WV, NOC at HQ and ROC at SRH. Each pilot project has a different theme: ours is Providing IDSS (Impact Based Decision Support Services in an urban environment). We have 4 main projects going on here that involve improving communication, building lasting partnerships with stakeholders and so on. Our core three so to speak are the uncertainty communication - that's my baby. It's gone further than just winter stuff, building a weather ready nation local entities which tests new concepts with hospitals and other entities and an impacts catalog...gaining threshold information from em's and planners and using specific info to enhance warnings. We have developed pyramids of support with triggers and the like. We are in year 3 of this project. Tampa is working more with ecological partners, New Orleans built a mobile unit to deploy to events, the NOC and ROC are regional and national DSS centers. Charleston is the severe weather enhancement pilot.

There are currently 18 of us with ERS positions. We are fully trained, FEMA certified and we deploy to major events and support state, national level EM's etc. that's another function we do and we are finding that our core customers love having a met deployed to an active eoc during activations as the workload and complexity of situations changes plus we gain feedback on what's good, what's not and how to improve.

All of the work that is being done by these 6 offices will be reviewed and considered for future work/best practices in the agency. It is very important stuff.

After the 3 years is up the pilot projects will end. Then not sure where I will go. But I'm here as a change agent but to build better things, to make us better and I'm a HUGE believer in relationships. You have to build trust to get places even if you take it on the chin a few times. Some of the feedback is tough but its good. We need it.

That's a bit of background as to why we are doing what we do and that there are 5 other entities testing other great things.

There is a super sharp colleague at Charleston who develop a tool... Check it out if you can. I won't market it as great since the last time I said "great" it was rammed down my throat ;-) lol.

www.preview.weather.gov/edd

Thx,

Ers

[ers-wxman1]

Or just:

preview.weather.gov/edd

[Deck Pic]

I want to take this time to explain what I'm involved with and what it means for the agency. I think doing this will answer some questions and bring a bit more transparency. Here goes...

My position is new in the NWS. I'm an ERS Meteorologist, Emergency Response Meteorologist. LWX is currently 1 of 6 offices that was selected to take part in a 3 year pilot project basically to test concepts that are core to the mission of the agency and to test and develop initiatives that will guide us into the future. So, the offices are: Sterling, Tampa, New Orleans, Charleston WV, NOC at HQ and ROC at SRH. Each pilot project has a different theme: ours is Providing IDSS (Impact Based Decision Support Services in an urban environment). We have 4 main projects going on here that involve improving communication, building lasting partnerships with stakeholders and so on. Our core three so to speak are the uncertainty communication - that's my baby. It's gone further than just winter stuff, building a weather ready nation local entities which tests new concepts with hospitals and other entities and an impacts catalog...gaining threshold information from em's and planners and using specific info to enhance warnings. We have developed pyramids of support with triggers and the like. We are in year 3 of this project. Tampa is working more with ecological partners, New Orleans built a mobile unit to deploy to events, the NOC and ROC are regional and national DSS centers. Charleston is the severe weather enhancement pilot.

There are currently 18 of us with ERS positions. We are fully trained, FEMA certified and we deploy to major events and support state, national level EM's etc. that's another function we do and we are finding that our core customers love having a met deployed to an active eoc during activations as the workload and complexity of situations changes plus we gain feedback on what's good, what's not and how to improve.

All of the work that is being done by these 6 offices will be reviewed and considered for future work/best practices in the agency. It is very important stuff.

After the 3 years is up the pilot projects will end. Then not sure where I will go. But I'm here as a change agent but to build better things, to make us better and I'm a HUGE believer in relationships. You have to build trust to get places even if you take it on the chin a few times. Some of the feedback is tough but its good. We need it.

That's a bit of background as to why we are doing what we do and that there are 5 other entities testing other great things.

There is a super sharp colleague at Charleston who develop a tool... Check it out if you can. I won't market it as great since the last time I said "great" it was rammed down my throat ;-) lol.

www.preview.weather.gov/edd

Thx,

Ers

 

dude..you have the coolest job ever...I can't believe I am currently reviewing dense corporate documents in a windowless office.  

[ers-wxman1]

dude..you have the coolest job ever...I can't believe I am currently reviewing dense corporate documents in a windowless office.

I'm doing this because I truly care. I really do. I didn't join the forum to get a red tag and market my status. In fact I started anonymous with a blue tag but no one would talk to me ;-). I'm not here to bash...I'm here to share passion and make friends in the process. I enjoy this and I enjoy the feedback. We need it.

[PhineasC]

Well unless you guys become much more transparent it's hard to judge exactly what that means from the outside. Input could be glancing and approving at least in some cases it seems. Otherwise there must be a team that only creates those maps given all the points and products.

I don't really blame NWS for being heavily internalized with their processes. I think it's just a big gov thing and there is not really any outside org pushing for policy changes etc like there is with most other big gov entities. Not to mention the org is hugely balkanized which presents a ton of problems in itself.

Is it really that much more opaque than CWG? Honest question. When you guys huddle to build a forecast is it an open telecon where anyone in DCA can dial-in and provide input? I get the argument that the difference is one is being paid with tax dollars, but strictly from a performance and process perspective...

[ers-wxman1]

Is it really that much more opaque than CWG? Honest question. When you guys huddle to build a forecast is it an open telecon where anyone in DCA can dial-in and provide input? I get the argument that the difference is one is being paid with tax dollars, but strictly from a performance and process perspective...

Internal only

[ers-wxman1]

If I may post one more time on this section let me explain these new products this way... Perhaps this clears up some confusion...

The best way to read those new maps is not in a 10/90 sense but simply... Expect at least this much (min), prepare for up to this much (max) but our highest confidence is this much (most likely). How we get to that conclusion... Start with number of members but I realize that we make edits and tweaks graphically that alter the distribution of percentiles. We want our risk planners to not focus on percentiles but a range of possible solutions and confidence in those solutions.

Hope this helps as the foundation. Whether they are too high based on the starting point will certainly be tweaked and re tweaked over months.

ERS

[MN Transplant]

If I may post one more time on this section let me explain these new products this way... Perhaps this clears up some confusion...

The best way to read those new maps is not in a 10/90 sense but simply... Expect at least this much (min), prepare for up to this much (max) but our highest confidence is this much (most likely). How we get to that conclusion... Start with number of members but I realize that we make edits and tweaks graphically that alter the distribution of percentiles. We want our risk planners to not focus on percentiles but a range of possible solutions and confidence in those solutions.

Hope this helps as the foundation. Whether they are too high based on the starting point will certainly be tweaked and re tweaked over months.

ERS

 

Has there been any consideration to take the 10% wording off of the images themselves, or at least make it less prominent?  If the graphs themselves are getting passed around on social media, people are probably not going to the NWS link to see the wording there.  Seems like the "expect this much" message should be on the actual images.

[ers-wxman1]

Has there been any consideration to take the 10% wording off of the images themselves, or at least make it less prominent? If the graphs themselves are getting passed around on social media, people are probably not going to the NWS link to see the wording there. Seems like the "expect this much" message should be on the actual images.

That's a valid point. Have you or anyone on this forum considered leaving feedback on the site? There is a tab for that. There is a survey that will ultimately go into my email. Thus far our core customers love it and there aren't many objections but people HERE have valid points and they would best be stated in this way. Once winter is over its our job to analyze all the feedback and make changes where appropriate.

[ers-wxman1]

Comments here could get lost. That's my fear. State your points on the feedback survey.

www.erh.noaa.gov/LWX/winter

Each product has a leave feedback link. Now if we see posts like this is worthless and stupid they will most likely be deleted like a false spotter who sends in cantaloupe hail in a qlcs line.

[ers-wxman1]

Believe it or not we actually had a couple events where the actual snowfall was higher then the max graphic. It did happen this year and since then have tried the 95th percentile as the worst case scenario.

[chris87]

Comments here could get lost. That's my fear. State your points on the feedback survey.

www.erh.noaa.gov/LWX/winter

Each product has a leave feedback link. Now if we see posts like this is worthless and stupid they will most likely be deleted like a false spotter who sends in cantaloupe hail in a qlcs line.

 

i know this seems like an extra step to most but i'll reiterate how important it is to go through these type of channels to provide feedback as it's much easier for a forecaster (or other relevant worker in our bureaucracy) to pass official feedback up the ladder to management.

[ers-wxman1]

i know this seems like an extra step to most but i'll reiterate how important it is to go through these type of channels to provide feedback as it's much easier for a forecaster (or other relevant worker in our bureaucracy) to pass official feedback up the ladder to management.

Feedback comes to our winter team. I've gotten a bunch this winter thus far.

[ohleary]

They did, but it was a different product, and equally flawed.

 

The 10th percentile amount is suggested as an amount that represents the least amount of snow the storm can produce or that one should expect this amount as a minimum.  I don't think any operational met would agree that at the time this was issued that 3.4" was representative of the least amount of snow the storm could produce.  I didn't follow these products as closely earlier in the winter, but I am curious what this number was for 12/10.

 

What about "Experimental" do you not understand?  These products are in development.  Hence they are not perfect.  If you don't like them, don't use them, it's that simple.  But to expect an experimental product to be perfect and rail like a maniac against them and the people who develop them is retarded.

[Deck Pic]

What about "Experimental" do you not understand?  These products are in development.  Hence they are not perfect.  If you don't like them, don't use them, it's that simple.  But to expect an experimental product to be perfect and rail like a maniac against them and the people who develop them is retarded.

 

on the website there is a solicitation for feedback, and in this thread there has been significant back and forth with other people giving their input, not just me. Most of the conversation has been productive.  So I don't understand your point.

[Deck Pic]

This is always your token response to me. I've read the thread and I felt you came off as overtly aggressive. That's my opinion and I find it hard to believe you really just wanted to find out what the new product was all about, rather you wanted to lower a blow.    

 

 

ers covered it and wxman made a nice post about it

 

The are mostly SREF based with a few other american ensemble members thrown in....they do get a human massage, but my guess is the human tweaking is probably limited....I doubt someone can take a 50% number of say 4-6" and cut it in half...not sure what the guidance is, but I doubt there can be a complete override.....so they are kind of hamstrung, and they are experimental products....next year they are going to get some euro ens members thrown in as well as maybe some others....which should smooth things out..nevertheless with auto generated snow algorithms usually running high I expect the products will still overdo snow a lot of the time...I shouldn't have laughed at some of the maps...It wasnt meant to insult anyone in particular....I do think they are somewhat off base...looks like the guidance has gotten wetter so I may end up eating some crow...I need to do a better job of criticizing constructively...I do think LWX should be more cautious of how they advertise certain products and how extensively....this kind of stuff can erode public confidence...

 

 

LWX did great with this storm and deserve kudos, but the products need to be judged based on the information available at the time...I approached it too harshly, but I stand by it 100%.  Even if not intended for the public (not so sure about this...it is readily available), a 10th percentile amount of 3.4" was totally unsupported by the envelope of data.  We know now that it is an experimental product, and they are largely hamstrung by an automated process heavily influence by SREFS and rounded out by some other American guidance.  I don't think they can completely toss a 32 member ensemble suite...only tweak.  It is good to know they are folding in some euro ensembles next year and it will make a flawed, but very cool and formidable product even better.  I don't think you will find a single operational met in DC metro who thinks 3.4" was a low goalpost.  CWG is at the very least on par with LWX as forecasters for DC metro, and at the time these products were being issued with those numbers, CWG had a 10th percentile goal post of less than 1".  I realize this is probably over your head, but I figured I'd recap.

 

 

A lot of storms completely bust here..that is DC...even this winter of several overperformers, the WSW on 12/10 busted in a lot of places - badly.  That isn't to say it was necessarily a bad forecast.. I agreed with it at the time.  Turns out it was another learning experience.  The signs of bust were there.

 

Now I don't think it is useful for a <6" storm to have a lower goalpost of complete fail from a certain range even if complete fail is probably 10th percentile for a lot of storms.  Saying every <3-6" storm could crap out is not useful guidance for planners and decision makers. Besides the things that have already been  exhaustively covered, at a certain range I'd probably use a 25%-75% range for planning purposes rather than 90-10, since 10% if portrayed accurately is often <=1".  25% for this storm was probably ~1-2" for immediate DC metro at the time these products were issued.

 

 

I think it is both.  Even if euro ensembles were folded in, a 10th percentile number is usually going to be higher than it should be since 10th percentile from a range for planning purposes is usually <1" except for our bigger events.  A lot of the public is dumb , but there are enough smart people out there who based on their understanding of percentiles, would consider a goal post or a number that is 10th percentile to be a virtual lock.

 

 

right.  What I am saying is this...I agree that conveying that a complete bust as a minimum from the range that is useful for high level planning is not helpful, EVEN if a <1" amount is in fact what should be the 10%.  Which is my "beef" and since the product seeks feedback, I am offering my feedback.  So in my opinion you are dealing with a flawed product, but a really cool one.  That's all.  I don't agree that at the range you issued the product that 3.4" was a figure that would be met or exceeded 90% of the time.  and YET if you had used a number that better reflects what is in fact the 10th percentile (imo), it wouldn't be helpful for planning purposes.  We can reasonably disagree on the number that has a 90% probability of being exceeded, but I don't think I am crazy for thinking it should have been way lower than 3.4" at the range the product was issued.  CWG (who busted badly imo and their grade of C is way too generous) are very talented mets, one of whom is a retired government met who is considered a winter weather expert, and at the time the 3.4" was issued, 3.4" was the higher end of their 1-4" (15th - 85th) forecast range.

 

 

dude..you have the coolest job ever...I can't believe I am currently reviewing dense corporate documents in a windowless office.  

[nj2va]

Comments here could get lost. That's my fear. State your points on the feedback survey.

www.erh.noaa.gov/LWX/winter

Each product has a leave feedback link. Now if we see posts like this is worthless and stupid they will most likely be deleted like a false spotter who sends in cantaloupe hail in a qlcs line.

 

Thanks for sharing this link.  I'll go in tonight when I'm home from work and add some comments. 

 

Also, thanks for explaining the products more here...both informative and interesting to read.

[ers-wxman1]

Thanks for sharing this link. I'll go in tonight when I'm home from work and add some comments.

Also, thanks for explaining the products more here...both informative and interesting to read.

Glad to be of help.

[blue sky]

I like the pilot programs that ers-wxman1 has mentioned.

I would love to see a pilot program were a central office forecasts the entire United States. Compare those forecasts with the multiple local offices. Then make a cost/accuracy decision on current NWS structure.

[isohume]

I like the pilot programs that ers-wxman1 has mentioned.

I would love to see a pilot program were a central office forecasts the entire United States. Compare those forecasts with the multiple local offices. Then make a cost/accuracy decision on current NWS structure.

 

 

That'd be great if all we did was the public fcst. Not sure they could centralize local research, COOP, media relations, fire wx, outreach/edu, radar/asos maintenance, state/county EM partnerships, upper air, aviation, NWR, severe ops, and hydro.

Perhaps in the future with some of these could be centralized and others contracted out. But with the current state of technology, the speed of govn't change, and the NWS mission as it is...it would likely be decades before a reasonable plan could be convincingly tested and approved by Congress.

[SLPressure]

I'm doing this because I truly care. I really do. I didn't join the forum to get a red tag and market my status. In fact I started anonymous with a blue tag but no one would talk to me ;-). I'm not here to bash...I'm here to share passion and make friends in the process. I enjoy this and I enjoy the feedback. We need it.

I wish to take this moment to say "Thank you" to ALL who have and are contributing to this dialogue. Having a thread dedicated to the NWS, and more specifically LWX, is fantastic. The reason for the creation of this thread highlights the need for discussion on the products, messages and reactions to those from the public.

While those of us on this board are, on average, more educated on matters concerning the weather than the general public, we are still "public". AND we still have a lot to learn regarding how the NWS is producing and conveying these issues to the public. To say nothing of the tax dollars that go into it all. This is, after all, OUR product. We all have an investment into seeing how it is run and eventually grows and improves. Feedback is essential. It is our job as the "public" to monitor and issue feedback ( praise and constructive criticism) and the NWS's job to absorb this feedback to further improve and advance their mission.

Keep up the good work.

Okay, enough of my rah rah rah! Back to the weather.

[ers-wxman1]

I wish to take this moment to say "Thank you" to ALL who have and are contributing to this dialogue. Having a thread dedicated to the NWS, and more specifically LWX, is fantastic. The reason for the creation of this thread highlights the need for discussion on the products, messages and reactions to those from the public.

While those of us on this board are, on average, more educated on matters concerning the weather than the general public, we are still "public". AND we still have a lot to learn regarding how the NWS is producing and conveying these issues to the public. To say nothing of the tax dollars that go into it all. This is, after all, OUR product. We all have an investment into seeing how it is run and eventually grows and improves. Feedback is essential. It is our job as the "public" to monitor and issue feedback ( praise and constructive criticism) and the NWS's job to absorb this feedback to further improve and advance their mission.

Keep up the good work.

Okay, enough of my rah rah rah! Back to the weather.

This is a very good post. Lets keep this thread going.

[WinterWxLuvr]

I have nothing but praise for LWX. I find their forecasts to be very good most of the time. Impossible job. One thing related to some of the disco from a couple days ago...I can't seem to find a good reason to put out min/max possibilities that seem so out of range that they are almost locks as far as what actually happens. A min of 1" and a max of 16" does me little good to know. It's like well, yeah. Why even bother?