Weather Service stops receiving satellite data

[user13]

NCEP HAS NOT RECEIVED A FULL FEED OF SATELLITE DATA FOR INPUT INTO THE NUMERICAL MODELS SINCE 22/0000Z…POTENTIALLY IMPACTING THE MODEL FORECASTS.NESDIS AND NCEP ARE INVESTIGATING THE ROOT CAUSE OF THE ISSUE. ONCE THE SITUATION IS RESOLVED ANOTHER MESSAGE WILL FOLLOW.

(Looks like they got some data back online)

 

- http://www.washingtonpost.com/blogs/capital-weather-gang/wp/2014/10/22/weather-service-stops-receiving-satellite-data-issues-warning-about-forecast-reliability/?hpid=z5

[jaxjagman]

Wed Oct 22 23:32:21 2014 GMT

NOUS42 KWNO 222332
ADMNFD
SENIOR DUTY METEOROLOGIST NWS ADMINISTRATIVE MESSAGE
NWS NCEP CENTRAL OPERATIONS COLLEGE PARK MD
2331Z WED OCT 22 2014
NCEP IS NOW RECEIVING THE FOLLOWING NESDIS SATELLITE DATA FOR 00Z
MODEL INGEST..
NPP - CRiS AND ATMS DATA
GOES SATELLITE DERIVED WINDS
GOES RADIANCES
GOES SOUNDING PRODUCTS
THE FOLLOWING DATA TYPES CONTINUE TO BE UNAVAILABLE FOR THE
MODELS..
MODIS IR AND WV WINDS
OMI OZONE DATA
AIRS HYPERSPECTRAL SOUNDER DATA
COSMIC GPS-RADIO OCCULTATION DATA
NESDIS CONTINUES TO WORK ON RESTORING ALL THEIR SATELLITE DATA
PRODUCTS..
IT IS DIFFICULT TO ESTIMATE THE EXACT IMPACT OF THE SATELLITE
DATA OUTAGE ON NUMERICAL GUIDANCE AT THIS TIME.. BUT THE
DEGRADATION OF THE MODELS INCREASES WITH AN EXTENDED OUTAGE..
SHRUELL/SDM/NCO/NCEP

[AlaskaETC]

WTF happened?

[jaxjagman]

Just curious,could part of the loss be to  flares?

 

 

 

http://www.swpc.noaa.gov/

[ohleary]

BTW, CWG's article is all hype, little substance and contains misinformation. Hype it and the sheeple will come.

[LithiaWx]

WTF happened?

  

Just curious,could part of the loss be to  flares?

 

 

 

http://www.swpc.noaa.gov/

HAARP

[Guest Imperator]

As per a statement by Accuweather, the ECMWF is not affected by this outage.

 

http://www.accuweather.com/en/weather-news/noaa-network-issue-may-impact/36161909

[ohleary]

As per a statement by Accuweather, the ECMWF is not affected by this outage.

 

http://www.accuweather.com/en/weather-news/noaa-network-issue-may-impact/36161909

 

Oh Accuweather said it, it must be true.

 

Trust me, this:

 

"The global forecast model suite run by the European Consortium for Medium-Range Forecasting (ECMWF), the most accurate forecasting model in the world, remains fully operational."

 

Is Accuweather's way of trying to thumb it's nose at the NWS.  NCEP's production suite also remained fully operational.

 

Trust me...the Europeans were also affected by the outage, and likely much worse than NCEP.  Lord almighty.

[ohleary]

And...btw, the title is not factual.  NWS/NCEP did NOT "stop receiving satellite data".  There was a partial outage.   

[Guest Imperator]

Oh Accuweather said it, it must be true.

 

Trust me, this:

 

"The global forecast model suite run by the European Consortium for Medium-Range Forecasting (ECMWF), the most accurate forecasting model in the world, remains fully operational."

 

Is Accuweather's way of trying to thumb it's nose at the NWS.  NCEP's production suite also remained fully operational.

 

Trust me...the Europeans were also affected by

the outage, and likely much worse than NCEP.  Lord almighty.

 

Just reporting what I read, and out of fairness, per the Washington Post article:

 

"Chris Vaccaro, a spokesperson for the National Weather Service, said officials at NCEP told him these model forecasts can still be considered credible, despite the outage. “[T]here’s a lot of redundancy in the observing system that can help to offset the data loss and that the model guidance will have integrity and will be sound,” he said in an email."

 

http://www.washingtonpost.com/blogs/capital-weather-gang/wp/2014/10/22/weather-service-stops-receiving-satellite-data-issues-warning-about-forecast-reliability/?hpid=z5

 

 

From what I understand, model output is pretty sound within 24 hours of said outage, but could start to degrade slightly if the outage lasted beyond 00Z today (24 hours). 

 

 

 

Edit: I may be incorrect, but this link seems to indicate ECMWF received Satellite data today. Whether they received less than they usually do, I don't know. 

 

http://old.ecmwf.int/products/forecasts/d/charts/monitoring/coverage/dcover!Geostationary-CSR!12!pop!od!mixed!w_coverage!latest!/

[Scuddz]

BTW, CWG's article is all hype, little substance and contains misinformation. Hype it and the sheeple will come.

You should write your own

[ohleary]

You should write your own

 

Show business isn't my bag.  Though if I had written the article, I'd have had the facts, unlike some weather blogger.

[ohleary]

Just reporting what I read, and out of fairness, per the Washington Post article:

 

"Chris Vaccaro, a spokesperson for the National Weather Service, said officials at NCEP told him these model forecasts can still be considered credible, despite the outage. “[T]here’s a lot of redundancy in the observing system that can help to offset the data loss and that the model guidance will have integrity and will be sound,” he said in an email."

 

http://www.washingtonpost.com/blogs/capital-weather-gang/wp/2014/10/22/weather-service-stops-receiving-satellite-data-issues-warning-about-forecast-reliability/?hpid=z5

 

 

From what I understand, model output is pretty sound within 24 hours of said outage, but could start to degrade slightly if the outage lasted beyond 00Z today (24 hours). 

 

 

 

Edit: I may be incorrect, but this link seems to indicate ECMWF received Satellite data today. Whether they received less than they usually do, I don't know. 

 

http://old.ecmwf.int/products/forecasts/d/charts/monitoring/coverage/dcover!Geostationary-CSR!12!pop!od!mixed!w_coverage!latest!/

 

Yeah...check out their AIRS coverage for 12Z today:

 

http://old.ecmwf.int/products/forecasts/d/charts/monitoring/coverage/dcover!airs!12!pop!od!mixed!w_coverage!latest!/

 

Doh!

[DTWXRISK]

Ohleary... you   are  really  over reacting .  

I  read your   comments  in the CWG  feedback....

"The outage didn't stem from within the NWS."   ---  cwg   never said it did

"So boo hoo to all that NWS is falling apart hype as well. " ....Um  you do know that when Mr  Samenow  wrote that story is  is  arguing for   congress
spending MORE  money   on for NWS.. 
|
"And yes, NASA has data. NASA isn't NOAA."    --   I  dont   cwg  was  saying NASA is part of NOAA

[icebreaker5221]

Edit: I may be incorrect, but this link seems to indicate ECMWF received Satellite data today. Whether they received less than they usually do, I don't know. 

 

http://old.ecmwf.int/products/forecasts/d/charts/monitoring/coverage/dcover!Geostationary-CSR!12!pop!od!mixed!w_coverage!latest!/

 

All the AMVs were missing, so they received less data than normal.

[Scuddz]

Show business isn't my bag.  Though if I had written the article, I'd have had the facts, unlike some weather blogger.

You don't have the facts.

[ohleary]

You don't have the facts.

 

If you're referring to the satellite outage specifically, you're incorrect. If not, I don't know what you are talking about.

[MN Transplant]

A little light reading on the impact of satellite data on models (meat starts on pg 62):

 

http://www.ecmwf.int/sites/default/files/Seminar_NWP_impact.pdf

[millwx]

For all interested... while I don't know the source of the problem, having worked on this data directly (note that the GOES Soundings products are one of many mentioned as missing... I personally put these into Operations at NESDIS... not blowing my horn, I'm just trying to let you know that I'm fairly familiar with the operations there).  Given that I don't work there anymore AND don't know the precise problem, I can't speak with 100% certainty.  But I can give you some better idea than the b.s. I see floating around (like, up-thread, apparently AccuWeather stating that this has no impact on the ECMWF... nothing could be further from the truth).  So, let me point out a few things (again, I'm not 100%, absolutely certain of these, since things might've changed since I've been there; but, I'd put money on it that I'm at least in the ballpark on most of these items)...

 

1) Whatever failed is massive.  This never happens.  I've met my share of govt workers who meet the negative stereotypes that some anti-govt political types have.  But they're the exception.  And most sys admin types are VERY on the ball.  Even our R&D systems in NESDIS never went down for this long.  For a segment of NESDIS operations to crash for DAYS... this is a big deal.  Unheard of.

 

2) However... raw data is not impacted.  The satellite ingest comes in at Wallops and feeds to NESDIS.  Moreover, NESDIS' SSD server is up and functioning properly.  So, have no fear about permanent data loss.  For example, we've been bemoaning this failure over in the Eurasian snow cover thread, as the National Ice Center is one site that's down.  Well, have no fear.  This outage is VERY frustrating (believe me, I know... I'm one of few who focuses on mid-Oct snow... not snow increase across the whole month; so, for me, this is the most critical time... this outage is killing me!), but all the raw data is there and recoverable.  NatIce (and OSPO, whose SST site is also down) should be able to recover and reprocess all data once they're back up and running.

 

3) I'm not going to crucify AccuWeather, as I didn't see their comment firsthand.  But let me make one thing clear... this data feed failure will impact the ECMWF model MORE than the GFS, not less.  NCEP is concerned about the poor (relative to non-remotely sensed data) signal-to-noise ratio of satellite data and gives it low weight in the NWP models.  The ECMWF takes a different view... while they recognize the limitations of the data, they believe it to be better than nothing in data void regions (oceans).  They are MUCH more robust in utilizing satellite data... especially the derived products, which are failing to be delivered.  If anyone tells you the EC is not impacted, they don't know what they're talking about.  It is, in fact, impacted MORE than the NCEP models.

 

4) However, to clarify, I should possibly have said, WERE being impacted.  According to NWS notifications today, there is a backup, "prehistoric" feed now in place to NCEP (and probably to the EC).  There was also an emergency network switch replacement performed this morning, which I suspect may be related to the problem (though, I could be wrong, as the NatIce and OSPO sites are still down).  So, I THINK any impacts on NWP are in the past, and things are rolling forward.  That said, as I noted in the parenthesis, the NatIce and OSPO sites are still down.  So, clearly, the issues are not completely resolved.

 

5)  On that note, and referencing my first point on this being a pretty serious outage, I've not heard anyone mention a timeline for a return to service for the data feeds and web sites still off-line.  Hopefully, this will be fixed quickly... as I said, such a failure is completely unprecedented.  But don't hold your breath; NESDIS has some major problems they're trying to resolve right now.  Be patient.

[wxmeddler]

Thanks you for giving your insight into this issue. Your expertise and willingness to share is much appreciated! I saw the AccuWx article, I had a feeling AccuWx was lying through it's teeth (or at least not doing their fact checking) about this issue. Despite being a serious outage, NESDIS has done a great job in getting things back in order. Even if it's not as efficient as it was pre-event. Lessons learned.

[Cypress]

interesting to note the change in command at NESDIS effective Nov 2nd, they tapped a NASA guy:

 

http://www.noaanews.noaa.gov/stories2014/20140929_volz.html

 

NOAA selects new leader for its Satellite and Information Service

September 29, 2014

 

As assistant administrator, Volz will shepherd NOAA’s programs to build and launch the next generation of environmental satellites: the Joint Polar Satellite System (JPSS) and the Geostationary Operational Environmental Satellite R-Series (GOES-R), and other missions, including the Deep Space Climate Observatory, known as DSCOVR. He’ll also manage NOAA’s current spacecraft fleet and NESDIS’ vast climate, oceanographic and geophysical data operations. Data from these NESDIS assets are used throughout NOAA, across the federal family and around the world for operational weather forecasts and climate impact assessments.
 
In his current position at NASA, Volz manages all of the agency’s earth science flight missions and associated activities, including 17 satellites currently operating on orbit, 12 in formulation and development, and others in the early study and design stages.

[The_Global_Warmer]

Accuweather makes money off the Euro.

Pretty simple.

[Guest Imperator]

For all interested... while I don't know the source of the problem, having worked on this data directly (note that the GOES Soundings products are one of many mentioned as missing... I personally put these into Operations at NESDIS... not blowing my horn, I'm just trying to let you know that I'm fairly familiar with the operations there).  Given that I don't work there anymore AND don't know the precise problem, I can't speak with 100% certainty.  But I can give you some better idea than the b.s. I see floating around (like, up-thread, apparently AccuWeather stating that this has no impact on the ECMWF... nothing could be further from the truth).  So, let me point out a few things (again, I'm not 100%, absolutely certain of these, since things might've changed since I've been there; but, I'd put money on it that I'm at least in the ballpark on most of these items)...

 

1) Whatever failed is massive.  This never happens.  I've met my share of govt workers who meet the negative stereotypes that some anti-govt political types have.  But they're the exception.  And most sys admin types are VERY on the ball.  Even our R&D systems in NESDIS never went down for this long.  For a segment of NESDIS operations to crash for DAYS... this is a big deal.  Unheard of.

 

2) However... raw data is not impacted.  The satellite ingest comes in at Wallops and feeds to NESDIS.  Moreover, NESDIS' SSD server is up and functioning properly.  So, have no fear about permanent data loss.  For example, we've been bemoaning this failure over in the Eurasian snow cover thread, as the National Ice Center is one site that's down.  Well, have no fear.  This outage is VERY frustrating (believe me, I know... I'm one of few who focuses on mid-Oct snow... not snow increase across the whole month; so, for me, this is the most critical time... this outage is killing me!), but all the raw data is there and recoverable.  NatIce (and OSPO, whose SST site is also down) should be able to recover and reprocess all data once they're back up and running.

 

3) I'm not going to crucify AccuWeather, as I didn't see their comment firsthand.  But let me make one thing clear... this data feed failure will impact the ECMWF model MORE than the GFS, not less.  NCEP is concerned about the poor (relative to non-remotely sensed data) signal-to-noise ratio of satellite data and gives it low weight in the NWP models.  The ECMWF takes a different view... while they recognize the limitations of the data, they believe it to be better than nothing in data void regions (oceans).  They are MUCH more robust in utilizing satellite data... especially the derived products, which are failing to be delivered.  If anyone tells you the EC is not impacted, they don't know what they're talking about.  It is, in fact, impacted MORE than the NCEP models.

 

4) However, to clarify, I should possibly have said, WERE being impacted.  According to NWS notifications today, there is a backup, "prehistoric" feed now in place to NCEP (and probably to the EC).  There was also an emergency network switch replacement performed this morning, which I suspect may be related to the problem (though, I could be wrong, as the NatIce and OSPO sites are still down).  So, I THINK any impacts on NWP are in the past, and things are rolling forward.  That said, as I noted in the parenthesis, the NatIce and OSPO sites are still down.  So, clearly, the issues are not completely resolved.

 

5)  On that note, and referencing my first point on this being a pretty serious outage, I've not heard anyone mention a timeline for a return to service for the data feeds and web sites still off-line.  Hopefully, this will be fixed quickly... as I said, such a failure is completely unprecedented.  But don't hold your breath; NESDIS has some major problems they're trying to resolve right now.  Be patient.

 

 

Thank you for explaining this thoroughly! 

[dtk]

3) I'm not going to crucify AccuWeather, as I didn't see their comment firsthand.  But let me make one thing clear... this data feed failure will impact the ECMWF model MORE than the GFS, not less.  NCEP is concerned about the poor (relative to non-remotely sensed data) signal-to-noise ratio of satellite data and gives it low weight in the NWP models.  The ECMWF takes a different view... while they recognize the limitations of the data, they believe it to be better than nothing in data void regions (oceans).  They are MUCH more robust in utilizing satellite data... especially the derived products, which are failing to be delivered.  If anyone tells you the EC is not impacted, they don't know what they're talking about.  It is, in fact, impacted MORE than the NCEP models.

 

This is very likely untrue.  Both systems will be impacted by the outage, but unless it is prolonged, it should be quite minimal.  The global observing system is quite robust with a ton of redundant information.  Both centers rely on satellite data quite heavily, not just ECMWF.  In fact, OSE and FSO experiments with both systems reveal that satellite based observations (in particular MW sounder/AMSU/ATMS and hyper spectral infrared sounder/AIRS/IASI) are the biggest contributors to reducing forecast error.  

 

The Europeans are different in that they are a bit more advanced in terms of using cloud and precipitation affected radiances but the US is aggressively working on this and catching up.  I don't understand your point about "derived products' since most operational centers use the level 1b radiances for assimilation, not retrievals....except for satellite derived AMVs mostly from geostationary satellites.  There are some retrievals that are used in offline mode to assist in QC or for generating lower boundary conditions (land states, SST, ice, etc.), but the atmospheric data assimilation systems use radiances (in the US and abroad).  It's not fair to poo-poo satellite assimilation in the US, given that they have led the way with the original implementation of radiance assimilation and development of a variational bias correction scheme.  Your characterization of how NCEP views satellite data versus ECMWF is flat out false.

 

Furthermore, all of the operational centers now use some variant of dynamic ensemble to help prescribe error background error statistics.  This makes the operational data assimilation systems more robust, since they can adjust automatically to changes in the observing system.  So, in summary, there will likely be some (small) pain, but felt equally across the operational centers.

 

I don't have time to put together a prolonged, proper post on the subject.  However,  I would be happy to provide references, literature, and examples if you're interested.

[dtk]

A little light reading on the impact of satellite data on models (meat starts on pg 62):

 

http://www.ecmwf.int/sites/default/files/Seminar_NWP_impact.pdf

 

Just to note that when interpreting FSO and OSE results, one has to realize that the statistics on impacts are derived for fairly long periods (usually at least a month or more).  Data assimilation systems have "memory" and ability to dynamically adjust since they are incremental.....using a model to continually propagate forward in time.  What I am trying to say is that short, episodic losses in some parts of the observing system are not likely to have a huge impact given that the global observing system is robust and redundant.

 

One area where it could hurt is in the cycling and updating of satellite bias correction coefficients in the variational, dynamic schemes.

[millwx]

This is very likely untrue.  Both systems will be impacted by the outage, but unless it is prolonged, it should be quite minimal.  The global observing system is quite robust with a ton of redundant information.  Both centers rely on satellite data quite heavily, not just ECMWF.  In fact, OSE and FSO experiments with both systems reveal that satellite based observations (in particular MW sounder/AMSU/ATMS and hyper spectral infrared sounder/AIRS/IASI) are the biggest contributors to reducing forecast error.  

 

The Europeans are different in that they are a bit more advanced in terms of using cloud and precipitation affected radiances but the US is aggressively working on this and catching up.  I don't understand your point about "derived products' since most operational centers use the level 1b radiances for assimilation, not retrievals....except for satellite derived AMVs mostly from geostationary satellites.  There are some retrievals that are used in offline mode to assist in QC or for generating lower boundary conditions (land states, SST, ice, etc.), but the atmospheric data assimilation systems use radiances (in the US and abroad).  It's not fair to poo-poo satellite assimilation in the US, given that they have led the way with the original implementation of radiance assimilation and development of a variational bias correction scheme.  Your characterization of how NCEP views satellite data versus ECMWF is flat out false.

 

Furthermore, all of the operational centers now use some variant of dynamic ensemble to help prescribe error background error statistics.  This makes the operational data assimilation systems more robust, since they can adjust automatically to changes in the observing system.  So, in summary, there will likely be some (small) pain, but felt equally across the operational centers.

 

I don't have time to put together a prolonged, proper post on the subject.  However,  I would be happy to provide references, literature, and examples if you're interested.

 

If my characterization is flat out false then the view at NCEP has changed markedly in the past few years.  I've worked directly on this, working with the JCSDA team.  I've got first-hand knowledge of this.  And, yes, they use (or used) retrievals (God, I still have nightmares over the horrendous BUFR encoding so the retrievals could be delivered to EMC and NCDC... EMC absolutely assimilated GOES satellite retrievals into the GFS).  I do know there was a move to get away from this (and, at the time, EMC downweighted the retrievals severely).  As such, the direct radiance assimilation may have since replaced the retrievals.  So, I will grant you that my assessment may be outdated some.  But they absolutely did assimilate retrievals for quite some time (many years)... though, again, perhaps not anymore.  There are also other derived products they were/are assimilating... like the satellite derived winds; and I would imagine those are still being utilized and assimilated.

 

Nothing I characterized was incorrect (emphasis on "was", because I suppose it could've changed in the past few years).  Moreover, I don't know who's "poo pooing" what, to use your colorful terminology.    You're being too defensive.  My characterization of NCEP's more conservative approach to utilization of the derived satellite products was NOT criticism.  I can see both points of view on the matter and, frankly, would probably side more with NCEP than the EC.  The shortcomings in the derived products are... well, don't get me started (I don't want to "poo poo" my own past work... though it was the best we could do with what we had to work with).  In fact, I also worked on the GOES-R initial spec team (both from a govt and contractor side) and was disappointed that even THOSE advancement still fail to get the observations where I'd like them to be.  So, actually, I commend NCEP for moving so swiftly towards direct radiance assimilation.  I don't think the assimilation of derived products is really the way to go (even the winds can be dubious - though I can understand at least making an effort to use those).  I think you misread my philosophical assessment (which absolutely was not incorrect... I dealt with it on a first hand basis, every single day)... it was not criticism.  You're being too defensive.  I have no problem with the path NCEP took on this.  I'm privvy to some of the political maneuvering a select few individuals there undertook along the way, which I wasn't so keen on.  But, in the end, their approach was the correct one, in my opinion.

 

One thing I will grant you... my comments may be outdated in terms of the model impact/usage.  But, if so, as you noted, the usage of satellite data has only INCREASED.  And, so, your comments only further reinforce the initial point of this argument... unless the EC is on a different data feed and has been receiving the data all along (which I'm sure is not the case), then, contrary to what AccuWeather stated, the ECMWF is, in fact, impacted by the data loss.  What I may be wrong about is that it's impacted "more".  Since I've been out of the loop for a while, I'll defer to you on that and thank you for correcting me on it.  But again, "more", "less", "the same"... no matter... the point was simple... the EC will be impacted by the data loss.

 

Oh, and for the record, I also completely agree with you that the impact should be minimal given the redundancies in the system.  My point wasn't that the EC would be massively degraded by this.  No way.  I was chiming in because an assertion was made that:  The GFS will be impacted negatively, but the ECMWF will not be.  On that, I'm pretty sure you and would agree 100% that this statement is incorrect.  Impacts may be trivial... but said impacts will occur in both models.

[dtk]

If my characterization is flat out false then the view at NCEP has changed markedly in the past few years.  I've worked directly on this, working with the JCSDA team.  I've got first-hand knowledge of this.  And, yes, they use (or used) retrievals (God, I still have nightmares over the horrendous BUFR encoding so the retrievals could be delivered to EMC and NCDC... EMC absolutely assimilated GOES satellite retrievals into the GFS).  I do know there was a move to get away from this (and, at the time, EMC downweighted the retrievals severely).  As such, the direct radiance assimilation may have since replaced the retrievals.  So, I will grant you that my assessment may be outdated some.  But they absolutely did assimilate retrievals for quite some time (many years)... though, again, perhaps not anymore.  There are also other derived products they were/are assimilating... like the satellite derived winds; and I would imagine those are still being utilized and assimilated.

 

Nothing I characterized was incorrect (emphasis on "was", because I suppose it could've changed in the past few years).  Moreover, I don't know who's "poo pooing" what, to use your colorful terminology.    You're being too defensive.  My characterization of NCEP's more conservative approach to utilization of the derived satellite products was NOT criticism.  I can see both points of view on the matter and, frankly, would probably side more with NCEP than the EC.  The shortcomings in the derived products are... well, don't get me started (I don't want to "poo poo" my own past work... though it was the best we could do with what we had to work with).  In fact, I also worked on the GOES-R initial spec team (both from a govt and contractor side) and was disappointed that even THOSE advancement still fail to get the observations where I'd like them to be.  So, actually, I commend NCEP for moving so swiftly towards direct radiance assimilation.  I don't think the assimilation of derived products is really the way to go (even the winds can be dubious - though I can understand at least making an effort to use those).  I think you misread my philosophical assessment (which absolutely was not incorrect... I dealt with it on a first hand basis, every single day)... it was not criticism.  You're being too defensive.  I have no problem with the path NCEP took on this.  I'm privvy to some of the political maneuvering a select few individuals there undertook along the way, which I wasn't so keen on.  But, in the end, their approach was the correct one, in my opinion.

 

One thing I will grant you... my comments may be outdated in terms of the model impact/usage.  But, if so, as you noted, the usage of satellite data has only INCREASED.  And, so, your comments only further reinforce the initial point of this argument... unless the EC is on a different data feed and has been receiving the data all along (which I'm sure is not the case), then, contrary to what AccuWeather stated, the ECMWF is, in fact, impacted by the data loss.  What I may be wrong about is that it's impacted "more".  Since I've been out of the loop for a while, I'll defer to you on that and thank you for correcting me on it.  But again, "more", "less", "the same"... no matter... the point was simple... the EC will be impacted by the data loss.

 

Oh, and for the record, I also completely agree with you that the impact should be minimal given the redundancies in the system.  My point wasn't that the EC would be massively degraded by this.  No way.  I was chiming in because an assertion was made that:  The GFS will be impacted negatively, but the ECMWF will not be.  On that, I'm pretty sure you and would agree 100% that this statement is incorrect.  Impacts may be trivial... but said impacts will occur in both models.

 

"EMC absolutely assimilated GOES satellite retrievals into the GFS".  This is (mostly) untrue.  Full disclosure:  I used to work at NCEP/EMC but no longer do.  I worked on the data assimilation team in the global branch.  I was a core developer of the atmospheric data assimilation system that is operational.  The only retrievals currently assimilated are AMVs, a derivative of space-based gps signal (though it is close to the raw measurement), and SBUV ozone retrievals.  Having said that, there are some retrievals of temperature and moisture being used in the NCEP legacy system, the old CDAS (i.e. the continuation of the NCEP/NCAR reanalysis which is used of monitoring).  This is completely outside of the GFS/GDAS and other operational systems.  Even the CFSR (and continuation thereof) assimilates radiances/brightness temperatures instead of retrievals.

 

There are other satellite based retrievals that are used to help prescribe boundary conditions (SST, sea ice, land states, etc.), but that is outside of the core model initialization and likely of second order.  Is this what you are referring to?

 

I'm sorry if I seemed defensive, but you flat out said that NCEP didn't trust satellite based data so intentionally underweighted them whereas ECMWF decided to use them (more robustly) over oceans instead of nothing.  This is a gross mischaracterization of attitudes, approaches taken, and implementations made at NCEP.  The operational centers have embraced the assimilation of more direct measurements from satellites long ago.  NCEP was a pioneer in this regard (there is a paper by Wu and Derber on the assimilation of ATOVS radiances from 1998 on this).  They started using radiances in 1995.  The only push to assimilate retrievals are from the producers of said retrievals or the observational science teams (in the US, this comes from folks within NESDIS/JCSDA and perhaps NASA), or perhaps some folks within academia since retrievals are easier (they map more directly onto model state space).

 

This is a good discussion and I'm happy to continue it.  

 

In terms of the broader point, we obviously agree that 1) impact should be minimal (though nontrivial), and 2) impact would be felt (quasi) equally across centers.  The statement from Accuweather is a joke.

[millwx]

  The only push to assimilate retrievals are from the producers of said retrievals or the observational science teams (in the US, this comes from folks within NESDIS/JCSDA and perhaps NASA), or perhaps some folks within academia since retrievals are easier (they map more directly onto model state space).

 

Hahahahaha!  You gave me a laugh there.  This is soooo very true.  Been there.  Done that.  Names will not be mentioned.  LOL.  But I know where you are coming from.

 

We agree more than we disagree.  And perhaps I'm dating myself, because perhaps it was too early in the process, or perhaps I was getting filtered information, but to my knowledge, the GOES Atmospheric Temperature and Moisture Soundings were, indeed, being assimilated into the GFS (whether via the GDAS or some legacy system, I don't recall).  Hell, I worked with EMC - limitedly - on this!  (And I recall our concerns on the science end... the incestuous nature of using the GFS as a "first-guess" for the Soundings - because the Soundings themselves are going into the GFS.)  Moreover, in so doing I know that NESDIS personnel (and perhaps this is where the "filtered" information comes from) were in a mini-furor over the fact that EMC was effectively down-weighting the Soundings, within the assimilation, to near zero; the EC, meanwhile, was far more aggressive in using them.  If that is incorrect, then my apologies... but I can tell you that it isn't "my opinion"; it was widespread, common "knowledge" within an entire agency.  Not trying to re-energize that furor here.  It doesn't matter.  A lot of those folks are gone now, and cooperation is much better with the JCSDA in place.  Moreover, by now I'm sure the entire assimilation process has evolved considerably.  This issue - again, dating myself here - was especially prominent in the early days, pre-JCSDA.  Frankly, I trust your judgment on it, since you worked even more directly on this than I did.  So, again, I'm not re-asserting that opinion as an argument.  Rather, I trust your statements... I'm just explaining to you that this "downweighting" issue was a simple fact inside of some circles in NESDIS.

 

FWIW, I'm definitely NOT trying to rile you up here.  If your three letter ID on here is telling, I might know who I'm talking to, hehehe.  ;-)  If so, we've actually worked together (peripherally) and I have the utmost respect for you (excellent scientist/programmer & very nice person).  And if you aren't who I suspect, well, it doesn't matter... I'm not trying to argue or rile you up anyway.  If there's anything I've said that you find utterly disagreeable, well, it's unfortunate because it's representative of a broader misunderstanding between the communities working on this.  Because everything I've said is (I should say, "was" - that's important) utterly indisputable in the eyes of NESDIS.  Was... like I said... I think they're long since past that now, thankfully.  So, I'm not trying to stir the pot... just explaining from where those comments/thought originate.  My only ongoing confusion is the GOES Soundings assimilation into the GFS.  I'm certain they were (perhaps not anymore) going in there, while you seem certain they weren't (at least  if I'm reading you right).  And I think we're both well versed on this.  So, that's a head scratcher to me.  Otherwise, I think we're on the same page here.  

 

Oh, and, yes, I'm aware of the long history of the radiance assimilation.  I also worked in the ATOVS arena (though less so than GOES), and I was there back in the timeframe you mentioned (started in the mid-90s).  I know NCEP was always aggressive in pursuing THAT avenue... AND, as I tried to clarify in my follow-up, I believe that was the correct approach.  My original statement was not one of criticism.  Even though I came from the derived products side, I always felt the direct radiance assimilation was so much "cleaner".  The only thing you can't really do with it (I don't think... though I could imagine some bass-ackwards way that the wheel could essentially be reinvented within the system) is the winds.  Beyond winds, radiances seem the wise way to go.  I might get cut down by my colleagues there for saying so, but it's true.

[dtk]

Hahahahaha!  You gave me a laugh there.  This is soooo very true.  Been there.  Done that.  Names will not be mentioned.  LOL.  But I know where you are coming from.

 

We agree more than we disagree.  And perhaps I'm dating myself, because perhaps it was too early in the process, or perhaps I was getting filtered information, but to my knowledge, the GOES Atmospheric Temperature and Moisture Soundings were, indeed, being assimilated into the GFS (whether via the GDAS or some legacy system, I don't recall).  Hell, I worked with EMC - limitedly - on this!  (And I recall our concerns on the science end... the incestuous nature of using the GFS as a "first-guess" for the Soundings - because the Soundings themselves are going into the GFS.)  Moreover, in so doing I know that NESDIS personnel (and perhaps this is where the "filtered" information comes from) were in a mini-furor over the fact that EMC was effectively down-weighting the Soundings, within the assimilation, to near zero; the EC, meanwhile, was far more aggressive in using them.  If that is incorrect, then my apologies... but I can tell you that it isn't "my opinion"; it was widespread, common "knowledge" within an entire agency.  Not trying to re-energize that furor here.  It doesn't matter.  A lot of those folks are gone now, and cooperation is much better with the JCSDA in place.  Moreover, by now I'm sure the entire assimilation process has evolved considerably.  This issue - again, dating myself here - was especially prominent in the early days, pre-JCSDA.  Frankly, I trust your judgment on it, since you worked even more directly on this than I did.  So, again, I'm not re-asserting that opinion as an argument.  Rather, I trust your statements... I'm just explaining to you that this "downweighting" issue was a simple fact inside of some circles in NESDIS.

 

FWIW, I'm definitely NOT trying to rile you up here.  If your three letter ID on here is telling, I might know who I'm talking to, hehehe.  ;-)  If so, we've actually worked together (peripherally) and I have the utmost respect for you (excellent scientist/programmer & very nice person).  And if you aren't who I suspect, well, it doesn't matter... I'm not trying to argue or rile you up anyway.  If there's anything I've said that you find utterly disagreeable, well, it's unfortunate because it's representative of a broader misunderstanding between the communities working on this.  Because everything I've said is (I should say, "was" - that's important) utterly indisputable in the eyes of NESDIS.  Was... like I said... I think they're long since past that now, thankfully.  So, I'm not trying to stir the pot... just explaining from where those comments/thought originate.  My only ongoing confusion is the GOES Soundings assimilation into the GFS.  I'm certain they were (perhaps not anymore) going in there, while you seem certain they weren't (at least  if I'm reading you right).  And I think we're both well versed on this.  So, that's a head scratcher to me.  Otherwise, I think we're on the same page here.  

 

Oh, and, yes, I'm aware of the long history of the radiance assimilation.  I also worked in the ATOVS arena (though less so than GOES), and I was there back in the timeframe you mentioned (started in the mid-90s).  I know NCEP was always aggressive in pursuing THAT avenue... AND, as I tried to clarify in my follow-up, I believe that was the correct approach.  My original statement was not one of criticism.  Even though I came from the derived products side, I always felt the direct radiance assimilation was so much "cleaner".  The only thing you can't really do with it (I don't think... though I could imagine some bass-ackwards way that the wheel could essentially be reinvented within the system) is the winds.  Beyond winds, radiances seem the wise way to go.  I might get cut down by my colleagues there for saying so, but it's true.

 

I'm not offended or anything, I just read too much into your original assertion.  In fact, it's pretty easy to figure out who I am and I'm sure you've gotten it right.  I work with everyone (at least I tried to during my time there), so it will come as no surprise if we had crossed paths.  Thanks for the kind words

 

In terms of the GOES temperature and humidity retrievals, I don't recall exactly when the switch occurred, but NCEP has been using the sounder radiances (directly) for a while now.  In fact, I was the one that did some of the original testing when we went from the averaged observations to those directly from the full field of view (and separated each of the four detectors).  That was probably around 2007 or so?  Before then, some 5x5 field-of-view averaged subset was being used.  The retrievals were still being generated and sent to EMC because those are used in the old CDAS for monitoring.

 

The comments regarding downweighting is interesting and that wouldn't surprise me in the least.  I was never involved in that kind of activity nor decision making, but I heard rumblings of stuff *like* that.  It is entirely possible that I was just sheltered or kept in the dark about it, or this kind of stuff went on prior to my arrival.  I know some of the personalities involved and I really never understood the whole thing.  I mean, the organizations are stretched thin enough the way it is and that kind of stuff drove me nuts. 

 

In terms of winds, there are technologies that people are working on it to do that sort of thing with satellite imagery.  You don't compute "winds" from the images/radiances/etc., but instead try to assimilate features/images directly.  It's almost like putting the AMV processing directly into the assimilation (but a bit more complex than that).  I'm sure that my friends in Wisconsin would not be thrilled to know that I would be in favor of doing feature/image assimilation directly if/when the technology is ready.  It is almost always best to try and assimilate as close as possible the actual measurement.  We also have space-based wind lidar to look forward to.  I hope that it can deliver even a fraction of what we are anticipating. 

[ohleary]

For all interested... while I don't know the source of the problem, having worked on this data directly (note that the GOES Soundings products are one of many mentioned as missing... I personally put these into Operations at NESDIS... not blowing my horn, I'm just trying to let you know that I'm fairly familiar with the operations there).  Given that I don't work there anymore AND don't know the precise problem, I can't speak with 100% certainty.  But I can give you some better idea than the b.s. I see floating around (like, up-thread, apparently AccuWeather stating that this has no impact on the ECMWF... nothing could be further from the truth).  So, let me point out a few things (again, I'm not 100%, absolutely certain of these, since things might've changed since I've been there; but, I'd put money on it that I'm at least in the ballpark on most of these items)...

 

1) Whatever failed is massive.  This never happens.  I've met my share of govt workers who meet the negative stereotypes that some anti-govt political types have.  But they're the exception.  And most sys admin types are VERY on the ball.  Even our R&D systems in NESDIS never went down for this long.  For a segment of NESDIS operations to crash for DAYS... this is a big deal.  Unheard of.

 

It wasn't a crash, but it was a very unfortunate impactive event.  That's all I want to say about that.

 

2) However... raw data is not impacted.  The satellite ingest comes in at Wallops and feeds to NESDIS.  Moreover, NESDIS' SSD server is up and functioning properly.  So, have no fear about permanent data loss.  For example, we've been bemoaning this failure over in the Eurasian snow cover thread, as the National Ice Center is one site that's down.  Well, have no fear.  This outage is VERY frustrating (believe me, I know... I'm one of few who focuses on mid-Oct snow... not snow increase across the whole month; so, for me, this is the most critical time... this outage is killing me!), but all the raw data is there and recoverable.  NatIce (and OSPO, whose SST site is also down) should be able to recover and reprocess all data once they're back up and running.

 

3) I'm not going to crucify AccuWeather, as I didn't see their comment firsthand.  But let me make one thing clear... this data feed failure will impact the ECMWF model MORE than the GFS, not less.  NCEP is concerned about the poor (relative to non-remotely sensed data) signal-to-noise ratio of satellite data and gives it low weight in the NWP models.  The ECMWF takes a different view... while they recognize the limitations of the data, they believe it to be better than nothing in data void regions (oceans).  They are MUCH more robust in utilizing satellite data... especially the derived products, which are failing to be delivered.  If anyone tells you the EC is not impacted, they don't know what they're talking about.  It is, in fact, impacted MORE than the NCEP models.

 

I won't comment on the actual model functionality, but data-wise, the EC and CMC and others possibly lost more data than NCEP, as NCEP has a direct link/circuit to NSOF that was unaffected.  The outage to NCEP wasn't due directly to the NESDIS event, as when it happened at 20Z on the 20th, we lost a few types but were mostly fine.  Then 23 hours later, we lost connectivity to two of their systems, a side effect from the major outage affecting everyone else.   It was a tricky issue and took a day to troubleshoot and correct by NESDIS system/network engineers.

 

4) However, to clarify, I should possibly have said, WERE being impacted.  According to NWS notifications today, there is a backup, "prehistoric" feed now in place to NCEP (and probably to the EC).  There was also an emergency network switch replacement performed this morning, which I suspect may be related to the problem (though, I could be wrong, as the NatIce and OSPO sites are still down).  So, I THINK any impacts on NWP are in the past, and things are rolling forward.  That said, as I noted in the parenthesis, the NatIce and OSPO sites are still down.  So, clearly, the issues are not completely resolved.

 

Things at NCEP are pretty much back to normal, and have been since around 00Z on 10/23, so I'm not sure what you mean about a prehistoric feed in place. What notifications did you see? 

 

5)  On that note, and referencing my first point on this being a pretty serious outage, I've not heard anyone mention a timeline for a return to service for the data feeds and web sites still off-line.  Hopefully, this will be fixed quickly... as I said, such a failure is completely unprecedented.  But don't hold your breath; NESDIS has some major problems they're trying to resolve right now.  Be patient.

 

As a side note, NCEP makes public it's model dump counts, hourly, and per model/cycle:

 

http://www.nco.ncep.noaa.gov/pmb/nwprod/realtime/