bluewave Posted July 31, 2022 Author Share Posted July 31, 2022 7 hours ago, Rjay said: Nope. Ewr is still wrong I am going to repost this from our discussion last week so we can find it for easy future reference near the end of the July thread. I like putting stuff near the beginning or end of a thread since it can be hard to find again when it’s in the middle. So we can show this to people when they want to post data from that site. While it took some time to find, this paper explains why the MADIS range test that the Gladstone site uses is so flawed. NYC is a chronically bad type of site mentioned below. The trees covering the ASOS has been an ongoing issue since the 1990s.http://www.thinkmind.org/articles/geoprocessing_2019_1_10_30010.pdf Range tests are not perfect. The record high United States temperature would fail MADIS’s range test, although it would pass MesoW- est’s test. Both MADIS and MesoWest further employ a suite of tests that go beyond their simple range tests. “Bud- dy” tests compare an observation to neighboring observa- tions. MADIS uses Optimal Interpolation in conjunction with cross-validation to measure the conformity of an ob- servation to its neighbors [2]. MesoWest estimates observa- tions using multivariate linear regression [5]. A real obser- vation is compared to the estimate, and if the deviation is high, then the real observation is flagged as questionable. These approaches are flawed in that they do not account for bad metadata, such as incorrect timestamps or incorrect locations. They do not account for chronically bad sites which produce bad data including data that may sometimes appear correct. Of even greater concern, they may not do a good job in assessing accuracy and may be incorrectly la- beling bad data as good and good data as bad. The consequences of ignoring data quality are great. How can we trust our applications and models if the inputs are bad? In turn, how can we better assess data for quality so that we can be confident in its use? In this paper, we present new evaluation results for our previously-published method including evaluation with several new data sets. These results are significant in that they demonstrate the challenges of evaluation of methods for data quality assessment of spatio-temporal weather sen- sor data. The rest of this paper is organized as follows: Sec- tion II presents relevant literature, Section III identifies general challenges, Section IV defines our approach, 1 1 Link to comment Share on other sites More sharing options...
CIK62 Posted July 31, 2022 Share Posted July 31, 2022 The last day of the month is averaging 79degs.(72/86). Month to date is 79.6[+2.1]. July will end at 79.6[+2.1] Reached 91 here yesterday. Today: 82-86, wind variable---e., s., increasing clouds, rain overnight?, 71 tomorrow AM. 75*(55%RH) here at 7am. 80* at 10am. 81* at 3pm. 82* at 4pm. Link to comment Share on other sites More sharing options...
wthrmn654 Posted July 31, 2022 Share Posted July 31, 2022 8 hours ago, Rjay said: Nope. Ewr is still wrong Though not always accurate, it essentially only showing it being about 1 degrees warmer during day and night. You would think it would be much higher. Link to comment Share on other sites More sharing options...
uncle W Posted July 31, 2022 Share Posted July 31, 2022 none of this would be a problem if thermometers weren't electronic... Link to comment Share on other sites More sharing options...
Cfa Posted July 31, 2022 Share Posted July 31, 2022 It’s too bad today won’t reach 90, I could’ve had my second heat wave. Hit 91 yesterday. Low of 62 this morning. Too cool for me. Link to comment Share on other sites More sharing options...
bluewave Posted July 31, 2022 Author Share Posted July 31, 2022 14 minutes ago, uncle W said: none of this would be a problem if thermometers weren't electronic... The electronic thermometers introduced a very small cool bias for maximum temperatures. https://www.ncei.noaa.gov/pub/data/ushcn/papers/menne-etal2010.pdf associated with the widespread conversion to electronic sensors in the USHCN during the last 25 years. However, the average of all unadjusted MMTS transitions is about −0.1°C for maximum temperature series and about +0.025°C for minimum temperature series. The adjustments for the impact of the MMTS on maximum tem- perature series in the USHCN version 2 data set are therefore somewhat inadequate, as reflected in Figures 2g and 3g. In fact, contrary to there being a positive (warm) bias as might be suggested by the exposure conditions at MMTS sites, there appears to be a residual, artificial negative bias in adjusted maximum temperatures (and little to no residual bias in adjusted minimum temperatures) Link to comment Share on other sites More sharing options...
wthrmn654 Posted July 31, 2022 Share Posted July 31, 2022 8 hours ago, Rjay said: Nope. Ewr is still wrong 1 hour ago, bluewave said: I am going to repost this from our discussion last week so we can find it for easy future reference near the end of the July thread. I like putting stuff near the beginning or end of a thread since it can be hard to find again when it’s in the middle. So we can show this to people when they want to post data from that site. While it took some time to find, this paper explains why the MADIS range test that the Gladstone site uses is so flawed. NYC is a chronically bad type of site mentioned below. The trees covering the ASOS has been an ongoing issue since the 1990s.http://www.thinkmind.org/articles/geoprocessing_2019_1_10_30010.pdf Range tests are not perfect. The record high United States temperature would fail MADIS’s range test, although it would pass MesoW- est’s test. Both MADIS and MesoWest further employ a suite of tests that go beyond their simple range tests. “Bud- dy” tests compare an observation to neighboring observa- tions. MADIS uses Optimal Interpolation in conjunction with cross-validation to measure the conformity of an ob- servation to its neighbors [2]. MesoWest estimates observa- tions using multivariate linear regression [5]. A real obser- vation is compared to the estimate, and if the deviation is high, then the real observation is flagged as questionable. These approaches are flawed in that they do not account for bad metadata, such as incorrect timestamps or incorrect locations. They do not account for chronically bad sites which produce bad data including data that may sometimes appear correct. Of even greater concern, they may not do a good job in assessing accuracy and may be incorrectly la- beling bad data as good and good data as bad. The consequences of ignoring data quality are great. How can we trust our applications and models if the inputs are bad? In turn, how can we better assess data for quality so that we can be confident in its use? In this paper, we present new evaluation results for our previously-published method including evaluation with several new data sets. These results are significant in that they demonstrate the challenges of evaluation of methods for data quality assessment of spatio-temporal weather sen- sor data. The rest of this paper is organized as follows: Sec- tion II presents relevant literature, Section III identifies general challenges, Section IV defines our approach, I'm not fully sure how to interpret this graph but it's the mean daytime analysis.. nighttime it's fairly fine ironically. When you do the day/ night comparison it drastically cuts the error down. But either way , the daytime error is pretty obvious. I wonder if anytime is reflecting light onto the sensor housing? Link to comment Share on other sites More sharing options...
wthrmn654 Posted July 31, 2022 Share Posted July 31, 2022 Anyone know by chance how far back this temperature debate may extend to? Like many years, only a couple, since it was first installed, etc? Link to comment Share on other sites More sharing options...
wthrmn654 Posted July 31, 2022 Share Posted July 31, 2022 Jeez, looking at the days where the it hit 100, it shows the analysis station didn't even fully record those days...... how can that be reliable?! Heck the analysis station being used is rarely doing full days... . Link to comment Share on other sites More sharing options...
wthrmn654 Posted July 31, 2022 Share Posted July 31, 2022 Also, has anyone tried contacting, [email protected] For issues with data quality, please contact the ASOS Operations Center Link to comment Share on other sites More sharing options...
wthrmn654 Posted July 31, 2022 Share Posted July 31, 2022 I'm not really sure what to think, right now I think the stations fine. Especially since it's been nearly spot on to analysis, other then peak time of a summer day...... Link to comment Share on other sites More sharing options...
donsutherland1 Posted July 31, 2022 Share Posted July 31, 2022 Morning thoughts… Sunshine will give way to increasing clouds. High temperatures will reach the middle and upper 80s in most of the region. Likely high temperatures around the region include: New York City (Central Park): 84° Newark: 89° Philadelphia: 89° Clouds and showers will give way to some sunshine tomorrow. It will be cooler, but hot air will return on Tuesday. Normals: New York City: 30-Year: 84.8°; 15-Year: 85.7° Newark: 30-Year: 86.4°; 15-Year: 87.8° Philadelphia: 30-Year: 87.4°; 15-Year: 88.2° Link to comment Share on other sites More sharing options...
Rtd208 Posted July 31, 2022 Share Posted July 31, 2022 High for the day yesterday was 92 here. Current temp 79/DP 62/RH 56% Link to comment Share on other sites More sharing options...
bluewave Posted July 31, 2022 Author Share Posted July 31, 2022 While the dewpoints weren’t as high this July as in recent years, the heat index was still very high due to the actual temperatures being near the warmest on record for July. Link to comment Share on other sites More sharing options...
forkyfork Posted July 31, 2022 Share Posted July 31, 2022 look how rare 80 degree julys used to be 1 Link to comment Share on other sites More sharing options...
doncat Posted July 31, 2022 Share Posted July 31, 2022 Little late, but had a 91° high yesterday. Link to comment Share on other sites More sharing options...
wthrmn654 Posted July 31, 2022 Share Posted July 31, 2022 1 hour ago, bluewave said: The electronic thermometers introduced a very small cool bias for maximum temperatures. https://www.ncei.noaa.gov/pub/data/ushcn/papers/menne-etal2010.pdf associated with the widespread conversion to electronic sensors in the USHCN during the last 25 years. However, the average of all unadjusted MMTS transitions is about −0.1°C for maximum temperature series and about +0.025°C for minimum temperature series. The adjustments for the impact of the MMTS on maximum tem- perature series in the USHCN version 2 data set are therefore somewhat inadequate, as reflected in Figures 2g and 3g. In fact, contrary to there being a positive (warm) bias as might be suggested by the exposure conditions at MMTS sites, there appears to be a residual, artificial negative bias in adjusted maximum temperatures (and little to no residual bias in adjusted minimum temperatures) 10 hours ago, Rjay said: Nope. Ewr is still wrong Jfk has been averaging out of range as well ironically. Link to comment Share on other sites More sharing options...
SACRUS Posted July 31, 2022 Share Posted July 31, 2022 Down to 62 and now 81. Much more clouds than yesterday and only clouds will get in the way of more 90s in the warm spots today. Link to comment Share on other sites More sharing options...
Rjay Posted July 31, 2022 Share Posted July 31, 2022 2 hours ago, wthrmn654 said: Jfk has been averaging out of range as well ironically. Ignore that site. Link to comment Share on other sites More sharing options...
SACRUS Posted July 31, 2022 Share Posted July 31, 2022 Some larger breaks in the clouds not between EPA and NJ, Temp up to 86 here. Link to comment Share on other sites More sharing options...
psv88 Posted July 31, 2022 Share Posted July 31, 2022 88 yesterday Link to comment Share on other sites More sharing options...
bluewave Posted July 31, 2022 Author Share Posted July 31, 2022 These charts do a good job showing how the tree growth began to cause a decline in NYC 90° days since the ASOS was established in 1995. The 1961-1990 and 1991-2020 climate normal periods are included. From 1961 to 1995 the Central Park ASOS wasn’t in the deep shade under the trees during peak heating hours. So the 90° day counts were more in line with Newark and LGA. The tree growth caused a decline in 90° days from near 20 in 1961 to around 16-17 in the early 2020s. If the site was properly maintained like other weather stations are, then NYC would have close closer to 30 days reaching 90° or higher like EWR and LGA. So the major media outlets forecasting high temperatures for NYC are all using flawed data. https://www.ncdc.noaa.gov/cdo-web/datasets/LCD/stations/WBAN:94728/detail ASOS 1995-11-01 Present OBSTRUCTIONS BEGIN DATE¹ END DATE¹ TREES 2021-04-06 Present TREES 2017-08-01 2021-04-06 TREE 020-340 deg 25-25 ft 20-54 deg 2016-06-15 2017-08-01 HYGR 090/4 TREES ENCIRLCE/25/20-54 IN FENCED ENCOLSURE 50 FT X 25 TH 1995-06-27 2016-06-15 https://mesonet.agron.iastate.edu/plotting/auto/?_wait=no&q=74&network=NYCLIMATE&station=NY5801&season=all&dir=above&var=high&threshold=90&year=1961&_r=t&dpi=100&_fmt=png 1 Link to comment Share on other sites More sharing options...
etudiant Posted July 31, 2022 Share Posted July 31, 2022 3 minutes ago, bluewave said: These charts do a good job showing how the tree growth began to cause a decline in NYC 90° days since the ASOS was established in 1995. The 1961-1990 and 1991-2020 climate normal periods are included. From 1961 to 1995 the Central Park ASOS wasn’t in the deep shade under the trees during peak heating hours. So the 90° day counts were more in line with Newark and LGA. The tree growth caused a decline in 90° days from near 20 in 1961 to around 16-17 in the early 2020s. If the site was properly maintained like other weather stations are, then NYC would have close to double the 90° day count at closer to 30 like EWR and LGA. So the major media outlets forecasting high temperatures for NYC are all using flawed data. https://mesonet.agron.iastate.edu/plotting/auto/?_wait=no&q=74&network=NYCLIMATE&station=NY5801&season=all&dir=above&var=high&threshold=90&year=1961&_r=t&dpi=100&_fmt=png Your charts do indeed tell a clear story. Would it help to widen the comparisons to include a couple of other Manhattan stations, the AMNH perhaps or the Greenwich Village station. That would neutralize the claim that EWR and LGA are reflecting increased heat island effects from airport expansion. Idk whether any other station has a comparably long history, but the results since 1980 should provide pretty clear indications of Up or Down. Link to comment Share on other sites More sharing options...
bluewave Posted July 31, 2022 Author Share Posted July 31, 2022 4 minutes ago, etudiant said: Your charts do indeed tell a clear story. Would it help to widen the comparisons to include a couple of other Manhattan stations, the AMNH perhaps or the Greenwich Village station. That would neutralize the claim that EWR and LGA are reflecting increased heat island effects from airport expansion. Idk whether any other station has a comparably long history, but the results since 1980 should provide pretty clear indications of Up or Down. The New Brunswick site did a good job of keeping the trees at the proper distance from the thermometer. It’s a more rural or suburban site. So the 90° count has steadily increased there since 1961 to near 30° like EWR and LGA. 1 Link to comment Share on other sites More sharing options...
qg_omega Posted July 31, 2022 Share Posted July 31, 2022 3 minutes ago, bluewave said: The New Brunswick site did a good job of keeping the trees at the proper distance from the thermometer. It’s a more rural or suburban site. So the 90° count has steadily increased there since 1961 to near 30° like EWR and LGA. That looks properly set Link to comment Share on other sites More sharing options...
SACRUS Posted July 31, 2022 Share Posted July 31, 2022 1PM Roundup New Brnswck: 89 PHL: 88 ACY: 86 TTN: 85 LGA: 85 TEB: 84 JFK: 84 EWR: 84 ISP: 83 BLM: 83 NYC: 82 1 Link to comment Share on other sites More sharing options...
SACRUS Posted July 31, 2022 Share Posted July 31, 2022 Just notched 90 locally here , sunny the past 35 mins. Link to comment Share on other sites More sharing options...
etudiant Posted July 31, 2022 Share Posted July 31, 2022 54 minutes ago, qg_omega said: That looks properly set Sure looks that way to me as well. Link to comment Share on other sites More sharing options...
Cfa Posted July 31, 2022 Share Posted July 31, 2022 7 hours ago, Cfa said: It’s too bad today won’t reach 90, I could’ve had my second heat wave. Hit 91 yesterday. Low of 62 this morning. Too cool for me. Spoke too soon. Link to comment Share on other sites More sharing options...
bluewave Posted July 31, 2022 Author Share Posted July 31, 2022 2 hours ago, qg_omega said: That looks properly set The freehold-Marlboro COOP site in Monmouth County has increased to around 30 days reaching 90° also. 2 Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now