Posted on Fri, 10 Mar 2017 by Weatherbit
Historical Weather Data is not always easy to find. The data that is easy to find, tends to be non-quality controlled or tough to parse. How does Weatherbit historical weather API build it's extensive historical weather data set? Find out here!
The NOAA Integrated Surface Database is probably the most well known, and complete data source for historical hourly observations in existence. However, they only allow access to their data via FTP, and the data is in a compressed format that makes it hard to quickly answer all of our tough historical weather questions.
The NOAA ISD database includes over 35,000 stations worldwide, with some having data as far back as 1901, though the data show a substantial increase in volume in the 1940s and again in the early 1970s. Currently, there are over 14,000 "active" stations updated daily in the database. The total uncompressed data volume is around 600 gigabytes; however, it continues to grow as more data are added. ISD includes numerous parameters such as wind speed and direction, wind gust, temperature, dew point, cloud data, sea level pressure, altimeter setting, station pressure, present weather, visibility, precipitation amounts for various time periods, snow depth, and various other elements as observed by each station.
There are plenty of historical weather apis on the market, but there are few that offer low price, high quality, extensive records, and speed of access into one package.
How does Weatherbit do it? It's simple! (or so we like to think)
All of our historical data is quality controlled to checked for outliers. For example, usually 3 standard deviations from the mean for historical temperature data observations is considered to be an out-lier. However, precipitation data is a little different, and tends to be much more noisy. We do not apply these standard deviation checks to precipitation. Instead we cross reference precipitation with other data sources like satellite, and Multisensor Precipitation Estimates.
Missing observations are one of the toughest aspects of historical weather data to tackle. We account for missing data by filling small gaps in temperature, humidity, wind speed, and wind direction by using a weighted interpolation based on the most recent observations.
Additional gaps are filled using observations from the nearest 5 stations not exceeding 40km away from the source. Finally, we use less authoritative sources like the Climate Forecast System Reanalysis to fill in any remaining large gaps.
These methods allow us to achieve about a 97% hourly historical weather data coverage. That is, our historical weather data api provides access to historical temperature data, historical precipitation data, and others with only about 3% missing hourly observations.
We are constantly looking to improve our data quality, and expose new historical weather data via our historical weather api. We are always open to any feedback, or suggestions.!.
P.S. - Be sure to checkout our new free historical weather data browser!