How to load historical weather data using Python without scraping!

In How to Import Weather Data in MySQL , we demonstrated how to load weather forecast data into MySQL using a python script that was run on regular intervals. Historical weather data is often just as critical for data science applications so in this article we demonstrate how to load weather history data using Python. We will use the Visual Crossing Weather API that offers a free tier that includes past weather data.

Why shouldn’t we scrape weather data from our favorite web site?

Scraping weather data means we simply visit a web site and either manually or programmatically copy the data from that web page. Programmatic scraping of weather data can be difficult to implement and then difficult to maintain. When the web site changes (even for very small changes), the scraping code may need changing. Most importantly, scraping weather data is against the usage terms of almost all web sites.

If we are looking for a reliable solution to retrieving weather data on regular intervals we need a more reliable solution. Using a Weather API avoids us have to scrape data.

Skip to the source code.

Prerequisites

Before starting, you need access to Weather API we are going to use. If you don’t have account already, you can sign up for a free account at Visual Crossing Weather Services. Visual Crossing offers a perpetual free tier of up to 1000 results per day. if you need more, you can pay-per-result or sign up for a monthly plan.

Our Python script has been written in Python 3.8.2 but should work in most recent versions. We have kept the library requirements to a minimum. In this sample we are going to use the CSV format to download the data so we include a library to help process the data that is returned by the API.

Here’s the full list of import statements in our script


import csv
import codecs
import urllib.request
import sys

Setting up the weather data input parameters

The first part of our script sets up the parameters for the script to download the weather data and the second part retrieves the weather data. The weather data is retrieved using a RESTful weather API so we simply have to create a web query within the Python script and download the data.

The first part of the sets up some parameters to customize the weather data that is retrieved


# This is the core of our weather query URL
BaseURL = 'https://weather.visualcrossing.com/VisualCrossingWebServices/rest/services/weatherdata/'

# Set up the location parameter for our query
QueryLocation = '&location=' + urllib.parse.quote(sys.argv[1])

# Set up the query type parameter for our query ('FORECAST' or 'HISTORY')
QueryType=sys.argv[2].upper()

# Set up the key parameter for our query
QueryKey = '&key=' + sys.argv[3]

# Set up the date parameters for our query. Used only for historical weather data requests
FromDateParam = sys.argv[4]
ToDateParam = sys.argv[5]

In the above code, we set up 5 parameters. for the script.

The first parameter is the location for which the weather data should be downloaded. This is in form of an address, partial address or latitude and longitude value (for example 35.46,-75.12).

The code then asks for a query type (‘FORECAST’ or ‘HISTORY’) to choose between downloading weather forecast data or historical weather data.

We then ask for the API Key which is provided when signing up for the API. You can access it from the ‘Account’ location within Weather Data Services application.

Finally, if this is a historical weather data request, the code requests a start and end date in the form YYYY-MM-DD, for example 2020-03-26 is the 26th March, 2020. The format of the date is important for the weather API query.

Many more API parameters are available. For more information on the full set of Weather API parameters, see the Weather API documentation.

Downloading the weather data

The next section of code creates the Weather API request from the parameters, submits the request to the server and then parses the result.


# Set up the specific parameters based on the type of query
if QueryType == 'FORECAST':
    print(' - Fetching forecast data')
    QueryTypeParams = 'forecast?&aggregateHours=24&unitGroup=us&shortColumnNames=false'
else:
    print(' - Fetching history for date: ', FromDateParam,'-',ToDateParam)

    # History requests require a date.  We use the same date for start and end since we only want to query a single date in this example
    QueryDate = '&startDateTime=' + FromDateParam + 'T00:00:00&endDateTime=' +ToDateParam + 'T00:00:00'
    QueryTypeParams = 'history?&aggregateHours=24&unitGroup=us&dayStartTime=0:0:00&dayEndTime=0:0:00' + QueryDate


# Build the entire query
URL = BaseURL + QueryTypeParams + QueryLocation + QueryKey

# Build the entire query
URL = BaseURL + QueryTypeParams + QueryLocation + QueryKey

print(' - Running query URL: ', URL)
print()

# Parse the results as CSV
CSVBytes = urllib.request.urlopen(URL)
CSVText = csv.reader(codecs.iterdecode(CSVBytes, 'utf-8'))

The first part of the code constructs the requests to form a single URL. In this example, we are sending the request as a GET request. Other request techniques are available such as POST, which is useful if your list of locations is long or even ODATA for specialized data import and data science applications. See the Weather API documentation section for more information.

The final two lines of the code download the requested weather data and then parse the data using a CSV library. As mentioned above, we are using Comma Separated Values for the output in this example, but JSON format is available too. The CSV data is encoded in UTF-8 encoding so we indicate that to ensure accurate decoding.

Using the weather data

We now have the historical weather data as a CSVText instance. From here we can use the data in many ways. For example we can analyze the weather data in R, load it into a database or simply display it to the user. In this example, we simply display the user to the screen.

The raw data is simply a table of weather data rows. In this case the historical weather data for each day requested.

Address,Date time,Minimum Temperature,Maximum Temperature,Temperature,Dew Point,Relative Humidity,Heat Index,Wind Speed,Wind Gust,Wind Direction,Wind Chill,Precipitation,Precipitation Cover,Snow Depth,Visibility,Cloud Cover,Sea Level Pressure,Weather Type,Latitude,Longitude,Resolved Address,Name,Info,Conditions
"Herndon,VA",01/01/2019,44.6,60.8,53.1,45.1,75.58,,23.2,36.3,269.29,41.4,0,8.33,,9.1,83.5,1014.8,"Mist, Fog, Light Rain",38.96972,-77.38519,"Herndon,VA","","","Overcast"
"Herndon,VA",01/02/2019,37.3,44.9,42.4,35.9,78.17,,8.2,,143.95,38.7,0,0,,10,98.3,1023.6,"",38.96972,-77.38519,"Herndon,VA","","","Overcast"

The code simply steps through the rows of CSV data, prints out the data.

RowIndex = 0

# The first row contain the headers and the additional rows each contain the weather metrics for a single day
# To simply our code, we use the knowledge that column 0 contains the location and column 1 contains the date.  The data starts at column 4
for Row in CSVText:
    if RowIndex == 0:
        FirstRow = Row
    else:
        print('Weather in ', Row[0], ' on ', Row[1])

        ColIndex = 0
        for Col in Row:
            if ColIndex >= 4:
                print('   ', FirstRow[ColIndex], ' = ', Row[ColIndex])
            ColIndex += 1
    RowIndex += 1

Full source code for downloading historical weather data in Python

For the full code listing, head over to our Github for this and other Python, Java and other Weather API examples.

Full source code.

Next steps

This simple code demonstrates how easy it is to download historical weather data using Python without needing the headaches of scraping historical data. With the weather data in Python, you can now start analyzing and using the data in your next project.

Do you have a question or an interesting way of using weather data in Python? Let us know in the comments below.

2 Replies to “How to load historical weather data using Python without scraping!”

  1. Hi,

    Apologies for this “”newbie” question, but am getting the following error running your “loading_historical_weather_data_into_python” script.

    Any idea what is causing this?

    Many thanks

    Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32
    Type “help”, “copyright”, “credits” or “license” for more information.
    >>> $ loading_historical_weather_data_into_python
    File “”, line 1
    $ loading_historical_weather_data_into_python
    ^
    SyntaxError: invalid syntax

    1. If you are simply looking to run the code, your can do so in any of a few ways. The most common way is to run it directly from your command line. In this case, you would run a command like this at your OS command prompt:

      python3 loading_historical_weather_data_into_python.py

      To give command line parameters, just place them after the script name with a command line like this:

      python3 loading_historical_weather_data_into_python.py "Herndon, VA" HISTORY YOUR_API_KEY 2019-01-01 2019-01-07
      (Of course, replace YOUR_API_KEY with your Visual Crossing Weather API key.)

      If you are already inside a python 3 environment, you could instead use subprocess.call() or os.system(). Both will allow you to specify command line options as are required for this script. However, they just run the script externally, so there is little benefit to running it inside an open environment.

      Good luck, and let us know if you have any issue getting the script to run. Please note we have made a few small changes to the script in the past couple of days so you may want to download the source again to make sure you have the latest.

      Regards
      Visual Crossing Support

Leave a Reply

Your email address will not be published. Required fields are marked *