In How to Import Weather Data in MySQL , we demonstrated how to load weather forecast data into MySQL using a python script that was run on regular intervals. Historical weather data is often just as critical for data science applications so in this article we demonstrate how to load weather history data using Python. We will use the Visual Crossing Weather API that offers a free tier that includes past weather data.
Why shouldn’t we scrape weather data from our favorite web site?
Scraping weather data means we simply visit a web site and either manually or programmatically copy the data from that web page. Programmatic scraping of weather data can be difficult to implement and then difficult to maintain. When the web site changes (even for very small changes), the scraping code may need changing. Most importantly, scraping weather data is against the usage terms of almost all web sites.
If we are looking for a reliable solution to retrieving weather data on regular intervals we need a more reliable solution. Using a Weather API avoids us have to scrape data.
Before starting, you need access to Weather API we are going to use. If you don’t have account already, you can sign up for a free trial at Visual Crossing Weather Services. VIsual Crossing offers a perpetual free tier – the initial free tier offers additional usage limits for a limited time. Once we have the code working in the free trial, we can drop back to the free tier or upgrade to a paid plan depending on our requirements.
Our Python script has been written in Python 3.8.2 but should work in most recent versions. We have kept the library requirements to a minimum. In this sample we are going to use the CSV format to download the data so we include a library to help process the data that is returned by the API.
Here’s the full list of import statements in our script
import csv import codecs import urllib.request import sys
Setting up the weather data input parameters
The first part of our script sets up the parameters for the script to download the weather data and the second part retrieves the weather data. The weather data is retrieved using a RESTful weather API so we simply have to create a web query within the Python script and download the data.
The first part of the sets up some parameters to customize the weather data that is retrieved
# This is the core of our weather query URL BaseURL = 'https://weather.visualcrossing.com/VisualCrossingWebServices/rest/services/weatherdata/' # Set up the location parameter for our query QueryLocation = '&location=' + urllib.parse.quote(sys.argv) # Set up the query type parameter for our query ('FORECAST' or 'HISTORY') QueryType=sys.argv.upper() # Set up the key parameter for our query QueryKey = '&key=' + sys.argv # Set up the date parameters for our query. Used only for historical weather data requests FromDateParam = sys.argv ToDateParam = sys.argv
In the above code, we set up 5 parameters. for the script.
The first parameter is the location for which the weather data should be downloaded. This is in form of an address, partial address or latitude and longitude value (for example 35.46,-75.12).
The code then asks for a query type (‘FORECAST’ or ‘HISTORY’) to choose between downloading weather forecast data or historical weather data.
We then ask for the API Key which is provided when signing up for the API. You can access it from the ‘Account’ location within Weather Data Services application.
Finally, if this is a historical weather data request, the code requests a start and end date in the form YYYY-MM-DD, for example 2020-03-26 is the 26th March, 2020. The format of the date is important for the weather API query.
Many more API parameters are available. For more information on the full set of Weather API parameters, see the Weather API documentation.
Downloading the weather data
The next section of code creates the Weather API request from the parameters, submits the request to the server and then parses the result.
# Set up the specific parameters based on the type of query if QueryType == 'FORECAST': print(' - Fetching forecast data') QueryTypeParams = 'forecast?&aggregateHours=24&unitGroup=us&shortColumnNames=false' else: print(' - Fetching history for date: ', FromDateParam,'-',ToDateParam) # History requests require a date. We use the same date for start and end since we only want to query a single date in this example QueryDate = '&startDateTime=' + FromDateParam + 'T00:00:00&endDateTime=' +ToDateParam + 'T00:00:00' QueryTypeParams = 'history?&aggregateHours=24&unitGroup=us&dayStartTime=0:0:00&dayEndTime=0:0:00' + QueryDate # Build the entire query URL = BaseURL + QueryTypeParams + QueryLocation + QueryKey # Build the entire query URL = BaseURL + QueryTypeParams + QueryLocation + QueryKey print(' - Running query URL: ', URL) print() # Parse the results as CSV CSVBytes = urllib.request.urlopen(URL) CSVText = csv.reader(codecs.iterdecode(CSVBytes, 'utf-8'))
The first part of the code constructs the requests to form a single URL. In this example, we are sending the request as a GET request. Other request techniques are available such as POST, which is useful if your list of locations is long or even ODATA for specialized data import and data science applications. See the Weather API documentation section for more information.
The final two lines of the code download the requested weather data and then parse the data using a CSV library. As mentioned above, we are using Comma Separated Values for the output in this example, but JSON format is available too. The CSV data is encoded in UTF-8 encoding so we indicate that to ensure accurate decoding.
Using the weather data
We now have the historical weather data as a CSVText instance. From here we can use the data in many ways. For example we can analyze the weather data in R, load it into a database or simply display it to the user. In this example, we simply display the user to the screen.
The raw data is simply a table of weather data rows. In this case the historical weather data for each day requested.
Address,Date time,Minimum Temperature,Maximum Temperature,Temperature,Dew Point,Relative Humidity,Heat Index,Wind Speed,Wind Gust,Wind Direction,Wind Chill,Precipitation,Precipitation Cover,Snow Depth,Visibility,Cloud Cover,Sea Level Pressure,Weather Type,Latitude,Longitude,Resolved Address,Name,Info,Conditions "Herndon,VA",01/01/2019,44.6,60.8,53.1,45.1,75.58,,23.2,36.3,269.29,41.4,0,8.33,,9.1,83.5,1014.8,"Mist, Fog, Light Rain",38.96972,-77.38519,"Herndon,VA","","","Overcast" "Herndon,VA",01/02/2019,37.3,44.9,42.4,35.9,78.17,,8.2,,143.95,38.7,0,0,,10,98.3,1023.6,"",38.96972,-77.38519,"Herndon,VA","","","Overcast"
The code simply steps through the rows of CSV data, prints out the data.
RowIndex = 0 # The first row contain the headers and the additional rows each contain the weather metrics for a single day # To simply our code, we use the knowledge that column 0 contains the location and column 1 contains the date. The data starts at column 4 for Row in CSVText: if RowIndex == 0: FirstRow = Row else: print('Weather in ', Row, ' on ', Row) ColIndex = 0 for Col in Row: if ColIndex >= 4: print(' ', FirstRow[ColIndex], ' = ', Row[ColIndex]) ColIndex += 1 RowIndex += 1
Full source code for downloading historical weather data in Python
For the full code listing, head over to our Github for this and other Python, Java and other Weather API examples.
This simple code demonstrates how easy it is to download historical weather data using Python without needing the headaches of scraping historical data. With the weather data in Python, you can now start analyzing and using the data in your next project.
Do you have a question or an interesting way of using weather data in Python? Let us know in the comments below.