Understanding the “fair use” system in Visual Crossing Weather

Visual Crossing Weather is the most powerful and cost effective way to obtain weather data for your data analysis project, weather app, or virtually any other weather use case. We pride ourselves in providing the highest-quality weather data at the most affordable prices. In order to protect our user community from runaway scripts, malicious abusers, and misguided attempts to abuse the system, we’ve implemented a fair use policy for some of our plans.

In this article we will explain what fair use is, why it exists, how it benefits you, and how to implement your query code to manage any fair use errors that you may receive.

What is “fair use” and why does it need to exist?

The fair use system was designed to ensure that all users of the system can enjoy the benefits of their subscription without limitations or downtime. It does this by identifying accounts that are greatly exceeding the normal usage patterns for their account level and then adjusting their activities to help them become better citizens of the system. The fair use system can do things such as deprioritize jobs by adjusting their queue placement or throttling the account temporarily to limit the number of concurrent queries permitted. Its goal is simple, to maximize system performance and stability thereby effectively managing the use of the shared weather database and query engine so that all users can get the weather data that the need quickly and efficiently.

Fair use is based on the type of query and service plan level. Certain types of queries are much more expensive than others and thus are more likely to be flagged by the fair use system, since it is monitoring the cost of the queries as an important criteria. This means that forecast queries alone will almost never trigger the fair use system. While forecast data is based on extremely complex data models that are run on some of the world’s fastest supercomputers and take many hours to complete, once processed the final forecast data can be served instantaneously. In fact, typically every front-line query server in our cluster caches the entire global forecast in memory. This makes forecast queries extremely quick and extremely low cost for the servers. In contrast, some of the most expensive queries are historical weather summaries that can involve hitting tens of thousands of data records across decades of observations.

Likewise, your service plan level also is part of the fair use calculation. Users on free and basic plans are more likely to see the effects of the fair use system. The more powerful your plan level, the more data access you expect and the higher querying level you can sustain without risk of exceeding fair use. This is something that you will want to keep in mind as you consider plan levels to support critical systems for your business or organization.

Before we continue our explanation, it is important to note that the Visual Crossing Weather database is hosted on the highly scalable Amazon Web Services (AWS) platform. AWS hosts titans such as Netflix, Twitter, Airbnb, and the BBC. Our infrastructure is designed to host hundreds of millions of queries from hundreds of thousands of users. It will have no problems handling your most demanding weather workload. So, if you find your weather use case chafing under the constraints of fair use or any other limitation of our standard plans, just reach out to us regarding our enterprise options. Using an enterprise plan, we can help you craft a weather data solution that can meet nearly any requirements. So, do not confuse hitting fair use with hitting the limits of our global weather database in general. With the correct access plan, our system can scale to support even your largest weather projects.

How does “fair use” benefit me?

The fair use system is designed to benefit all users who want to get their weather data without worrying about unexpectedly slow queries and unnecessarily high cost that can plague a poorly managed shared resource. It is designed to keep both accidental, run-away scripts as well as malicious attackers from disrupting normal system usage. It is also designed to keep costs low for all systems users by ensuring that the vast majority of reasonable users don’t carry the burden of paying for accounts that are taking far more than their share of system resources, either accidently or via purposeful abuse. Your primary benefit is a smooth-running weather query system that serves weather data to your application without fail and at the lowest costs in the industry.

Note also that the fair use system helps to protect users who accidently configure run-away scripts or other unplanned, excessive query activity. Since we all make mistakes, at some point you may also benefit from the run-away protection. While the system can’t always protect you from your mistakes, it can sometimes catch a script or application that has “run away” and is consuming far more weather queries than is within the normal pattern. By throttling and eventually erroring accounts that are generating enormous numbers of queries or extremely high volumes in a short time period, the fair use system can help protect users from accidentally consuming their entire data allotment or running up unexpected bills. While you should never count on the fair use system protecting your account from your own mistakes, you’ll be thankful if at some point it does.

How to implement weather query code to intelligently interact with the “fair use” system

For most customers and in most cases, there is no need to ever think about the Visual Crossing Weather’s fair use system. It works silently in the background 24×7 to ensure that your weather data service is available and responsive whenever you need it. However, a smart developer always wants to write their code to handle all conditions that it might encounter and this includes hitting fair use limits.

There are three primary situations that your weather query application may wish to consider regarding the fair use system. The first situation is query throttling. In this situation, the query engine will use its queuing system to demote the job priority for the errant account. This will cause jobs that typically run very quickly to return a little more slowly than usual. This change is subtle enough that your code likely will not even detect this case unless it is very adept at monitoring overall query performance. However, if you do have such a monitoring system in place, seeing this slowing pattern may be the first sign that you are getting close to exceeding the limits for your service plan.

The second issue, and most important consideration for coders, is the return of an explicit error as a response to your query. The primary error that your code will see in this case is code 429. The textual description returned with this error nicely explains the reason.

Your usage has exceeded the fair usage for the current day. You can continue to execute queries however you cannot execute more than one concurrent request.

If your code encounters this error, it means that it must stop running multiple, concurrent queries and reduce the query level to only one at a time for the remainder of the day. This condition is often triggered by using most or all of the threads available for your plan to run complex queries. The maximum concurrency limit given for a service plan is intended for supporting high concurrency, small dataset downloads. For complex queries and sustained query load such as to populate databases and data lakes, using a lower concurrency limit will produce more consistent results and avoid the risk of encountering issues.

As an aside, your code should always check all returned results for error codes. One of the most common issues that our tech support team sees is when people fail to check the return code and don’t realize that error condition exists. In nearly all cases, the error description will help you understand the cause of the problem and how to remedy it. If you receive a HTTP code that is not equal to success (200) then capture both the error code and the response content of the request. You can often save yourself a lot of time just by catching and reading the error message.

A third issue and the most extreme that your code may encounter is related to queries that are so complex and have been demoted in priority so far that they encounter a timeout. Annoyingly, the exact delay for timeouts are rather unpredictable in the complex world of the Internet. There are various hops in the route that your query takes between your client and our servers, and many of those hops can introduce their own timeouts. The timeout period that your query hits will be the shortest period set by any hop in the network path. So while you might set a long timeout in your code, for example, your query might hit a timeout at some other place in the network.

Timeouts can appear to be random and thus sometimes are difficult to debug. Also, there can be various reasons for your code to encounter a timeout. So it can be hard to pinpoint the exact cause. The best advice for handling timeouts is to avoid them in the first place. One key way to do this in the weather query arena is to make your queries as small and light as possible. For example, if you are querying multiple locations and/or multiple years of historical data, considering reduce the number of both. You can read more about this in our article on query segmenting. Beyond this, the best way to solve timeout issues is to carefully track the query patterns before and after the error. If you see slowness and throttling of normally faster queries before the timeout, there is a chance that you might be trigging the throttling from the fair use system. On our side, our servers will eventually process your query or return an error. However, some network setting between your client and our servers may not be willing to wait and instead issue a timeout. If you have this issue and are unable to get to the bottom of it yourself, you can reach out to our support team. They can tell you if your account has activated our fair use system.

Summary

We hope that this article has helped to shed some light on the usually-invisible fair use system and how it can affect your weather queries. It acts as a silent partner helping to ensure that you can get the weather data that you need without delay and ensuring that you continue to pay the lowest possible prices for your data. However, if is useful to understand how the system works and to prepare your code in case it is triggered for your account. After reading this article, you should be better prepared to manage this case in your code and then adjust your system accordingly to ensure that you as well as out entire user community has continuous access to the weather data that they need.

Questions or need help?

If you have a question or need help, please post on our actively monitored forum for the fastest replies. You can also contact us via our support site or drop us an email at support@visualcrossing.com.