From ISD to Modern Multi-Source Historical Weather Processing

Visual Crossing historically used NOAA’s Integrated Surface Database (ISD) as an important source of global surface weather observations.

ISD helped establish many of the processing techniques that remain relevant to historical weather today, including:

• Identifying weather stations
• Interpreting station metadata
• Handling multiple reports within an hour
• Converting observation times
• Applying quality-control flags
• Processing accumulated precipitation
• Detecting missing or invalid observations
• Combining observations from multiple nearby stations

ISD is no longer used as an active Visual Crossing production source.

Modern Visual Crossing historical weather processing uses a broader collection of observation feeds and archives, including sources such as:

• METAR and aviation observations
• NOAA GHCNh hourly observations
• NOAA GHCNd daily observations
• Deutscher Wetterdienst (DWD) observations
• Météo-France observations
• Other national and regional meteorological networks
• Mesonet and specialized station networks
• Radar, satellite, and other remote observation sources where appropriate

These sources differ significantly in their formats, reporting frequencies, quality controls, station identifiers, historical coverage, and weather elements.

Visual Crossing normalizes and evaluates these different sources before using them to create historical weather for a requested geographic location.

This article uses ISD as a case study because it illustrates many of the challenges involved in processing raw station observations. It also explains how those same concepts apply to the current multi-source Visual Crossing Weather Data system.

For a broader explanation of how station observations are combined for a geographic point, see How Historical Weather Data Records Are Created from Local Weather Station Observations.

Why historical weather requires significant processing

Raw weather-station data is rarely ready to use directly.

An individual upstream source may contain:

• Multiple reports from the same station within an hour
• Missing weather elements
• Different reporting intervals for different elements
• Quality-control flags
• Encoded or scaled numeric values
• Accumulated precipitation covering several hours
• Duplicate observations
• Corrected observations
• Different station identifiers for the same physical location
• Station relocations or metadata changes

Different source networks may also overlap.

For example, the same airport may appear in:

• An operational METAR feed
• An hourly historical archive
• A national weather-service archive
• A daily climatological dataset

These records cannot simply be appended together and treated as independent measurements.

Before historical weather can be produced consistently, source observations need to be:

1. Decoded
2. Normalized
3. Matched to station metadata
4. Quality checked
5. De-duplicated
6. Time aligned
7. Evaluated for the requested weather element
8. Combined with other available observations where appropriate

The move from ISD to multiple observation sources

No single weather archive provides the best coverage, timeliness, quality, and weather-element availability for every location and period.

Visual Crossing therefore uses multiple observation sources rather than depending on one global archive.

Depending on the requested location, date, and weather element, available sources can include:

• METAR
• GHCNh
• GHCNd
• DWD
• Météo-France
• Other national and regional station networks
• Specialized observation networks
• Radar and satellite products

Each source has different strengths.

For example:

• An operational feed may provide very recent observations quickly.
• A historical hourly archive may provide broader long-term coverage.
• A daily climate dataset may provide authoritative daily precipitation, minimum temperature, or maximum temperature.
• A national meteorological service may provide denser or higher-quality observations within its own country.
• Radar or satellite data may supplement sparse surface observations for certain weather elements.

The Weather Data Engine therefore evaluates sources according to their suitability rather than assuming that one dataset should always be preferred everywhere.

Hourly and daily datasets have different roles

Not all observation sources operate at the same time resolution.

For example:

GHCNh

provides hourly and sub-daily observations, while:

GHCNd

provides daily climate summaries.

Those datasets are not interchangeable.

Hourly datasets are useful when you need to reconstruct weather through the day.

Daily datasets can be particularly useful for values such as:

• Daily precipitation
• Daily maximum temperature
• Daily minimum temperature
• Snowfall
• Snow depth
• Other climatological summaries

A daily observation source should not simply be expanded into hourly data without considering what the field represents.

Similarly, hourly observations should not always be assumed to reproduce an independently reported authoritative daily total exactly.

Visual Crossing treats each source according to its native reporting resolution and intended use.

The same station can appear in multiple datasets

A physical weather station may be represented in more than one upstream source.

For example, an airport station may appear in:

• An operational METAR stream
• A historical hourly archive
• A national observation dataset
• A daily climatological archive

If all of those records were treated as separate stations, the same underlying observation could be counted multiple times.

Visual Crossing therefore uses station identifiers, coordinates, metadata, report times, and source information to identify overlapping observations.

The system can also apply source priorities or quality rules so that multiple representations of the same report do not incorrectly increase its influence.

Later or more authoritative versions of an observation can also replace earlier provisional versions.

This is one reason why very recent historical weather can sometimes change after first becoming available.

See How Historical Weather Data Is Updated for more information.

Normalizing different observation formats

Every weather provider has its own schema.

One source may report:

temperature = 23.4

while another encodes a temperature as an integer that must be divided by ten.

One source may represent missing data as:

null

while another may use a sentinel value such as:

9999

A timestamp may represent:

• The exact observation time
• The end of an observation interval
• The beginning of an interval
• The reporting time rather than the measurement time

Before observations from multiple networks can be compared, they must be normalized into a common representation.

Typical normalization steps include:

• Converting units
• Decoding scaled values
• Mapping weather-element names
• Converting timestamps
• Interpreting reporting periods
• Standardizing quality flags
• Identifying missing values
• Mapping station identifiers
• Normalizing precipitation accumulation periods

This creates a common internal weather representation that can be used across many different source datasets.

ISD as a case study

Although ISD is retired from the current Visual Crossing production pipeline, it remains a useful example of the challenges involved in raw weather-data processing.

The Integrated Surface Database historically combined observations from thousands of stations worldwide and exposed many of the same issues that still exist in modern observation feeds.

These include:

• Multiple station-identification systems
• Multiple report types
• Observations that do not occur exactly on the hour
• Quality flags
• Encoded numeric values
• UTC timestamps
• Multi-hour precipitation accumulations
• Missing values
• Changes in station availability over time

The following sections use ISD to illustrate those processing concepts.

Identifying an ISD weather station

ISD stations were commonly identified using two components:

USAF
WBAN

Together, these values formed a station identifier.

For example, an airport station could be represented using a combined identifier such as:

724030-93738

The exact identifier depended on the station metadata published by NOAA.

This illustrates an important general issue in multi-source weather processing:

A station name alone is not enough to identify a weather station reliably.

Station names can:

• Change
• Be duplicated
• Contain spelling variations
• Refer to airports, cities, or geographic areas rather than a precise sensor location

Visual Crossing therefore relies on identifiers and station metadata rather than station names alone.

Station metadata

Station metadata can include:

• Station ID
• Name
• Latitude
• Longitude
• Elevation
• Country
• Network
• Active date range
• Observation type

This information is essential when deciding whether a station is appropriate for a requested geographic location.

For example, two stations may be geographically close but at very different elevations.

That can be important for:

• Temperature
• Snow
• Wind
• Mountain weather

Station metadata can also change over time.

A station may:

• Move
• Change elevation metadata
• Change identifier
• Stop reporting
• Resume reporting
• Change observing equipment

Historical processing must therefore consider station metadata in the context of the observation date.

Finding stations that were active during a period

One useful property of historical station inventories is that they show when stations were active.

A station that exists today may not have existed 30 years ago.

Likewise, a historical station may no longer be operational.

When selecting observations for historical weather, Visual Crossing must consider whether the station actually reported during the requested date.

This becomes particularly important for long historical datasets.

The observation network around a location can change substantially over several decades.

Weather stations do not necessarily report exactly on the hour

A common assumption is that an hourly weather station reports at exactly:

13:00
14:00
15:00

Real observation feeds are often more complicated.

A station may report at:

13:51
14:51
15:51

Another station may report at:

13:55
14:55
15:55

Other networks may report every:

5 minutes
10 minutes
20 minutes
30 minutes

or on irregular schedules.

Historical processing therefore needs a method for associating source observations with the output time period.

The best method depends on:

• Source type
• Observation frequency
• Weather element
• Requested output resolution

Multiple observations can occur within one hour

Some weather stations report more than once per hour.

For example:

14:05
14:20
14:35
14:50

If the desired output is one hourly record, the processing system must determine how those observations should be represented.

For a state variable such as temperature, possible approaches might involve selecting or summarizing observations associated with the hourly period.

For accumulated fields such as precipitation, the reporting interval must be interpreted correctly before values can be combined.

The important point is that:

One hourly output record does not necessarily correspond to one raw upstream station report.

Report types

ISD included multiple report types.

Different report types could represent different observation products, including routine surface reports and other report categories.

This is another common feature of weather-data feeds.

Two records from the same station and timestamp may not necessarily be equivalent observations.

Processing must consider:

• Report type
• Source
• Quality
• Completeness
• Whether one report supersedes another

Simply selecting the first record for a timestamp can produce incorrect results.

Quality-control flags

Many weather observation networks include quality indicators.

A value may be flagged as:

• Valid
• Suspect
• Failed
• Corrected
• Estimated
• Missing

ISD used source-specific quality flags attached to weather elements.

Other modern sources use their own quality-control systems.

Visual Crossing converts source-specific quality information into rules that determine whether an observation should be used.

A suspect temperature value should not automatically be treated with the same confidence as a valid observation.

Quality decisions can also be element-specific.

For example:

• Temperature may be valid.
• Precipitation may be missing.
• Wind may be suspect.

The station should not necessarily be rejected completely because one element is invalid.

Encoded and scaled values

Raw meteorological datasets often store numeric values in encoded forms.

For example, a source may store:

234

to represent:

23.4°C

Other sources may use special numbers for:

• Missing data
• Trace precipitation
• Sensor errors
• Overflow values

These formats are convenient for compact data storage but require careful decoding.

Incorrect interpretation of scaling factors can create values that look numerically valid but are physically impossible.

For example:

temperature = 234°C

might actually represent:

23.4°C

This is why source documentation and metadata are critical when ingesting raw weather observations.

UTC versus local time

Many global observation archives store timestamps in UTC.

For example:

2026-01-15 18:00 UTC

may correspond to:

2026-01-15 13:00

in a location using UTC-5.

Historical weather users generally expect timestamps and daily summaries to correspond to the local timezone of the requested location.

This means the processing system must correctly convert observations from their source timestamps into the appropriate local timeline.

Timezone conversion is especially important for:

• Daily precipitation
• Daily minimum temperature
• Daily maximum temperature
• Midnight boundaries
• Daylight saving time

An observation near midnight UTC can belong to the previous local calendar day in the Americas or the next local day farther east.

For more information, see Dates and Times in the Weather API.

Daylight saving time

Local time introduces another complication: daylight saving time.

During a spring transition, a local day may contain only:

23 hours

During an autumn transition, it may contain:

25 hours

A local hour may also appear twice when clocks move backward.

This means hourly historical processing should not assume that every local calendar day contains exactly 24 unique clock-hour labels.

The Timeline Weather API includes:

datetimeEpoch

to provide an unambiguous timestamp.

Precipitation is particularly difficult

Precipitation is one of the most complex station fields to process.

A weather station may report precipitation covering:

1 hour
3 hours
6 hours
12 hours
24 hours

depending on the observation network and reporting schedule.

A value such as:

0.50 in

does not mean much unless you also know the period over which that amount accumulated.

For example:

0.50 in over 1 hour

is very different from:

0.50 in over 24 hours

Historical processing must therefore consider both:

• Amount
• Accumulation period

Precipitation reports often describe a period ending at the observation time

Suppose a station reports at:

18:00

and provides:

6-hour precipitation = 0.60 in

That value generally represents precipitation accumulated during a preceding interval associated with that observation time.

It should not be interpreted as:

0.60 in fell exactly at 18:00

This distinction is essential when constructing hourly precipitation.

Multi-hour precipitation cannot always be reconstructed exactly

Suppose the only available report says:

6-hour precipitation = 0.60 in

but no hourly precipitation observations are available.

We know the total amount over the six-hour interval.

We may not know precisely when within that period the rain occurred.

For some analytical purposes, the amount could be distributed across the covered hours, but this introduces an approximation.

For example:

0.60 / 6 = 0.10 in per hour

would preserve the total precipitation but would not reconstruct the actual timing of the event.

For applications where timing matters, higher-frequency source observations or remote precipitation sources may provide better information.

Trace precipitation

Weather observation networks can also report:

trace

precipitation.

A trace means precipitation was observed but the amount was below the normal measurable threshold.

It should not automatically be interpreted as:

0

nor should it necessarily be assigned an arbitrary measurable amount.

The correct treatment depends on the analytical use case and the source conventions.

Daily precipitation can come from different source logic than hourly precipitation

An authoritative daily precipitation observation may not exactly equal the sum of the hourly precipitation records available from another source.

Reasons can include:

• Different observation windows
• Missing hourly reports
• Late reports
• Quality corrections
• Different instruments
• Different source networks

This is one reason Visual Crossing can use both hourly and daily datasets.

A daily climatological source can provide valuable validation or supplementation for a daily total even when the underlying hourly observation network has gaps.

Temperature observations

Temperature appears simpler than precipitation, but it also requires careful processing.

Possible issues include:

• Missing observations
• Sensor errors
• Duplicate reports
• Unit conversion
• Station relocation
• Elevation differences
• Rounding

A station may also report more frequently than the required output resolution.

The Weather Data Engine must select or summarize observations appropriately for the requested record.

Daily maximum and minimum temperature

A daily maximum temperature is not necessarily identical to:

max(hourly temperature output)

and daily minimum temperature is not necessarily identical to:

min(hourly temperature output)

A daily climate source may include extrema observed between the standard hourly reporting times.

The native observation period can also differ from the local midnight-to-midnight calendar period.

For authoritative analysis, use the returned daily fields:

tempmax
tempmin

rather than assuming they can always be reconstructed exactly from the displayed hourly records.

Wind observations

Wind fields can include:

windspeed
winddir
windgust

Wind direction requires special treatment because it is circular.

For example:

350°

and:

10°

are only 20 degrees apart.

A simple arithmetic average gives:

180°

which is incorrect.

Wind direction must therefore be handled using circular or vector mathematics rather than an ordinary numeric average.

Wind gusts also represent maxima over observation periods rather than normal state averages.

Missing observations

Weather stations do not report every weather element continuously.

A station may have:

• Temperature but no precipitation
• Wind but no snow
• Pressure but no visibility
• Hourly observations with occasional gaps

Visual Crossing therefore evaluates observations at the individual element and record level.

A station does not necessarily need to be rejected entirely because one weather element is missing.

Other nearby stations or source networks can provide the missing information.

Using surrounding stations to detect suspect data

Nearby weather stations provide useful context for quality control.

For example, suppose one station reports:

temperature = 110°F

while all nearby stations report approximately:

70°F

That does not automatically prove the 110°F value is wrong, but it is a strong indication that the observation deserves additional scrutiny.

Nearby observations can help identify:

• Sensor errors
• Unit problems
• Stuck sensors
• Impossible jumps
• Incorrect precipitation
• Metadata errors

Spatial comparison is one useful quality-control technique among several.

Different sources can disagree

When multiple networks cover the same area, they may not report identical values.

Possible reasons include:

• Different instruments
• Different station locations
• Different observation times
• Different quality-control procedures
• Different reporting precision
• Different update latency

The Weather Data Engine must therefore evaluate which observations are most appropriate rather than assuming that disagreement means one source is necessarily wrong.

Source priority can vary by weather element

The best source for:

temperature

may not be the best source for:

precipitation

or:

snow

or:

wind

Visual Crossing can therefore evaluate source suitability by weather element.

This is important when blending data from station networks, national datasets, radar, and satellite observations.

Source priority can also vary by time period

Observation networks evolve.

A source that provides excellent coverage today may not have existed 20 years ago.

Conversely, a historical station may have stopped reporting.

As a result, the preferred source for a geographic point can change over the historical record.

This is normal.

A long historical series should represent the weather at the requested location rather than depend permanently on one station or one upstream archive.

From normalized observations to location-based weather

After observations are decoded, normalized, quality checked, and de-duplicated, Visual Crossing can evaluate them for the requested geographic location.

Historical weather for an arbitrary point is not normally just a copy of the single nearest station.

Multiple observations can be considered.

Factors can include:

• Distance
• Station quality
• Elevation
• Weather-element availability
• Observation quality
• Source quality
• Temporal availability

Closer and higher-quality observations will generally have more influence than less suitable observations.

However, the exact set of observations can vary from one hour or weather element to another.

For more information, see How Historical Weather Data Records Are Created from Local Weather Station Observations.

The station list can change within a day

A daily result may be associated with more stations than the configured station target.

This does not necessarily mean that all of those stations contributed simultaneously.

For example:

• Station A may provide temperature for the morning.
• Station B may fill a gap during the afternoon.
• Station C may provide precipitation.
• Station D may become necessary when another station stops reporting.

The set of available observations can therefore change through the day.

This is why a list of stations associated with a daily record should not automatically be interpreted as a fixed weighting of every station across every weather field.

Remote observations supplement station data

Surface stations are only one class of historical weather source.

Depending on the element and location, Visual Crossing can also use remote observations such as:

• Radar
• Satellite
• Other gridded observation products

These can be particularly useful where:

• Surface stations are sparse
• Precipitation varies strongly over short distances
• Marine locations lack traditional land stations
• Remote sensing provides better spatial coverage

For more information, see Using Remote Data Sources in the Weather API.

Recent historical data can change

Observation feeds do not all arrive at the same time.

A very recent historical record may initially be created using the sources available at that moment.

Later:

• Additional observations may arrive.
• A national archive may publish a more complete record.
• An upstream provider may correct an observation.
• A higher-quality source may become available.

Visual Crossing can then update the historical result.

For this reason, recently observed weather should not always be assumed to be permanently fixed immediately after the event.

See How Historical Weather Data Is Updated.

Why multi-source processing improves historical weather

Depending on a single observation archive has several disadvantages.

No individual source has:

• Perfect global coverage
• Complete historical depth
• Every weather element
• No missing observations
• No reporting delays
• No quality issues

A multi-source system can instead use the strengths of different datasets.

For example:

METAR

may provide timely airport observations.

GHCNh

may provide historical hourly observations.

GHCNd

may provide valuable daily climatological measurements.

DWD

can provide detailed German observations.

Météo-France

can provide authoritative French observations.

Other regional networks can improve local coverage elsewhere.

Remote observations can provide additional spatial information.

The result is a more resilient historical-weather system than one based entirely on a single global station archive.

Why ISD is still a useful example

ISD is no longer an active Visual Crossing production source, but the lessons from processing it remain relevant.

Weather-data engineering still requires solving the same fundamental problems:

• What station produced this observation?
• Where was that station?
• When was the value actually observed?
• What period does the value represent?
• What units are being used?
• Is the value valid?
• Is it a duplicate?
• Is there a corrected version?
• Is another source better?
• How should multiple observations be combined?

Modern data sources may use different file formats and APIs, but those core questions have not changed.

Summary

Visual Crossing historical weather has evolved from relying heavily on datasets such as NOAA ISD to a modern multi-source observation system.

Current processing can include sources such as:

• METAR
• GHCNh
• GHCNd
• DWD
• Météo-France
• Other national and regional observation networks
• Radar
• Satellite
• Other remote observation sources

Each source must be decoded, normalized, quality checked, and evaluated before it can be used.

Important processing challenges include:

• Station identification
• Metadata management
• Multiple reports per hour
• UTC and local-time conversion
• Quality flags
• Encoded values
• Duplicate observations
• Precipitation accumulation periods
• Daily versus hourly data
• Missing observations
• Source overlap
• Source prioritization

ISD remains a useful historical case study because it demonstrates many of these issues clearly, even though it is no longer part of the active Visual Crossing production source set.

For more information about how processed station observations are combined for a geographic location, see How Historical Weather Data Records Are Created from Local Weather Station Observations.

For information about historical data revisions, see How Historical Weather Data Is Updated.

For remote observation sources, see Using Remote Data Sources in the Weather API.

For the complete historical weather interface, see the Timeline Weather API documentation.