Blog Posts

Smart Server Hopping: How My GFS Downloader Navigates NOAA and AWS for Reliable Downloads

If you’ve ever tried pulling fresh GFS (Global Forecast System) data only to watch your download slow to a crawl, you know the frustration. I’ve recently been polishing a Perl script called Wx_GFS_Download.pl that handles the job with a bit of built-in cleverness and very little drama. It’s not revolutionary meteorology, but it reliably fetches GFS files while keeping things lightweight and entertaining for a fellow Perl hacker or weather nerd.

My weather data adventures began in aerospace, working on DMSP (Defense Meteorology Satellite Program) satellites, including the SSM/T-2 microwave sounder, at Aerojet Electro-Systems in Azusa, CA. Later, I worked with Landsat and MODIS programs at Hughes Santa Barbara Research Center.

After more than a decade in defense, I moved into finance. Around 2005, I was building tools for Citadel’s Energy Trading desk, delivering specific GFS slices to meteorologists each morning as fast as possible. Those early projects showed me that speed and dependability often matter more than complexity when real decisions depend on the data.

The downloader prefers the NOMADS server but keeps Amazon S3 ready as backup. It checks speeds and automatically hops if throughput drops below 20 MB/s. No parallel downloads are used since both hosts throttle multiple connections, so the script stays sequential and smart. The script primarily uses DBI, HTTP::Request, and LWP::UserAgent under the hood. Logging is handled by Log::Log4perl and Slack provides a coarser level of status messages.

It constructs paths like this for each date, cycle, and forecast hour:

$aws_s3_base = 's3://noaa-gfs-bdp-pds';
$nomads_base = 'https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod';
@urls_base   = ($nomads_base, $aws_s3_base);

$dirRoot = sprintf "%s/%s/c%02d/f%03d", $ENV{WX_GFS_DATA}, $gfs_date, $cycle, $forecast;
# dirRoot: /Volumes/Weather/GFS/2026-07-22/c06/f004
$path_file = sprintf "%s/GRB_Full/gfs.t%02dz.%s.f%03d", $dirRoot, $cycle, $product, $forecast;
# path_file: /Volumes/Weather/GFS/2026-07-22/c06/f004/GRB_Full/gfs.t06z.pgrb2.0p25.f004

The actual fetch switches cleanly between HTTP and the AWS CLI:

if ($url =~ /^https/) {
    # NOMADS
    # url: https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260722/06/atmos/gfs.t06z.pgrb2.0p25.f004
    # path_file: /Volumes/Weather/GFS/2026-07-22/c06/f004/GRB_Full/gfs.t06z.pgrb2.0p25.f004
    $req = HTTP::Request->new(GET => $url);
    $res = $ua->request($req, $path_file);
} 
else {
    # AWS S3
    # url: s3://noaa-gfs-bdp-pds/gfs.20260722/06/atmos/gfs.t06z.pgrb2.0p25.f004
    # path_file: /Volumes/Weather/GFS/2026-07-22/c06/f004/GRB_Full/gfs.t06z.pgrb2.0p25.f004
    $cmd = "$aws s3 cp --quiet --no-sign-request --region us-east-1 $url $path_file";
    $res = system("$cmd");
}

Speed checking and switching look roughly like this:

$dlSpeed = sprintf "%0.1f", ($file_size / 1048576) / $elapsedTime;
# file_size: 544,214,418 bytes
# elspasedTime: 9.4 seconds
# dlSpeed: 57.8 MB/sec
if ($dlSpeed < $dlSlow) {
    @urls_base = reverse(@urls_base);
    $text = sprintf "%s slow download (%0.1f < %0.1f MB/sec), switching to %s", 
           $urlBase, $dlSpeed, $dlSlow, ($urlBase eq 'NOMADS') ? 'AWS S3' : 'NOMADS';
    $log->info("$text");
}
else {
    $text = sprintf "%s %s download (%0.1f >= %0.1f MB/sec), continuing with %s", $urlBase, ($dlSpeed < 50) ? 'medium' : 'fast', $dlSpeed, $dlSlow, $urlBase;
    # text: NOMADS fast download (57.8 >= 20.0 MB/sec), continuing with NOMADS
    $log->info("$text");
}

A MariaDB GFS_Status table acts as the script’s memory. It tracks each file’s status, enables automatic retries, and performs quick sanity checks so you don’t waste time on corrupted data. A simple upsert keeps everything consistent.

In real runs the hopping shows up clearly in the logs. Here’s a snippet from a recent day:

• Trying NOMADS…
• NOMADS slow download (13.5 < 20.0 MB/sec), switching to AWS S3
• Trying AWS S3…
• AWS S3 slow download (15.5 < 20.0 MB/sec), switching to NOMADS
• NOMADS medium download (27.4 >= 20.0 MB/sec), continuing with NOMADS

The script supports the full 0–384 hour range across all four daily cycles (00, 06, 12, 18 UTC), though I typically only run 12 - 24 hours of forecast data per run. The downloader can pull down the Primary and Secondary forecast files around 30 seconds per pair, which is much faster than the database loading process, which we'll cover in a later blog. The current version works great when you need complete fields. Targeted small subsets, like the old Citadel days, are a different story and may deserve their own approach when throttling isn’t as punishing and parallel downloads are more performant.

All in all, Wx_GFS_Download.pl is a solid, no-nonsense tool that gets the weather data where it needs to go without fanfare. If you’re a Perl person who likes reliable automation or a weather hobbyist who wants data without the headaches, it might give you some ideas for your own projects. This is just the first in a series, next we’ll look at what happens after the files land. The eventual goal is to have both a straight-forward API for accessing any or all GFS data values for any forecast, as well as a service to provide formatted files in CSV, SQL, Excel, KML, and maybe a couple others, as well.

Stay tuned, and happy forecasting!