Blog Posts

This is just a quick look at a process we run for checking on Northern Lights activity, The gist of it is we download a file from a NOAA site, convert it from JSON to CSV, do a quick check to make sure the data is new (dtCreated and dtValid), then load it into the database. The NOAA file has 65,160 records with 5 fields per record, so it's a decent chunk of data.

Here's a log file from the old server:

2021/06/01 00:20:01 INFO> CPU.pm:194 - CPU load: 1.58 <= 8.00
2021/06/01 00:20:01 INFO> AuroraPercentDB_Load.pl:134 - Getting https://services.swpc.noaa.gov/json/ovation_aurora_latest.json
2021/06/01 00:20:06 INFO> AuroraPercentDB_Load.pl:149 - dt Created: 2021-06-01 05:11:00
2021/06/01 00:20:06 INFO> AuroraPercentDB_Load.pl:150 - dt Valid:   2021-06-01 06:34:00

And here's a log file from the new server:

2021/06/28 22:35:23 INFO> CPU.pm:194 - CPU load: 1.50 <= 8.00
2021/06/28 22:35:23 INFO> AuroraPercentDB_Load.pl:134 - Getting https://services.swpc.noaa.gov/json/ovation_aurora_latest.json
2021/06/28 22:35:23 INFO> AuroraPercentDB_Load.pl:149 - dt Created: 2021-06-29 03:26:00
2021/06/28 22:35:23 INFO> AuroraPercentDB_Load.pl:150 - dt Valid:   2021-06-29 04:50:00

Notice the gap in time between the 2nd and 3rd line of the first run (00:20:01 to 00:20:06) is five seconds and the gap for the second run (22:35:23 to 22:35:23) is zero seconds. There are really only two things happening in that time, downloading the file and parsing the JSON format into a structure the program can more easily understand. The time required to download the file shouldn't change from old to new server since they are both on the same network. So, the new server is parsing the JSON format much quicker for some reason.

I expected things to be faster, but that's much better than I had hoped.