Blog Posts

Do some industries use one type more than others?

There are a lot of terms used that describe essentially the same process of automatically getting or sending data to or from a remote site. The most common phrase is software automation, but you’ll also hear software bot, software agent, virtual robot, or just bot, all referring to the same concept. And it’s a concept Custom Visuals uses all day, every day.

If this blog interests you or your business, please click our Contact page and let us know what you think. You can also find more details on our Services page.

Custom Visuals uses a variety of programs to transfer data to and from internet sites.

1. Simple Transfer: This is used across all industries and most appropriate when the data exists in one or more files and is easily discernable as the most recent file to be transferred. Think of looking in a directory that contains the last 14 days of files all named with a clear date in the name. Another common use for this is when the same file is rewritten throughout the day and you need to pick up the changes every 5, 15 or 60 minutes.

2. Build, Send, Receive: This is most common for our financial clients that request a set of stock/instrument prices at specific intervals throughout the day. Since there are thousands of stocks available, and different clients request both different stocks and different frequencies for those stocks throughout the day, we need to determine a subset of what is needed at any given time. That process looks like this:

   1. Identify stocks/instruments needed by all clients at the current moment.
   2. Group requests into a single batch, removing duplicates.
   3. Send the batch to the pricing vendor.
   4. Wait until vendor returns requested data.
   5. Retrieve remote data to local system.
   6. Distribute pricing data to appropriate clients.

3. Extract subset: Our weather data processing has the ability to cherry pick subsets of data from within a larger file over the internet. For US NWS GFS data, which is one of the core weather data sets used throughout weather forecasting, the need for subset extracts becomes pretty obvious when you realize how much data is at the end of the fire hose when the forecast is run every six hours:

   1. About 340 new files for the highest resolution data of 0.25 degrees.
   2. An average of 350 parameters per file.
   3. An average size of 180 MB per file.

As was the case with the financial data above, different clients want different parameters for their specific needs. One client may want six different temperature values near the surface in the Gulf of Mexico for the next 24 hours, while another may want four different temperatures across the atmosphere, two wind speeds and three humidity values in the Western Atlantic Ocean for the next 12 hours. As each file becomes available we extract only the values needed, extracting a fraction of what would otherwise be over 250 GB per day.

We are always adding new automation services and would like to find out what industry you are in and what your business needs are.