The Mapterhorn terrain pipeline takes as input elevation data from around the world and outputs an aggregated elevation dataset for interactive web map visualizations. By its nature, the pipeline parallelizes well spatial with fully independent tasks that cover different areas. So far, the pipeline was run on a single host with an AMD Ryzen 9 7950X cpu and a pipeline run took roughly 2 weeks. But since the pipeline will be expanded to aerial imagery with potentially much more input data, we did explore how tasks could be distributed to multiple hosts. Here we will share some insights into how one can find cheap compute on the second-hand market.
The pipeline has two main stages: aggregation and downsampling. In the aggregation stage, input data is read, reprojected, and smoothly merged at the local maximal zoom level. The reprojection, merging, and encoding to webp images are all cpu-intensive tasks. The 7950X processes roughly 100 GiB of input data per hour in the aggregation stage.
In the second stage, maximal resolution data is downsampled to lower zoom level. The resolution is halved at each step. Here the main compute intensive task is again the webp image encoding. The downsampling stage takes roughly as long as the aggregation stage.
The aggregation and downsampling was already parallelized for a single host with multiple threads. Then we added a small Manager/Worker mechanism which allows multiple hosts to collaborate. The main idea is that a central host, in this case the existing 7950X, holds the input data on an SSD and shares it via a network Samba folder over LAN with other hosts. The main host runs a Manager script which listens to task requests from Workers. Each host in the cluster runs a Worker script which populates the cpu with threads that request tasks from the Manager and execute them.
Sharing data over the network should not be a bottleneck because 100 GiB per hour is only like 28 MiB/s on average.
It is probably a good idea for economical and ecological reasons to work with used hardware. In Switzerland there is a nice second-hand portal called Tutti.ch and people sell all sorts of stuff there. Choosing the right components is a bit tricky but websites like cpu-monkey.com are really helpful for comparing the multithreaded performance of cpus.
It turns out that only very few systems that are sold compete with the performance of a 7950X. One great offer that we bought for CHF 650 was an AMD Ryzen 9 8945HX in a mobile-on-desktop configuration from Minisforum with a fan and 32 GiB DDR5 memory included. It was placed in a spare case with a CHF 60 power supply and an SSD from an old laptop was added. And ready was the first additional host for the new Mapterhorn compute cluster.
Initially we connected the two hosts with a direct LAN cable, gave static IP addresses, and mounted the Samba shared folder. It turned out the second host reaches close to 100 percent cpu load after some minutes of initial data loading. Cpu-monkey.com states that the multithreaded performance of the 8945HX is about 80 percent of the 7950X. And indeed a test showed that the aggregation stage speed of the 8945HX host was almost 80 percent of the 7950X.
Both machines have onboard 2.5G ethernet. To add more machines we bought a 5 port 2.5G switch for CHF 50. Network equipment in general seems to be quite cheap.
But that 8945HX was rather a lucky buy and similar offers did not show up for quite some time. However, there is a nice second-hand market around Xeon cpus.
Xeon cpus can have many cores and have two threads per core. Attractive from a performance per price point of view are in particular the Xeon E5 26xx v4 and the first generation Scalable ones such as Xeon Gold. Those work with DDR4 memory which is significantly cheaper than state of the art DDR5 and still not as old as DDR3. Also, there are mainboards that have two sockets so you can have dual cpus in a single host.
Performance-wise the Xeon E5 2680 v4 e.g. is at about 36 percent of the multithreaded performance of a Ryzen 9 7950X. So with two you get to about 72 percent. The Xeon Gold 6138 e.g. gets you with one cpu to 48 percent so two of them are basically the same as one 7950X.
The E5 2680 v4 has 14 cores / 28 threads and the Gold 6138 has 20 core / 40 threads. The Mapterhorn pipeline uses around 1 GiB of memory per thread, but maybe that can be reduced.
Price-wise those cpus are sold around CHF 30 for the E5 and CHF 80 for the Gold, but finding actually those models is a bit tricky. More often you will see the models with fewer cores.
The E5 looks in principle like a good option but desktop machines that support two cpus are a bit rare on Tutti. Servers with dual cpus would be available but those are a bit too loud for an office use case which the Mapterhorn cluster currently is.
But for the Gold cpus there is a seller on Tutti which has multiple machines and cpus and can configure your hardware as needed. He just sold us two ThinkStation P920 with dual Xeon Gold 6138 for CHF 750 each RAM excluded. DDR4 sells at around CHF 2 per GiB, we will see how much is needed.
One obvious but somehow still remarkable fact is that if you double your processing power, the execution time is halved if everything parallelizes perfectly. For the Mapterhorn terrain pipeline that means that we cut the 14 days duration to roughly 8 days by adding the second 8945HX host. So we saved 6 days of execution time by adding one machine. But now with the second step of adding the two P920s, we might cut the time in half again going from 8 to 4 days, but we only saved 4 days with two extra machines. Thinking that further, you would save 2 days with another 4 machines u.s.w.
So with a fixed compute load your pipeline execution time improvement per additional host gets smaller with each host you add to the cluster. It is logical but still somehow surprising. Probably as soon as your pipeline duration is on the scale of a few days it is not worth adding more compute any more.
Curious to see where those 4 machines will take us.