The term High Performance Computing, HPC, evokes a lot of powerful emotions whenever mentioned. Even people who do not necessarily have vocational knowledge of hardware and software will have an inkling of understanding of what it’s all about; HPC solves problems at speed and scale that cannot be achieved with standard, traditional compute resources. But the speed and the scale introduce a range of problems of their own. In this article, we’d like to talk about the “problem journey” – the ripple effect of what happens when you try to solve a large, complex compute issue.
It all starts with data…
Consider a 1TB text data set. Let’s assume we want to analyse the frequency of words in the data set. This is a fairly simple mathematical problem, and it can be achieved by any computer capable of reading and analysing the input file. In fact, with fast storage devices like NVMe-connected SSDs, parsing 1TB of text won’t take too long.
But what if we had to handle a much…