3.1. Introduction¶
Science in the modern era is increasingly synonymous with scientific computing. Whether scraping social networks for data, analysing ever-growing databases of empirical results, training intuitive machines, reproducing nature in simulation, or even simply formatting a paper for publication, there would be very few scientists in the world today who have not found it necessary to develop at least some computing literacy [Wilson et al., 2014].
This revolution in the means of knowledge creation has arguably happened much faster than the adaptive timescale of the institutions it serves. Inefficiency, redundancy, and - all too often - the catastrophic loss of results, have all hampered the productive potential of this new technology. On the output level, new doctrines such as the FAIR program [Wilkinson et al., 2016] are beginning to address the issue of data persistence and access in principle, but are struggling in practice due to a worsening divergence of incompatible implementations [Jacobsen et al., 2020]. On the methodological level we find a proliferation of redundant solutions to related problems, a lack of reliable means of determining and disseminating best practices, and a state of alternating inability or unwillingness to absolutely require reproducibility after publication. Finally, on the most basic level, the skills portfolio of many workers is increasingly unequal to the challenges of devising, operating, managing, and packaging the ever-lengthening computational toolchains that best-in-field research now requires [Wilson et al., 2017].
With universities and scholars under pressure around the world, required to deliver more and more with less and less, it is clear that the engine of scientific progress is not sustainable in its present form. While new policies, better training, and stronger institutional supports all have a role to play in managing the symptoms of this pathology, the deeper cause is yet to be addressed. The scientific method today is simply too complicated for scientists. That is the challenge we must plainly recognise, and creatively overcome.
In this chapter, we break down what we call the ‘Complexity Crunch’ into several component issues, detailing some of the very promising steps being made by the community to meet these, while also identifying some persisting gaps in capability that we endeavour to fill (Motivation). We then turn to the matter of whether a general solution to the ‘Crunch’ can exist, articulating a doctrine that describes the necessary qualities of such a solution, which we dub ‘Abstractified Knowledge Production’ or AKP (Doctrine). Having captured the problem, we now present our prototype AKP instrument: ‘Everest’, an open-source framework (Framework) and Python-based implementation (Implementation) for high-level scientific computing. We demonstrate this new capability with a brief study of the iconic Lorenz Attractor (Demonstration). Finally, we discuss the shortcomings, lessons learnt, and future direction of our solution (Discussion).