Contents

3.5. Implementation

We have discussed the motivations, underlying doctrine, and overarching framework of the Everest project. Now it must be shown how it actually works. In so doing, it will hopefully become clear just why the outermost user interface has the appearance it does, and why we have made the claims we have about the potential of this approach.

3.5.1. Object-oriented programming

Everest is written in a heavily object-oriented paradigm. As not all will be familiar with OOP, a couple of terms will be defined here.

In Python and other languages, an ‘object’ is a persistent software asset with an address in memory that can hold data in the form of ‘attributes’ and run snippets of code called ‘methods’. A program under OOP typically involves the creation, interrelation, activation, and destruction of an ecosystem of such objects to achieve the program’s intention.

Python accesses objects by stating their name, which is assigned in an entirely context-dependent frame: one object can be known by many names in many places. Both attributes and methods are accessed using dot notation - for example, myobject.foo accesses the ‘foo’ property, which may be either an attribute or a method. Methods, once referenced, may be ‘called’ to execute their code with zero or more inputs using curved brackets - for example, myobject.mymethod(myinput) -> output. Objects may themselves be callable: for instance, myobject(myinput) -> output.

Some objects are designed principally to ‘do’ things, others principally to ‘hold’ things. For container-like objects, Python provides a square brackets notation (also called ‘subscripting’) which has the modal sense of retrieving something from somewhere - for example, myobject[myslice] returns certain datas from ‘inside’ myobject according to the criterion myslice. The Pythonic operation of ‘slicing’ into an object in this manner is related to the AKP notion of ‘incision’; consequently, it will be seen that Everest syntax relies heavily on the square brackets notation.

Objects in Python and other languages always have a ‘type’, a kind of object to which they belong. An object’s type can be viewed by asking type(myobject). These types are called ‘classes’ and their names are usually capitalised - for example, MyClass. Every object has a type and is said to be an ‘instance’ of that type (what AKP calls a ‘concretion’). Just as a labrador is a type of dog, which is a type of mammal, which is a type of animal, at the same time as being a type of pet, which is a type of domestic animal, which is a type of animal, classes in Python can subclass and superclass one another, forming branching ‘inheritance trees’ which provide properties and behaviours to objects whose classes participate in them.

Class inheritance may be tested by asking issubclass(A, B), which returns True if A is identical to B or is a subclass of it. Objects may be tested for their membership by asking isinstance(myobject, B), which returns True if the class that myobject is an instance of is either B or a subclass of it. Sometimes it is useful to define a class that behaves like an object’s class without that object needing to be aware of it: Python supports these as ‘abstract base classes’ and they are frequently employed to define interfaces. It is a convention of OOP that instances of subclasses are accepted anywhere that instances of their superclasses are accepted, but not vice versa. For example, integers are real numbers, but the converse is not true; thus, functions which are valid for real numbers are valid for integers, but not the other way around.

Finally, in Python, classes are themselves objects, and so can own and do things without ever needing to be instantiated. Being objects, classes have their own type: the class that a class is an instance of is called its ‘metaclass’. Everest pushes the Pythonic class machinery much further than typical Python applications do in order to more naturally map code objects to AKP constructs.

The highly object-oriented nature of Everest is one major sense in which it deviates from previous approaches to large-scale computing. Whereas scientific computing traditionally has taken a procedural approach to computation, focussed on ‘piping’ data flows through shells on supercomputer clusters, Everest recognises that - for a work of scientific computing to be persistent - it must ultimately produce an ‘object’ of some kind. That object must be able to be stored in non-volatile memory - be that a hard drive or a reel of magnetic tape - and it must naturally offer a range of means of access so its import may be recognised in human minds where social knowledge is processed. Actions in Everest begin, rather than end, with the designation of objects, leaving the control flow behind those actions sufficiently muted as to almost reproduce a declarative (database-like) programming idiom. This is the soul of Everest.

3.5.1.1. Everest classtools

To support Everest’s high-order object-oriented approach, it was necessary to develop new tools for class definition on top of those provided within Python. The classtools subpackage provides a number of ‘adderclasses’ that ‘decorate’ (wrap or add functionality to) native Python class definitions, allowing for many features including automatic addition of ‘dunder’ methods, reproducibility enforcement, and more.

An AdderClass is an abstract base class that can be used to decorate other classes, directly adding its own attributes and methods in order to transform the subject into an implicit inheritor of itself. Adder classes are useful because they allow the separation of the functionality-adding and ‘quality’-altering natures of class inheritance, freeing up the true inheritance tree to more strictly reflect the latter over the former. They also lend themselves to metaprogramming (writing code that writes other code).

3.5.1.1.1. MRO classes

Everest provides and uses many adder classes, but the most important for this exegesis is the MROClassable class. This feature allows ‘inner classes’ (classes that are attributes, rather than instances or subclasses, of other classes) to be defined cooperatively across multiple layers of a class’s inheritance tree - and even include the owning class as one of their bases.

In Python and other object-oriented languages, the various classes that form the bases of an object’s type may offer different definitions for the same attributes, including methods. Rather than simply overwriting the parent bindings, objects and classes are provided access to those bindings in a separate namespace, enabling the subclasses to define functionalities that extend or augment the parent functionality. For classes with deep inheritance trees, the control flow of the program is successively passed up in a stable, linear succession called the Method Resolution Order or ‘MRO’.

When a class (the ‘added-to class’) is processed by the MROClassable adder, the adder looks for certain class-defined special names and, for each name, loops through the MRO chain for any inner classes assigned to that name. These classes are then used as the bases of a new class, which is finally assigned to the added-to class under the original name. As an optional extra, the added-to class can itself be factored into the new ‘MRO class’ as either a base or a metaclass.

The ‘MRO class’ procedure has many uses throughout Everest. For just one example, the Case class is a ‘metaclassable MRO class’ of the schema it belongs to. This allows the schema to enforce a condition that all instances of itself are subclasses of its own special case class - and preserve this property even for its own subclasses. While there are other ways to achieve the same effect in Python, none are so compact.

3.5.2. The Incision Protocol

Everest extends the traditional Python subscripting (square brackets) notation to make it both more flexible and more strict. The new notation is called the Incision Protocol, to differentiate it from conventional Python ‘getting’ and ‘slicing’ protocols. All public Everest objects which support subscripting do so according to this protocol. To those with some experience of NumPy, Python’s primary numerical computing package, the syntax will be familiar: it is effectively an extension of NumPy ‘fancy indexing’ with dimensions that may arbitrarily typed, irregular, infinite, or uncountable. Fundamentally, its intention is to generalise Pythonic ‘getting’ syntax to support the broader sense of an object containing having ‘space’.

In mathematics, a space can be loosely defined as a set with some added structure. Beyond that, the concept of space is defined in many contrasting ways for many contrasting purposes. For clarity, let us instead appropriate the Greek equivalent, ‘chora’ (plural ‘choraes’), and defines it as a set equipped with a mechanism for retrieving one or more elements. The Incision Protocol realises this definition with the Chora class, a type of immutable Python set with sockets that anticipate the imposition of structures by its subclasses. The Chora class is both the simplest type and the base type of all choraes recognised by the Incision Protocol; any instance of such is a chora, lowercase.

In its basic use case, the Chora class can be used just like a standard Python set by instantiating it with a collection of unique, arbitrary elements. Chora, however, has two crucial differences.

Firstly, instances Chora are not limited to contain finite sets. They can just as easily digest sets of unknown or even infinite length. The set of every possible combination of alphanumeric characters is a perfectly acceptable chora, as is the set of all real numbers. Many such useful sets are defined ‘out of the box’, most built on fast C level protocols that are much more powerful than typical user implementations. A consequence of being both unordered and potentially unlimited is that a basic Chora instance is not iterable, thus procedural access can only be carried out by random sampling without replacement.

Secondly, Chora instances support element retrieval by passing a request through the square brackets notation: mychora[myquery] -> mycontent. Under the Incision Protocol, subscipripting operations on chora are called ‘incisions’, and any input to an incision is called an ‘incisor’. There are three kinds of incisor:

  • ‘Trivial’ incisors simply return the chora incised, or alternatively an identical copy of it; they are the equivalent of ‘retrieving’ every point in space, thus simply reconstituting that space under a new name.

  • ‘Broad’ incisors return a view of the chora with added conditions and retrieval methods: in other words, they return a subspace of the incised space with new metrics attached.

  • ‘Strict’ incisors return a single element from a chora, or - put another way - prescribe a vector singling out a particular point within the chora.

The basic Chora class supports three retrieval methods - one for each of the basic incisor types:

  • Retrieval of its complete self by trivial incisor, returning itself.

  • Retrieval of a single element from itself by equality with a strict incisor.

  • Iterative retrieval from itself by the same process across the elements of a broad incisor. The Chora class offers one more standard behaviour as part of the strict incision operation. If a retrieved element is itself a chora, the return value of the incision is instead the incision of that chora by the incisor. This special kind of strict incision is called ‘deep incision’, and the incisor carrying it out is a ‘deep incisor’. This additional behaviour was a significant part of the motivation for the Incision Protocol and it has many implications, as shall be seen.

While users are free to subclass Chora according to their purposes, Everest offers many subclasses of its own.

  • The Ordered class adds the sense of ordering of elements, and hence the notion of Python ‘slicing’, implemented here as a kind of broad incision across a given range; for example, myordered[cond1:cond2:cond3] would return the chora of all elements of myordered matching cond3 after and including the first element satisfying cond1 and up to but not including the first element satisfying cond2. A special subclass of Ordered is the Reals class, the set of all real numbers. While Ordered chora can be sliced, they still cannot be iterated nor indexed. These properties are added by subclasses:

    • The Countable class, which imposes countability on its elements, and thus supports iteration over its bounded subspaces, though not over itself. An ordered chora ‘sliced’ with a ‘stride length’ returns a countable chora. A special subclass of Countable is the Integers class, the set of all natural numbers.

    • The Limited class, which allows the assertion of either a ‘highest’ element or a ‘lowest’ element or both. An ordered chora ‘sliced’ with either a start or stop index returns a limited chora. A doubly-limited chora supports explicit and reproducible random sampling by slicing with a random seed. For example, myreal[0:1:'aardvark'] returns an infinite, countable sequence of randomly selected real numbers between zero and one; any two such operations with the same random seed (e.g. the ‘aardvark’ string here) will return the same sequence.

    • The Tractable class, which subclasses both Countable and Limited and therefore supports full Python still count-based indexing - for example, mytractable[2:5] would return a Tractable traversing the three elements from position two to position four inclusive. An ordered chora sliced with a stride length and one or both of a start index and a stop index returns a tractable chora. The addition of both limits and countability permits iteration, making Tractable an implicit subclass of the native Python Iterable base class; however, only tractables with lower limits can be successfully iterated across.

  • The Transform class takes any other chora and applies a given unary function to every element.

3.4. Framework 3.6. Demonstration