The Rational Architecture

As a basis for analysis, let us use a simplistic model of the workings of an AI mind. The model simply divides the thinking of the AI into two parts:

1. A world model (WM) contains the sum of its objective knowledge about the world and can be used to predict the effects of actions, plan actions to achieve given goals, and the like.

2. A utility function (UF) that establishes a preference between world states with which to rank goals.

In practice, the workings of any computationally realistic AI faced with real-world decisions will be intertwined, heuristic, and partial, as indeed are the workings of a human mind. At present, only programs dealing with limited, structured domains such as chess playing are actually formalized to the extent of separating the WM and the UF. However, it can be shown as one of the fundamental theorems of economics that any agent whose preference structure is not equivalent to a single real-valued total function of world states can be offered a series of voluntary transactions that will make it arbitrarily worse off – even by its own reckoning! To put it another way, any agent that doesn't act as if it had a coherent UF would be an incompetent decision maker. So as we increasingly use AIs for decisions that matter, we should try to build them to match the model as closely as possible as an ideal.