September 06, 2012

The scale of progress: why we are overconfident about our ability to judge feasibility

Brain posterComprehensive Copying Not Required for Uploading | Accelerating Future - Michael Anissimov has a sensible response to PZ Myer's panning of my brain emulation whitepaper. I don't think I have much to add.

I think the core question is a matter of intuitions of scale and progress: if technology X requires a humongous amount of computations/devices/research/whatever, is that an argument for regarding it as abject nonsense? In some cases one can make a clear case it does (certain computations are not doable with the resources in the universe), in others it doesn't (storing a record of all citizens, the entire human genome or all webpages are all quite doable, yet at some earlier point likely looked entirely implausible). Often there is disagreement: people I respect regard the idea of taking apart planets and converting them into technology as total silliness, yet other people I respect think that it will eventually become feasible.

Normally we estimate the scale of work needed to do something, compare that to the available resources, and make a judgement of whether it is easily doable or not. This fails for future projects because the scale and the amount of available resources is uncertain. Worse, the resources may change over time. The standard example is computing resources, where fast growth makes previously infeasible activities doable (consider how statistical language translation has evolved, or the current spread of face recognition). New algorithmic insights might make resources even cheaper (consider the jump from an O(N^2) to O(N log N) algorithm). But the same is true for many material resources: getting large amounts of paper, books or rice grains is doable on a budget today while just a few decades ago it would have been very expensive.

AI probability density (skew gaussian and triangular)The obvious rejoinder to scepticism about the feasibility of something is to plot the relevant growth curve, extrapolate it and see when it crosses the line where the project becomes feasible. Doing it extra carefully, you might add plausible uncertainties in the growth and the resources needed, getting a probability distribution for feasibility dates.

But that still hinges on the lack of system-changing insights or changes: cryptanalysis can bring the date of cracking an encryption system into the present from a previously safely astronomically remote future. The discovery that the goal is not worth it or cannot be done in this way dissolves the whole issue. Trends and distributions of resources/demands work well when we are unlikely to have these crucial consideration discoveries. In domains dominated by progress through discrete insights rather than gradual refinement or growth predictability is low.


  • First, most people are bad at extrapolating into the future in any way. They will also have a very weak sense of the orders of magnitude involved, and underestimate the power of exponential growth. They will hence often assume that if something cannot be done at all today it can never be done.

  • Second, enthusiasts tend to assume there are not going to be any discoveries blocking things. Pessimists tend to assume there will be. Both then pick evidence to support their views, with the pessimists having the winning "but you can never plan for unknown unknowns!" (However, that goes the other way around too: there might be big simplifications around the corner - maybe P=NP or brains are really simple).

  • Third, discussions hence become more dominated by people's prior intuitions about the domain. People who think they understand an important sub-domain better than the original claimant tend to overestimate their ability to estimate overall feasibility.

  • Fourth, trend extrapolations is nice because you can at least attempt to include empirical data, but the reliability is not great. Looking at core principles that can prove or disprove things is much better. But as the previous points argue, they have to be actual core principles rather than just good-sounding arguments: arguments tend to be biased and people overconfident in their strength.

Here is my suggestion for what uploading sceptics should do: demonstrate the absence of scale separation in the brain. If all levels are actually dependent on lower levels for their computational functions, then no brain emulation can be done. But it is not enough to just say this is the case, you need to show it (using, presumably, methods from statistical mechanics, systems biology and other disciplines). Regardless of the answer it is a useful project.

Posted by Anders3 at September 6, 2012 06:18 PM