Judge Entries

Enter how likely you think this essay is to be real.

Astronomical odds and the limits of quantified ethics

When small systems designed to deal with small numbers are tweaked to include very large numbers, absurd things happen. This is one of the most fun facts about the natural world, especially physics, and it is also a very fun fact about ethics and morality. Human values evolved in a world where the concept of a "million" or a "billion" was completely meaningless, and where things that required astronomical odds were things that did not happen. It is not intuitively true that the certainty of one human's death is ethically equivalent to a one in seven billion chance of wiping out the species (leaving aside the idea that the continuation of humanity as a whole is its own ethical imperative beyond the lives of its components, which I understand is a subject of disagreement). I would argue that it is not obviously true in any meaningful sense at all. Quantifying human values at a macroscopic, easily tangible level is one thing, but using the same system to process huge numbers that it wasn't built to process can produce some ridiculous and hilarious results, which have a tendency to turn human values into very inhuman consequences. Luckily, it's not the sort of decision the average human has to make. It is the sort of decision a super-intelligent AI might have to make, but we kind of expect those vastly superior to us to make bafflingly alien choices that we don't understand or feel any responsibility for, so it'll probably be okay.

Your current score is 0%. The current average score for all users is 56%