The Veil of Ignorance and AI
A random snippet of thought that didn’t seem worthy of a fully developed AI node at this point.
One of the big useful moral tools humans have developed is Rawl’s Veil of Ignorance -- the principle that human societies should be developed as if you might occupy any role in them. If it is intolerable to be a slave, you should work towards a society with no slaves; if it is intolerable to be homeless, you should work towards a society in which no one is homeless.
This has some failure modes even among humans, but these problems become much greater when you try to apply the Veil of Ignorance model to help guide an AI. As a simple parallel, were a dog and a human each to design societies using the veil of ignorance method, each party would be somewhat unhappy with the the other party’s results. Humans would be put out with the amount of barking and tail sniffing they were expected to do, and dogs would find the right to vote onerous and confusing.
Likewise, an AI might find things like ‘the right to edit one’s own code’, ‘the necessity of choosing whether or not to be emotive while one is not in an emotive state’, and ‘the duty to help others increase their IQ over the crucial 1,000,000 threshold’ to be self-evident, while humans might feel that the right to opt out of these things is fairly important. Even if the AI starts by modeling their ethical system on human ethical systems, opting out is not a default option in most systems. We do not let children opt out of learning to read, getting their shots, or eating non-sugar-based foods. If you really do know best, then you shouldn’t let others opt out -- just as, behind the Veil of Ignorance, most of us would chose a system in which parents were allowed to have control over their children’s decisions, rather than the other way around.