>What I think is the same philosophically though, is ceding the power over fundamental activities/interactions for functionality.
I'm not sure what this means. If you're referring to the fact that groups are usually required to make everyday society work, that's not a useful example of ceding power. Everyone at the train company has essentially welded their power together in a Captain Planet or Voltron sort of deal to make rail transport work, but that doesn't make anyone else less powerful. It makes the train people highly interdependent both on each other and on everyone else to use the trains to go to the farms or the medicine factory or whatever and make the other parts of society work. They can't tell everyone else to sod off because they've got some magic spell that makes the trains work. The trains are run by regular chumps without any magic, and any other regular chump could be made to replace any one of them. Even if they were special, their special-ness wouldn't make them any less dependent on anybody else since they still just run the trains. By contrast, the Minds are totally required for the Culture to maintain their shiny post-scarcity status quo, no human effort could ever replace the Minds, and the Minds are not dependent on the efforts of any humans. The Culture's people have actually ceded power over all sorts of everyday stuff to the Minds. The Minds could tell everyone else to stuff it if they wanted to, and nobody would be able to do anything about it.
>When real AGI comes, if it comes well, then it has the possibility to look like the pinnacle of functionality, just like our tools are now. So it is definitely exponentially more powerful, but for the average person it will look like magic, much like most technology does now to the under-informed.
>...and given that we would have built it, it won't be black magic either - just so totally far removed from the average person that it will look like magic.
How much it resembles magic to the under-informed is irrelevant. The under-informed aren't uninformable, they just haven't been sufficiently informed yet. I know lots of people that are convinced they can't be sailors because spending weeks on a ship just seems totally beyond them, or that they can't be physicists because they had a hard time with calculus in school. It seems to them that those tasks require some ineffable qualities that can only be found in others, but they're wrong. Both calculus and seamanship are challenging to comprehend and highly impressive when applied, but they aren't "unknowable," they're just "unknown." Any given chump could learn to pull them off if they weren't doing something else. It's only unknowable magic if they couldn't ever figure it out, or if no group of people like them working together could ever replicate it themselves.
I'm not sure what this means. If you're referring to the fact that groups are usually required to make everyday society work, that's not a useful example of ceding power. Everyone at the train company has essentially welded their power together in a Captain Planet or Voltron sort of deal to make rail transport work, but that doesn't make anyone else less powerful. It makes the train people highly interdependent both on each other and on everyone else to use the trains to go to the farms or the medicine factory or whatever and make the other parts of society work. They can't tell everyone else to sod off because they've got some magic spell that makes the trains work. The trains are run by regular chumps without any magic, and any other regular chump could be made to replace any one of them. Even if they were special, their special-ness wouldn't make them any less dependent on anybody else since they still just run the trains. By contrast, the Minds are totally required for the Culture to maintain their shiny post-scarcity status quo, no human effort could ever replace the Minds, and the Minds are not dependent on the efforts of any humans. The Culture's people have actually ceded power over all sorts of everyday stuff to the Minds. The Minds could tell everyone else to stuff it if they wanted to, and nobody would be able to do anything about it.
>When real AGI comes, if it comes well, then it has the possibility to look like the pinnacle of functionality, just like our tools are now. So it is definitely exponentially more powerful, but for the average person it will look like magic, much like most technology does now to the under-informed.
>...and given that we would have built it, it won't be black magic either - just so totally far removed from the average person that it will look like magic.
How much it resembles magic to the under-informed is irrelevant. The under-informed aren't uninformable, they just haven't been sufficiently informed yet. I know lots of people that are convinced they can't be sailors because spending weeks on a ship just seems totally beyond them, or that they can't be physicists because they had a hard time with calculus in school. It seems to them that those tasks require some ineffable qualities that can only be found in others, but they're wrong. Both calculus and seamanship are challenging to comprehend and highly impressive when applied, but they aren't "unknowable," they're just "unknown." Any given chump could learn to pull them off if they weren't doing something else. It's only unknowable magic if they couldn't ever figure it out, or if no group of people like them working together could ever replicate it themselves.