For a walkthrough of the many logical fallacies of believing computers will by themselves evolve into a "superbeing", I encourage you to read the first chapter of Moral Codes by professor Alan Blackwell.
The chapter also contains a great primer on the two main different kinds of AI. The first kind is what used to be called "cybernetics", or "control systems", with a sensor (like a thermostat) controlling an output (like a heating pump) . And the second kind of AI being the one that intends to imitate human behaviour for its own sake.
As Blackwell writes: "Many public discussions of AI do not acknowledge this distinction between the objectively useful and (sometimes) straightforward engineering of practical automated machinery, and the subjective philosophical enterprise of imitating the way that humans interact with each other."
Blackwell goes on to argue that many AI researchers have blurred the lines between computing and philosophy, without formal training in the latter. There is a great deal of magical thinking going on, with no basis in scientific reasoning.
«The magical powers of genies and deities are not precisely defined in stories, because clear definitions would limit the power of the narrative device. If anyone did commit to a technical definition of AGI, the contradictions would become clear, because the definition itself has to be coded in some way. Of course the answer can’t be a circular one - you can’t say that the necessary code definitions will be written by the AGI itself! Intelligence that could manifest itself in any way, without the inconvenience of a clear definition, is a plot device like the shape-changing T-1000 of Terminator II which might be described as an “artificial general body” in the way it is unlimited by any definition of shape, and thus able, in the imaginary universe of the movie, to become anything at all.»
Read the chapter here.
https://moralcodes.pubpub.org/pub/1mn2q39n/release/6
And by all means, keep on reading after that 😉