[Coda]

The Zeroth Law, in Asimov's writing, is an advanced form of the First Law that sufficiently advanced robots can formulate on their own. It states that a robot may not allow harm to come to humanity as a whole, or through inaction allow humanity to come to harm. This was an outgrowth of how AI had to reason about how a small harm to a human could prevent a larger harm (for example, punishing a child caused the child to be upset, which is harm, but it taught the child how to prevent bigger injuries).

Regarding AI and diseases: Any treatment that an AI comes up with would have to go through the same evaluation procedure that human-created treatments do. If a test population improves without side effects, you move on into further tests and then on into mass production. If a test population suffers horrendous side effects or gets worse, you abandon the drug and go back to the drawing board. So in that regard, AI is really just a tool for researchers to use in pursuit of what they're already doing.

Nanotechnology is a separate argument from AI. In particular, the problem with nanotechnology is the fact that they CAN'T have robust AI because they're tiny little things that act more like chemistry than like robots. There's no room for advanced computation. So while it's a worthwhile avenue of discussion, it's a DIFFERENT avenue of discussion.

Re: whacking metal with a crowbar -- You'd be surprised how much you can learn from whacking metal with a crowbar. It's a crude analogy, but the same techniques human engineers use to evaluate stresses in human-designed structures will work for evaluating stresses in AI-designed structures. The AI isn't going to invent brand-new laws of physics (unless they do! that would be super interesting!) -- the more likely surprise is going to come by developing a novel way of using things that can subsequently be analyzed to see what you can learn from it. That's how medicine has ALWAYS worked -- we discover molecules with medicinal properties first, then we start researching the pharmacology behind how it does its thing. Going the other way around was a flop -- pharmacologists spent a couple decades doing "rational drug design" and trying to engineer molecules specifically to pursue a hypothesized mechanism of action... it didn't really work any better than the typical method of "let's take this active molecule, swap out parts of it, and see what happens" and it cost a lot more. So an AI doing rational drug design would essentially be creating new base molecules for pharmacologists to study.