Do we need deterministic AI?

Prompt: "Output a name of one of the states of the United States."

It will not tell me the names of the planets. Or the names of prescription drugs.

Why is this level of determinism not enough?