Science!

That is the direction from which Arthur C Clark looked at it. There is no explanation because it's utterly beyond the observer.

Consider a house with voice activated features. "abracadabra" and there is light, music, a fire, whatever. Bring a medieval person to it.

But you don't have to use people from the medieval era. Just because people have learned to take technology for granted doesn't change it in any way from having learned to take magic for granted.
Consider a smart phone. Few people can explain how that works. They think they can explain it: "It's a screen, a processor, radio waves"...  But actually they can't.  Their explanation doesn't add up to much more than "It's technology" == "It's magic".
And they can't make one. Just as they can't make magic.

The magician otoh, if he can do something that has an intentional and predictable consequence, then he does no other thing than applies some physical laws. It doesn't matter what you call it, "midichlorians", whatever, because, just as you said, it then has to belong to physics, even if it's made up physics.

But it also doesn't matter if you don't call it something. Just as it doesn't matter if you don't try to explain it. It's just that it's beyond you. But cause and effect must be essential for the magician, or it wouldn't work when, if and as he wants. So the only difference is that you don't try to *explain* it. Very convenient.