Minecraft already knows the 3D position of every entity and where your camera is looking. ESP just converts the enemy's 3D position into 2D screen coordinates using the game's camera/projection system, then draws the box there.
For aim, it calculates the direction from your player's eyes to the target and converts that into the yaw/pitch the camera needs to look at. It doesn't necessarily move your physical mouse. It can directly change the player's rotation.