Minecraft already knows the 3D position of every entity and where your camera is looking. ESP just converts the enemy's 3D position into 2D screen coordinates using the game's camera/projection system, then draws the box there.
For aim, it calculates the direction from your player's eyes to the...