There is another modification that would improve the accuracy of the final shape a lot. As the paths are built, instead of just adding two points on each side of the path, divide the circle circumference into N equal segments, and add a point for each segment (with the new points of course oriented relative to the path direction just as the current two points are). All thre points on one half of the circle belong to the green line, all the points on the other side belong to the red line. The line will look really ugly before point deletion, and this will slow down the step of deleting points, but the start and end of the tool path, and the tool path for any sharp turns (such as the 90 degree turns in the example above) will be much more accurately represented. The higher N, the more accurate the path, but the slower the algorithm. This could cause weird artefacts if successive circles are too far apart, but if that happens either more circles could be added, or N could be reduced.
Also, instead of marking the points to be deleted as a NaN, it might be faster to delete them on the fly. That way, they don't get checked a second time. The relative speed of the two approaches probably depends on the shape of the path though.