This is the one set of legacy APIs I didn't implement in minisphere, the Complex primitive.  Mainly because I'm not sure what it is, exactly!  Best I can tell from the documentation, it renders a circle inscribed inside a rectangle, but then I read more and it seems more complicated than that.  Either way it hasn't been a big loss as I haven't run into a game that uses it yet, but I'm still curious what purpose it serves.

I made a platformer that I never widely released (still have it) that makes use of that complex effect to do a "loony toons" like screen effect where a hole consumes the screen, pinching it out to blackness. It's an effect that would have been hard had it been implemented entirely in code.

It might have even been used for some torch lighting effect in caves for some games. Like in Pokemon when you are in the dark cave and all you see is a circle about the player character.

I think if you coded the effect to only cut a hole in a rectangle you've covered most of it's practical use cases that I have seen.

It sounds like you should really be able to cut any kind of shape out of a rectangle, not just circles. Take the Bowser cut-out in Mario 64 or the different cut-out shapes in Banjo Kazooie, for example.

But is this all really necessary to have as a feature? Can't you just have a picture of your cut out and zoom that in? The use case seems so limited...

Re cut "any" shape from a rectangle - I'm pretty sure N64 games had an alpha mask bitmap that was scaled and subtracted from (added to?) the current frame. I had a couple of Sphere tech demos that used that very technique on Surface objects (I think using particle images to mask a Link from LOZ sprite sheet or something).

Re Looney Toons iris, primitive shape cutouts - It is indeed very difficult and tedious to do without a properly implemented Complex, though I can't say I used it myself in anything due to my own lack of understanding of its usage. I once wrote an ellipse iris in/out transition for NTrans using 4 outer rectangles (to cover everything outside the radii) and a series of one-pixel thick ellipses (even before a native Circle/Ellipse primitive) to cover the rest. It was not a fun time.

I highly recommend further research into the Complex shape so we can do primitive/polygonal cutouts natively.

I think the best way to do this is to just use Surfaces. When I did my last LD entry, that's how I drew a black rectangle with a transparent circle cut out.

Fun fact, the prospect of implementing a Complex was what made me think that the Sphere drawing API was a bit bloated.

The only thing that concerns me with using a surface for this effect is that you can't reuse the image--you have to re-render the cutout every frame as it gets larger/smaller.  You can't do the usual Surface optimization where you draw the surface once and then convert it to an Image for fast compositing, it has to stay as a Surface.  Which would probably hurt performance compared to rendering the primitive normally.

The only thing that concerns me with using a surface for this effect is that you can't reuse the image--you have to re-render the cutout every frame as it gets larger/smaller.

Sure you can. Well, mostly you can. You can render the largest size necessary, and scale it down. If you really want to limit the Surface's size, you can use extra boxes on the edges.
There surely are cases where this would be worse in some ways, but in many cases would be more than enough.

You could do a very fast real-time approximation with polygon filling. More accurate the circle, the more polygons you need.

I guess you could say the effect is... complex.