Judge Dread? Or welcome to my first LLM-related post.
As LLMs and especially coding agents became part of our daily workflows, we started relying more on actual judgement of models we use — judgement, to me, is any decision they default to based on their training and pre-processing without humans actually directing them with prompts or skills.
It might be a simple decision about using tabs or spaces, or a more complex one about how to structure your tests and test fixtures, but you can already get a lot done by offloading decision-making as well.
While this is fast improving, I wanted to test my
understanding by asking them to do an extremely common test by
Simon Willison of generating a vector image of a
pelican riding a bicycle,
but with a twist: instead of asking for an SVG,
ask for a conceptually different vector format
PostScript (really, a programming language), which is
closer to the plotting/printing model from 80s, with instructions
like moveto, Y-axis actually going upwards, for loops,
matrix transformations etc — enough difference in semantic
interpretation and less training data to throw a frontier model off :)
My expectation was that they would be a generation or two behind compared to what Simon is getting with SVGs.
I'll let you be the judge if my foresight was right or not.
Prompt: Generate a native PostScript image of a pelican riding a bicycle.
Some quick notes
- I tried converting PostScript files with ImageMagick first, but after seeing it miss a few things during conversion, decided to use GhostScript loader from within Inkscape instead and export a PNG out.
- Gemini 3.7 Flash had a broken first version which it was able to fix with the Ghostscript error message passed back (I think it was wrong parameters for ellipse command).
- Opus 5 took around 4 iterations of rendering to a raster image and reviewing (also hitting a PS syntax error), but managed to improve with each cycle. I did not check the costs, but it was probably expensive :)
- My first few attempts were with a slightly different prompt of "Make an PS or EPS (PostScript) of a pelican riding a bicycle" ("an" typo included — I probably went for EPS first and never corrected the language). This resulted in greyscale images in ChatGPT confirming that this might be worth looking into. I did not approach it scientifically at all, but switching to "Generate" to match Simon's version always produced a colorized version.
- All of the text in this post was hand-typed, including the em-dashes.