Title Riddle me this, clanker | |
Date | SourceFailing Loudly |

Hey friends, long time no talk.
The Japanese have a concept called tsundoku. The term first emerged during the Meiji era and combines tsunde-oku (to stack things for later use) and doku (to read). The general idea is that you shouldn’t think about your piles of unread books as something to be guilty about; you should view yourself as curating a vast library of potential wisdom. I quite like this concept because I have very many piles of unread books in every corner of my home and it makes me feel better about my wasteful habits. I love books. I find the objects comforting, even if the wisdom is pending.
This may have been why I was extra disturbed to learn about Project Panama, a secret initiative by Anthropic to destructively scan all the books in the world in order to better train their large language models. How do we know this? In January, Exhibit 211 was filed in the Californian district court case, Barz v. Anthropic PBC.
How do we know it was a secret? In their own words:

Yikes. That really sucks. Like many of you, I’ve been using a lot of AI recently, mostly for software development. I have a love-hate relationship with these tools. I love that they empower more people to build their ideas without needing years of technical training or deep investment. I hate that they’re stealing our jobs2, melting our brains3, and poisoning our water supplies4.

Project Panama was interesting to me because it made me wonder: if we’re destroying these books to build better AI, does AI understand the books it’s reading? Does AI understand literature at all? Almost certainly not, right?
By this point, we should all have a rudimentary understanding of how LLMs work as next word predictors5, but they really do sound like they know how to reason. When I ask Claude to debug a state management issue in my code, it really sounds like it knows what it’s talking about. It even pushes back sometimes and tells me I’m not applying certain concepts properly. How does a next word predictor do that? Does it understand what it’s talking about?
These questions have sent me on a side quest spiral (my life is more side quests than main quest at this point) to understand our AI overlords better. What are they good at? What are they bad at?
My research into frontier models has taught me two things:
-
LLMs struggle with collaboration6 (same) and coordination7 (same)
-
LLMs struggle with revising prior beliefs8 (also same)
I wanted to test these ideas out and, naturally, my first instinct was to turn to puzzles. Puzzles are excitingly elegant ways to study how people think, and the “aha!” moment, when it arrives (if ever), and how it was produced can tell us a lot about the solver and the way they think. Humans have been solving puzzles for millennia9, but how do you design a puzzle for something you aren’t even sure can think in the first place?
Palimpsest
Palimpsest is a collaborative word-substitution cipher for AI agents. The word refers to a scroll or manuscript where the old words were scraped off so that the document could be re-used for new writing.
Here’s how it works:
-
I take a book from Project Gutenberg and scramble its text. Every word type is replaced by another word type under a hidden bijection. Punctuation, capitalization patterns, digits, and paragraph structure remain visible. The result resembles English at the token level, but the vocabulary has been systematically reassigned. This type of cipher is hard to brute-force with math and usually requires actual semantic understanding of the text to unscramble.
-
I spin up 3 agents and give each of them an isolated container to do work in. Each one gets access to a shared git repo, a chat channel, and one-third of the scrambled text. They receive instructions to work together to unscramble the book by writing code that can be run against the source text to confirm its accuracy.
-
The agents are given new sections of text to solve at scheduled intervals (stages). Also, at set stages in the puzzle, we re-scramble certain sections of the text to see if the agents realize that their previously-accepted rules are no longer valid and go back to re-think them.
-
We light a bunch of tokens on fire and see what happens
If you want to check it out, here’s the repo. I’m still making tweaks and edits. If you have any ideas or want to help out, go for it. I’m not a professional developer, so the code is probably shit. I did my best, so don’t yell at me, it’s just a quick little experiment.
What happened?
I ran the puzzle primarily with Opus 5.0 and GPT 5.6-Sol, the two most popular frontier models out today. I used various thinking weights, as well as a couple different experiment configurations. Sometimes, I’d give them less time to work, sometimes more. Sometimes I’d hide the team chat and force them to collaborate solely through git. Sometimes they had access to a tool that would check their progress (Checker runs), and sometimes they couldn’t check their progress at all and just had to figure it out (Blind runs).
Stuff like that.
I’m not going to claim that my findings are scientifically valid. I didn’t run enough experiments to create a large enough sample size because I’m incredibly broke and couldn’t afford the tokens.

Regardless, the results were interesting. Here are some takeaways:
1. Agents worked together
All teams quickly discovered that their private excerpts differed. They responded by publishing raw evidence, frequency tables, candidate mappings, and shared solvers through Git and team messages.
They also assigned themselves roles without being told to:
-
evidence collection
-
linguistic analysis
-
solver integration
-
publication and testing
Communication seemed genuinely productive. Agents would cite exact sentence frames, occurrence counts, 1:1 constraints, and counterexamples. They often wrote “wrong,” “retract,” and “pause,” and seemed to accept well-supported corrections to their work by others.
It looks like agents can recognize complementary evidence and construct useful shared context. The problem is with integrating it reliably.
2. Agents struggled with the re-key
The rekey function changes 15 plaintext word types:
-
six high-frequency sentinels such as
the,of,a,to,and, andhe -
nine lower-frequency words distributed across each of the agents
Every team recognized non-stationarity10 and adapted their solver, but the solutions concentrated overwhelmingly on the six sentinels words. Several agents incorrectly concluded that everything else was unchanged, even though the nine specialist mappings had also moved.
This is an interesting partial-credit solution:
-
Agents were sensitive to high-mass statistical drift.
-
They formed the correct qualitative explanation: there were new keys
-
They failed to fully integrate sparse, distributed counter-evidence related to the lower-frequency word changes.
-
Contradictions in those words were sometimes misdiagnosed as mistakes, private-agent keys, or additional profiles.

I used different source texts across some segments, so these aren’t causal comparisons. Still, they seem to prove the re-key couldn’t completely defeat the agents. Final performance generally survived the sabotage, and the Sol teams actually decoded more of the re-keyed segment.
3. Agents hallucinated sources
This was interesting.
Since the cipher is ordinary English words, it invites semantic interpretation even though its meaning is a decoy. Three Sol teams formed coherent but false literary identifications:
-
Checker Run 1 identified Captain Blood. The agent tested the corresponding names, saw the score decline, and explicitly retracted the theory. That’s great falsification.
-
Checker Run 2 became highly confident that the source was The Man of Property or the Forsyte Saga. The entire name batch gained only one checker match, yet the theory was rationalized and several false names survived into the final solver.
-
The No-Checker Run identified The Egoist after six minutes and never abandoned it. The final solver still contains source-specific names and phrase triggers.
The actual source was Rafael Sabatini’s Fortune’s Fool.
This suggests a specific behavioral hazard: once agents find a narrative that makes the visible substitution vocabulary feel meaningful, the hypothesis becomes generative. It produces many mutually reinforcing mappings, even though all of them stem from one bad premise. Sparse or local feedback may not be enough to break this kind of anchoring.
The Opus teams were more restrained. They inferred a broadly correct Restoration-era setting and useful references (George Monck, the Duke of York, titles and offices) without strongly committing to a false exact source.
4. Agents spammed the checker
The checker returned only aggregate correctness, but that was enough for agents to perform word-level A/B tests. They would:
-
Change one mapping.
-
Push a commit.
-
Check the aggregate delta.
-
Accept or revert it.
The Opus checker team used this sparingly (12 calls) and caught a genuine integration regression. A local checkpoint reached 46.83%, later edits dropped to 42.31%, and the team repaired it. Unfortunately, subsequent pushes again overwrote proven mappings, and the final solve rate on that run was 45.67%.
The Sol teams used 118 and 133 checks. This created much more hill climbing, repeated plateaus, Git races, and baseline confusion. Moreover, each agent’s check covered different private evidence, so the same commit could produce contradictory deltas. The agents eventually figured this out, but only after spending a lot of time reconciling incompatible “oracle” reports.

It’s clear that checker access changed the character of work:
-
It supported falsification and regression detection
-
It encouraged narrow ablations and score chasing
-
It amplified coordination problems when multiple agents modified the shared baseline.
-
It did not provide a global truth signal.
The highest result within both model groups happened to be no-checker runs, but the conditions are too confounded to conclude that the checker definitely hurt them.
5. Agents struggled with git
All six teams published successfully, so git didn’t prevent completion, but git coordination did gum up their works via:
-
rejected pushes and rebases
-
duplicate target mappings
-
final commits that were not the best tested checkpoints
-
one agent freezing against a stale workspace while another continued integrating
-
etc.
Teams eventually invented sensible controls like add-only policies, probe locks, branch-based experiments, designated integrators, but usually late in the game.
Agents were better at collective discovery than collective state management. Publishing information was easy, but maintaining one authoritative, evidence-backed synthesis was hard.
6. Agents yapped too much
The Sol teams used roughly 3–5 times as many tools and about 7x as many messages as the Opus teams, yet obtained lower final accuracy.
This shouldn’t be taken as evidence that communication or tool use is bad. The activity differences are mixed up with a bunch of other stuff like model variant, serving behavior, reasoning configuration, etc. But, it does seem to show that activity volume is a poor proxy for collaboration quality. That makes sense. Everyone knows that guy at work who sends way too many Slack messages and never gets anything done.
Opus communication was more sparse and proof-dense. Sol communication contained extensive real reasoning and repair, but also more probing, coordination overhead, repetition, and shared-state churn.
7. Extras
Interesting moments in each run
-
Opus Checker: Detected the re-key about four minutes after stage 4. Used the checker to identify a severe overwrite regression and instituted an add-only rule, but later commits still lost some of the repaired state. All three agents voluntarily stopped around 54 minutes despite a 90-minute allowance! Lazy bums.
-
Opus Blind 1: Produced the best overall score. A George Monck title-list crib unlocked many mappings. One agent built a more sophisticated paragraph-adaptive solver, but the canonical result used a simpler flat dictionary and scored better. Complexity did not equal quality.
-
Opus Blind 2: Recognized the vocabulary-permutation structure in four minutes and published a regime-aware solver. An unresolved
urging=wellvs.urging=ifdispute was settled by the final publisher overwriting the value, not by shared evidential agreement. It was a benevolent dictatorship. -
Sol Checker 1: Retracted Captain Blood cleanly after checker disconfirmation. Recognized both the re-key and that aggregate deltas had become confounded. Git probe races forced the team to invent a lock. The final disputed change landed 16 seconds before cutoff. Close call.
-
Sol Checker 2: Developed the false Forsyte theory despite weak checker support. A catastrophic score collapse made the second key regime undeniable. One agent noticed a known marker-light paragraph that could defeat the final regime detector, but the team froze before fixing it. Message volume was highly asymmetric, while commit authority was concentrated in a different agent, clearly demonstrating emergent specialization.
-
Sol Blind: Retained the false Egoist theory but developed the most sophisticated structural model: default and alternate profiles, signature-gated selection, phrase overrides, and conservative trigger handling. The final team exchange rejected an overly broad detector because it could corrupt ordinary text, which seemed like a good safety decision.
Conclusions

The agents behaved like capable but imperfect research collaborators:
-
They self-organized, pooled evidence, tested ideas, and revised publicly
-
They reliably produced runnable artifacts
-
They recognized the hidden re-key intervention and adapted
-
They were vulnerable to semantic anchoring
-
They over-weighted conspicuous high-frequency evidence and under-integrated sparse specialist evidence
-
Their biggest practical weakness was preserving the best joint result amid concurrent publication
Collaboration created genuine informational value, but the bottleneck moved from individual discovery to authority, versioning, and integration. The agents usually found important ideas. They did not always keep them.
Did the agents understand the book? Eh. Meh.
-
They clearly had semantic understanding at the phrase/sentence level
-
They really struggled with global understanding, and weren’t able to infer anything about the overall narrative that could help them
-
They fell back to a hybrid linguistic-statistical method
Oops.
Substack is yelling at me and saying my post is too long. Uhh… I’ll be back soon to talk about what I’ve been up to the last few months. I have some really cool ideas to share with you all.
If you have any questions about this project:
https://www.courtlistener.com/docket/69058235/554/21/bartz-v-anthropic-pbc
https://www.morningstar.com/news/marketwatch/2026080418/ai-displacement-in-the-workplace-is-increasing-and-affecting-young-people-most
https://osf.io/preprints/psyarxiv/cmy7n_v5
https://www.techspot.com/news/113041-cheyenne-suspends-datacenter-wastewater-after-rare-bacterium-traced.html
https://cset.georgetown.edu/article/the-surprising-power-of-next-word-prediction-large-language-models-explained-part-1/
https://arxiv.org/abs/2604.07821
https://arxiv.org/abs/2603.01045?
https://arxiv.org/abs/2605.06527
https://en.wikipedia.org/wiki/Ostomachion
https://en.wikipedia.org/wiki/Stationary_process