Writing automation workshop with Mike Taylor at Every Think Week
Mike Taylor shows how he automated his writing by having Claude optimize a writing skill against a head-to-head eval of his own published posts, and why human ideas, interviews and editors still matter.
Watch on YouTube ↗Charlie's summary
- ▶ 3:37
Automation is not a slop factory: you need real insights, and AI works as your ghostwriter
Mike thinks this process fails for people with no ideas to share: “go and live an interesting life.” Using AI to write is defensible if you treat it as a ghostwriter and can defend the post afterwards. He fully automated his LinkedIn and an e-book with GPT-4, and a client couldn't tell. It left him with an existential crisis, because he couldn't remember what he had written. He is open about AI authorship and credits Claude in bylines.
- ▶ 6:42
The breakthrough is an eval that asks which of these two is closer to what Mike really wrote?
Asking a model to write “in my style” fails, since models are trained away from impersonation and keep AI tells. Mike's fix: turn a published post into a brief of facts only, have Claude write it with a candidate skill and with a baseline Claude-written skill, then have a judge pick which is closer to the original. Mike says it works because relative comparison is easy to judge, needs no hand-made rules, and scales by rerunning.
- ▶ 11:17
Hold out test data, and train on the same shape of writing you want back
Mike split his writing into training (105 pieces), tuning (26) and a final exam (54 held out), so the skill can't just memorize his posts. Mike says 10 pieces is the minimum (8 train, 2 held back), and 50 is where it gets really juicy. Match the format: train separate skills for tweets, LinkedIn posts and long-form. His style also changed over the years, so he narrowed the training set to avoid blended instructions.
- ▶ 15:23
Let the AI write the rules: the skill it produced described Mike better than he could himself
Mike told Claude to keep rewriting the skill until it beat the baseline, reaching about 80% head-to-head wins. He never told it how to write a skill, yet it found his register (“earnest, measured, paragraph driven explanatory prose”), his name-dropping, and that headings added points. Mike says it is like “AI is prompt engineering itself”, so your effort should go into curating the data set. Add each newly published post to it.
Give your email to unlock the rest of the key points — for every episode.
Do more with Charlie
Free — sign in with your email, no password.