This is an archive of past discussions with User:Rsjaffe. Do not edit the contents of this page. If you wish to start a new discussion or revive an old one, please do so on the current talk page.
I've come across a couple of articles you tagged as having been created with AI and I just wondered how you coud tell this was true. They weren't sourced and were pretty vague in their content but I was wondering if there was a script or tool you used or whether it was a judgment on your part. I work a lot with draft articles these days and it would be helpful to be able to tell whether or not the content was original. Thank you! LizRead!Talk!23:05, 3 February 2023 (UTC)
There are several tools I use. The first is made by the company that produces ChatGPT. It is based on an earlier version but seems to work well. I don't count it as AI-created unless the certainty is above 99%. It usually is above 99.9%. There isn't a bell-curve distribution of readings. Most are either close to 0% fake (that is, human-produced) or 99.98% fake (LLM model, e.g., ChatGPT produced). I've never had anyone argue that the tag was incorrect, and I've probably tagged about 40 articles so far.
The tool is at https://openai-openai-detector.hf.space/. I paste in the text, omitting headings and inline ref tags. A second detector, if you're interested, is https://detector.dng.ai/. I almost always just use the first. Sometimes the AI text is embedded within other text (e.g., a custom written opening paragraph), in which case I omit the custom-written text when testing.
I've been looking for "tells" before testing. The text is more "lifeless" than human text typically is, tends to have uniform length sentences, and may have a paragraph at the end that is a summary paragraph. The text is also unlikely to have in-line references. This doesn't catch all of them but tends to be a high-likelihood way of finding them. Some of these AI-generated articles also have AI-generated references. The references are almost always fake: the AI confabulates reasonable-sounding references! — rsjaffe🗣️00:49, 4 February 2023 (UTC)
I'm just stumbling across this after a heads-up in the newsroom for The Signpost. We included a teaser about AI generated articles in a recent issue. Can you tell me how to find the articles you tagged? ☆ Bri (talk) 16:20, 13 February 2023 (UTC)
@Bri Search for hastemplate:AI-generated in article and draftspace. I think several of them did make it to article space but were then draftified. I'd have to go through the edit history to figure out which ones those were. — rsjaffe🗣️16:39, 13 February 2023 (UTC)
Also, a couple have had the template removed after the author re-edited the page to remove the AI-generated text. — rsjaffe🗣️16:40, 13 February 2023 (UTC)
Thanks for the pointer to the draft. Do you have a personal reaction to the quality of the generated articles? Briefly, are you for or against this method of creation? ☆ Bri (talk) 16:46, 13 February 2023 (UTC)
Strongly against. These language generators are good at stringing together reasonable-sounding text, but:
1. Don't rely solely on what Wikipedia recognizes as reliable sources.
2. Don't have a sense of what is real and what is fake.
3. Will fabricate information to fill in gaps.
4. Cannot identify where any specific bit of "information" came from.
So the text can fail WP:RS and WP:V, yet may look very convincing.
However, they're good at helping to break "writer's block", by giving an example of how to write something. So I see having some text generated by ChatGPT would be helpful as a writing prompt, but without using any of the generated text directly—just looking at it and then writing a well-researched article similar to the prompt. — rsjaffe🗣️17:06, 13 February 2023 (UTC)
Cool, let's follow up more at The Newsroom, I see one of the other editors contacted you by email and I'm not sure who was first (doesn't really matter now). ☆ Bri (talk) 21:11, 13 February 2023 (UTC)
Before I do anything else, could you affirm that it's OK if I use the conversation above in The Signpost as a mini interview? Also, do you prefer we refer to you by the name on your userpage, your on-Wiki username, or both? ☆ Bri (talk) 21:44, 13 February 2023 (UTC)
