Steven, a graduate in technology studies and longtime New World Notes reader, has been working with a non-profit org to help develop a system for tracking infectious disease, and his higher-ups have been encouraging the team to use ChatGPT to build out the data base.
The trouble is, when Steven asked ChatGPT for knowledge of recent outbreaks of monkeypox, he got this result:
ChatGPT fails to mention that there is also a current outbreak in Africa. Nor mentions the outbreak last year. (Nor that it's now called "mpox".)
Again, this answer was going to be put into a database of infectious diseases for use by NGOs and government bodies, before a human (i.e. Steven) caught the error.
"[I'm] not quite sure this is the slam dunk my boss thinks this is," as he puts it about ChatGPT.
To be sure, the program does include some warnings about the possibility of hallucinations (i.e. incorrect output) and that version 3 — the one this non-profit project is using — is only trained on data up to 2021. But Steven is concerned that the average user may not notice these qualifiers.
"[ChatGPT has a] quick dialogue box when you first open the page, but who reads that?"
And again, Steven's superior is the one who ordered staff to develop the database with ChatGPT!
"[The boss] cited a couple Insider articles," then got excited by the hype cycle. And this isn't using ChatGPT's output for entertainment value, but in a serious application:
"We are trying to develop a number of crowdsourced surveillance systems for infectious disease in order to essentially create 'an apples to apples information pipeline' for government, enterprise, and the civic population all working together in concert."
This isn't to say ChatGPT is completely useless in this context.
"Overall I was impressed with website scraped data it was able to pull," as Steven puts it. "But my problem is that it wasn't giving explicit sources, which you kinda need, especially around science. [That] and the 'bias' for as recent as possible data."
So ChatGPT is useful — but with qualifiers:
"If you need information quick that can be scraped, and don’t need a source, it’s good. But anything that needs a source or up to date info is going to be problem. To me it’s not so much intelligence as it is a fast scraper of large sets of data. Understand that in the advent of mega data that can be useful, but for small scale applications, it’s not really all that much help."
ChatGPT will need serious improvements before being more than that.
"Essentially what the next stage AI researchers need to do is make a 'living' data set because you can’t artificially choose a set date and then call this an intelligence. At best ChatGPT is text-based analytic software."
This has been my own personal impression with ChatGPT. I recently experimented with it on a writing project (not this blog or my book, of course!), and the output was so mediocre or wrong, it would take more time correcting and polishing its output, than just writing the whole thing myself. And forget about asking it for help with a topic you actually know well.
ChatGPT is fun and valuable in limited use cases. But with over 100 million people now using it, my concern is there are many millions who approach it with highly inflated expectations, literally risking people's lives by applying ChatGPT's output to situations for which it is simply not designed for.
To judge by my Twitter feed, where every other tweet treats ChatGPT as if it is an omniscient deity, that can't be said loudly enough.
"The medical techno-futurists are seeing AI as the future of medicine," as Steven puts it, "but the thing is it needs to be guided by human clinicians. But we all know that one doctor who could give a shit about guidance and quality care, and just accept the AI diagnosis when it is the exact wrong thing."
Wagner James Au / New World Notes 
Leave a comment