In April of 2023, the capabilities and limitations of ChatGPT in programming were tested at LAB with a simple case: can ChatGPT produce working code for producing phonetic passwords (Vainio 2023). The results weren’t good. It was able to regurgitate some methods for producing phonetic passwords, but the algorithms it implemented did not produce the required results (for examples, see ChatGPT 2023a; 2023b; 2023c; 2023d).
A password being phonetic just means that it feels like a real word but isn’t. Phonetic passwords were chosen as the case, because it’s a problem that’s easily understandable for people, but difficult for machines, which lack the human experience of what it feels like to say or remember a word (Vainio 2023).
PATHFINDER (2024) is project at LAB in collaboration with partners around Europe. One of goals is to enhance the comprehension of AI technology among students. As programming is a field, where generative AIs are expected to have a lot of impact, even if the nature of that expected impact has changed, it should be interesting, especially for students, to see whether these tools have progressed and has the new cutoff date for the data affected the results.
Things have progressed but not as much as promised
A year ago, ChatGPT was able to produce three options and after being prompted about it agreed that Markov chains, a widely recommended method, would also be usable. Now ChatGPT offered five different methods, but still didn’t mention Markov chains without additional prompting. These new methods do include popular ones, such as passphrases as they were introduced by Munroe (2011) and the popular method of substituting letters with similar looking symbols. However, while the methods fill the assignment in the sense that they can be used to produce memorable passwords, the results wouldn’t be considered phonetic passwords. (ChatGPT 2023e; 2024a)
An actual notable progress was clear from the Markov chain example. While a year ago ChatGPT did not include randomness properly in its implementation of the algorithm, the new version did, but arguably this implementation is not optimal for the purpose. Also, the AI made a mistake in the test data and there was a bug in the code which it was able to fix, but not necessarily satisfactorily. (ChatGPT 2023e; 2024a)
It should be noted that ChatGPT does include randomness. Anyone else having the same or similar discussion might receive wildly different results. In fact, another attempt with a different account produced useless results as ChatGPT completely misunderstood the assignment (ChatGPT 2024b) falsely conflating two different concepts around phonetics.
There have been promises of rapid development in AI, but based on this specific example, the progress has mostly just been from embarrassingly bad results to just bad results (for examples, see ChatGPT 2024c).
Author
Aki Vainio is a senior lecturer of IT at LAB and as a part of this role he works as an expert in various RDI projects. What you have just read is an example of the randomness of his worldview.
References
ChatGPT. 2023a. Phonetic Passwords by ChatGPT – Algorithm 1. Cited 5 Apr 2024. Available at https://akivainio.com/phonetic-password-codes/password-client.php?algorithm=2
ChatGPT. 2023b. Phonetic Passwords by ChatGPT – Algorithm 2. Cited 5 Apr 2024. Available at https://akivainio.com/phonetic-password-codes/password-client.php?algorithm=1
ChatGPT. 2023c. Phonetic Passwords by ChatGPT – Algorithm 3. Cited 5 Apr 2024. Available at https://akivainio.com/phonetic-password-codes/password-client.php?algorithm=3
ChatGPT. 2023d. Phonetic Passwords by ChatGPT – Algorithm 4. Cited 5 Apr 2024. Available at https://akivainio.com/phonetic-password-codes/password-client.php?algorithm=4
ChatGPT. 2023e. Phonetic Password Algorithms. Open AI. Cited 5 Apr 2024. Available at https://chat.openai.com/share/8c09b98e-25d2-4884-92a6-92992f354e29
ChatGPT. 2024a. Phonetic Password Generation. Open AI. Cited 5 Apr 2024. Available at https://chat.openai.com/share/bab58317-d40f-465a-a5cf-5c60be31a1b0
ChatGPT. 2024b. Phonetic Password Generation. Open AI. Cited 5 Apr 2024. Available at https://chat.openai.com/share/60c52e57-e245-4ed8-b641-17d986d5e061
ChatGPT. 2024c. Phonetic Passwords by ChatGPT. Cited 5 Apr 2024. Available at https://akivainio.com/phoneticstest2024/
Islandworks. 2021. Username, Password, Shredded image. Pixabay. Cited 5 Apr 2024. Available at https://pixabay.com/photos/username-password-shredded-paper-5948556/
Munroe, R. 2011. Password Strength. xkcd. Cited 5 Apr 2024. Available at https://xkcd.com/936/
PATHFINDER. 2024. Welcome to the Erasmus+ Pathfinder Project. Pioneering AI technology in higher education to facilitate innovation and nurture the development of entrepreneurial resources. Netlify. Cited 22 Mar 2024. Available at https://erasmus-pathfinder.netlify.app/
Vainio, A. 2023. Ohjelmointi tekoälyn avulla – Case: foneettiset salasanat. LAB Pro. Cited 5 Apr 2024. Available at https://www.labopen.fi/lab-pro/ohjelmointi-tekoalyn-avulla-case-foneettiset-salasanat/
