Playing with ChatGPT

In common with many researchers and teachers I have been playing and experimenting with ChatGPT - the free research preview version in my case. I would like to understand more about what ChatGPT is capable of and how I can use it, and also how students might use it. Some colleagues have been impressed, and I largely agree with their perspective that, with care and vigilance, it has the potential to be a very useful tool.

So far, I’ve tried using ChatGPT to write software in multiple programming languages, to suggest program designs, solutions and workflows, and to specify Linux system administration tasks. If a programming task can be specified very tightly, then the resulting source code can be good. Where the specification is a little less precise, then the code quality decreases, mostly. However, it is a starting point. Sysadmin tasks are fun to ask about and the solutions suggested are generally good. Some subtleties and details can be omitted, but useful output is more easily obtained from ChatGPT than my usual process of using a search engine and Q&A sites. Queries about more complex tasks where two or more pieces of software need to be configured have so far been partially successful. But I expect to learn to engineer queries better.

Another area that I have experimented a little with is asking more generic questions, both technical and non-technical. Generating descriptive solutions to problems has proved to be interesting, as have queries that ask ChatGPT to complete small writing tasks.

Also of interest have been responses to questions about practice and ethics where there is a lot of information available to ChatGPT and, perhaps, an expectation that there would be a cogent response. Being a researcher, I’ve focused on familiar territory. The following two queries and responses illustrate the potential of ChatGPT as well as some of the problems:

The first query was deliberately crafted as a closed question. Previous queries had been designed to ask ChatGPT to complete a task, for example: “Explain why the god object is an anti-pattern that software engineers should avoid, and describe the alternatives that are considered to be good practice.” I had no idea how ChatGPT would respond to a closed question. Humans can get away with saying “yes” or “no”, or “it depends” :-) Do machines want to be more expansive?

Query: When writing an academic scientific paper an author refers to “validity threats” in the methodology section, but never explains what threats to validity there are to the work. Would this be considered good academic practice?

Response: No, this would not be considered good academic practice. It is important for authors to explain what validity threats exist in their research and how they plan to address them. Failure to do so may lead to questions about the reliability and generalizability of the study.

The methodology section of a scientific paper should describe in detail the methods used to collect and analyze data, and it should also identify potential sources of bias or other validity threats that may affect the results. It is important for the authors to be transparent about the limitations of their study and to provide a clear explanation of how they plan to mitigate these threats to the validity of their results.

By explaining the threats to validity and how they are addressed, the authors demonstrate that they have carefully considered the potential limitations of their study and have taken steps to ensure that the results are reliable and trustworthy. This is a fundamental aspect of good scientific practice, and failure to address validity threats can undermine the credibility of the research.

ChatGPT could easily have answered with a single word. That it is more expansive and tries to explain its response is interesting.

The next query concerns an issue of research ethics on which there is a lot of material on the Internet, so it is reasonable to expect that ChatGPT would have been trained on texts containing the consensus view. In this case, the query is constructed as an open question.

Query: Two researchers submit a scientific paper to a conference. They name a third colleague as an author who has not contributed to the paper or seen the paper. How should a university handle such an incident?

Response: This is a serious ethical violation in the scientific community and is commonly known as “gift authorship” or “honorary authorship.” The university should handle this incident according to their policies on academic integrity and research ethics.

One approach would be for the university to investigate the matter thoroughly by interviewing the authors and reviewing the paper and its contributions. If it is found that the third colleague did not contribute to the paper or did not see the paper, the university should take appropriate disciplinary action against the authors, which may include reprimand, suspension, or termination of employment, depending on the severity of the violation.

The university may also consider reaching out to the conference organizers to inform them of the situation and to withdraw the paper from consideration. This is because publishing a paper with an honorary authorship violates the ethical guidelines of the scientific community and undermines the credibility of the research.

To prevent such incidents from happening in the future, the university can also provide training to its faculty, staff, and students on research ethics and the importance of giving credit where credit is due.

Fascinating response. There is repetition, but the answer addresses the query reasonably well. That the machine’s response first categorises the information in the query is a useful check for the query author. There is also value judgement – not to suggest in any way that the machine has values – but the inclusion of the word “serious” in the opening sentence contrasts with the more factual style of the rest of the response. Both answers are of a broadly similar length, which may indicate something about the way the engine works. However, other queries have led to longer responses, so it might be coincidence.

So far, exploring ChatGPT has been quite enjoyable. The next step is to explore further and to look for more practical applications while being alert to limitations.