AI tool ChatGPT passed only part of U.S. bar exam, but did well on other assessments

A popular, recently developed artificial intelligence tool called ChatGPT has academics and schools across North America rethinking approaches to coursework and crucial testing to avoid cheating or plagiarism. claim the tool, launched last year, has the ability to pass medical and legal licensing exams and business exams in the United States, and argue workers in these industries should be concerned. Preliminary research done in the United States suggests this is partly true. Some studies found the tool performed equally to students for some exams, while for others it did not meet the threshold to consider it a success. ��

Several Twitter users posted �� last month claiming ChatGPT, which stands for Chat Generative Pre-trained Transformer, has passed the United States Medical Licensing Examination, the bar exam and the MBA operations exam. One user's tweet was shared thousands of times. ��

Rating: Partly true��

is an artificial intelligence text generator developed by , a research and deployment company, and released in November 2022. The tool is as it is not connected to the internet, and instead answers requests based on data stored on internal servers.��

The tool has quickly garnered widespread attention for its ability to produce ideas for song lyrics, poems and scripts. It's also being used to write essays. ��

According to , posted on the CTV News website, "ChatGPT is distinct for being able to generate fodder of varying expertise, ranging from high school to university-level compositions, while other online tools can correct grammar, tone and clarity."��

Passing the bar��

conducted by professors at Michigan State College of Law and Chicago Kent College of Law at Illinois Tech found ChatGPT scored a 50.3 per cent on the multiple choice, or multistate bar exam, portion of the test. Researchers say this surpasses the baseline score for those who respond with guesses, which is 25 per cent.��

"We believe that these results strongly suggest that an LLM (large language model) will pass the multiple choice component of the Bar Exam in the near future," researchers wrote, referring to the type of artificial intelligence ChatGPT falls under. ��

However, there are two other sections of the , which is the standardized test applicants seeking to practise law must pass in the majority of U.S. jurisdictions. ChatGPT was only tested on the��multiple choice section of the exam.��

In another , researchers from the University of Minnesota Law School examined how well ChatGPT was able to write law school exams without human assistance. The group used the tool to generate answers on four real exams given at the university, and graded them blindly as part of the regular grading processes for each class. The artificial intelligence tool received an average grade of a student at a C+ level, achieving a low but passing grade. ��

One of the professors involved with the study told ��the goal of the tests was to explore ChatGPT's potential to assist lawyers in their practice and to not help students in exams. ��

"ChatGPT struggled with the most classic components of law school exams, such as spotting potential legal issues and deep analysis applying legal rules to the facts of the case," professor Jon Choi told CNN.

"But ChatGPT could be very helpful at producing a first draft that a student could then refine."��

Medical licensing exam��

When it came to the tool's performance on the , a spearheaded by researchers from the medical technology startup Ansible Health found ChatGPT "performed at or near the passing threshold," without any specialized training or reinforcement. ��

This exam also consists of three sections, but unlike the bar exam ChatGPT was tested on all three portions of the medical licensing exam. The data show the artificial intelligence tool would have received a failing grade for one or two of the exam sections, depending on which method the researchers used to test the program.��

The report, which hasn't been peer reviewed, stated the results suggest, "large language models may have the potential to assist with medical education, and potentially, clinical decision-making."��

Business operations exam��

A written by Christian Terwiesch, co-director of the Mack Institute for Innovation Management at the Wharton business school, University of Pennsylvania, found that with help from a human expert ChatGPT received a B to B- grade on a typical MBA course exam at the university. ��

Terwiesch noted the tool does well at "basic operations management and process analysis questions," but makes surprising mistakes when it comes to grade-school level math calculations. ��

What this means for Canadian schools��

Post-secondary schools across the country are having conversations around the benefits and downsides of ChatGPT and other artificial intelligence tools. ��

While there have been , many institutions in Canada are still crafting policies. ��

At Royal Roads University in British Columbia, staff are developing a set of recommendations around generative artificial intelligence. ��

"I don't expect that artificial intelligence is going to disrupt education and eliminate institutions," said George Veletsianos, a professor in the school of education and technology at the school and a Canada Research Chair in innovative learning and technology. ��

"But, I anticipate they will have impacts on teaching and learning."��

Veletsianos expects this technology is already being used in the business world. To ignore this technology would be setting up students for failure, he added.��

"We ought to be in the job of helping students develop a better world. What that means is being able to use the technology of the day, but also critically think about them."��

Veletsianos said that to assume a student will use the tool for cheating is problematic. Instead, he said, teachers should engage with their students about the limitations and benefits of incorporating it into their work. ��

OpenAI has launched a that can help teachers detect if a student or artificial intelligence wrote their homework, though the company cautions it's not foolproof. ��

Sources��

Claim can be found on Twitter (), () and�� ()��

() and�� ()��

()��

About Canadian Press fact checks��

You can find out more about �� here. To reach our fact-checking team with any tips, corrections or comments, please email us at cpfactcheck@thecanadianpress.com.��

- This article was updated on March 6, 2023, to correct a typo

��