​ChatGPT Teacher’s Sense of Efficacy Scale (Chat-T)

Evaluating Graduate Education Students’ Self-Efficacy with the Use of Artificial Intelligence Agents in Teaching and Learning

Research Problem

​A recent survey has shown that over half of post-secondary students have used AI agents to complete assignments or tests (DeLaire, 2023). D’Andrea (2023) quotes University of Saskatchewan educational ethics research Sarah Eaton who notes that while “[t]here are strong indications from Microsoft and Google that by the end of 2025, AI technologies will be fully integrated into Microsoft Office and the Google Suite of products.” But, educators are unprepared for the deep integration of such tools into student activity. Preparing educators to effectively leverage AI agents, and to discourage their misuse, requires targeted supports. This research investigated the use of an adapted version of the TSES, called the ChatGPT Teacher’s Sense of Efficacy Scale (Chat-T), to identify gaps in educators’ perceptions of efficacy with the use of AI agents such as ChatGPT in their teaching and learning practice. 

Conceptual Framework

Research Questions

  1. What is the internal reliability and validity of the Chat-T research instrument?
  2. How does targeted training and practice impact educators’ perceptions of self-efficacy with the use of AI agents in teaching and learning practice?
  3. What additional targeted supports do educators need to increase their confidence with the use of AI agents in teaching and learning practice? 

Significance of the Research

​This research investigated the impacts of targeted training and a targeted hands-on experience with the use of AI agents, such as Chat GPT, on perceptions of self-efficacy with the use of AI agents in teaching and learning practice amongst graduate Education students. This research also aimed to establish the internal reliability and validity of the Chat-T research instrument, adapted from the TSES instrument, and its utility as a tool for gauging the effectiveness of professional development activities and the identification of gaps in confidence requiring further targeted supports. The results of this research provide information and a new tool that may be useful to administrators, policymakers, and others involved with planning for and supporting the integration, and effective and ethical use of AI agents in teaching and learning practice. 

Results and Publications

Conference Presentations
Power, R. and Hambrock, H. (2024, April 10). Seamless teaching and learning from an AI perspective. [Webinar]. AI for Education IRN-ISSE Webinar Series 2024. 
Power, R. (2024a). Educator Confidence with AI: A Case Study and a New Research Tool. [Presentation File]. Invited presentation at Artificial Intelligence: The Balance of Innovation and Prevention, 21 March 2024, Cape Breton University, Sydney, NS, Canada. 
Publications
Power, R. (2024b). Evaluating Graduate Education Students’ Self-Efficacy with the Use of Artificial Intelligence Agents. Journal of Educational Informatics, 5(1), 3-19. ​https://journalofeducationalinformatics.ca/index.php/JEI/article/view/269
Power, R. (2025). Seamlessly Integrating and Evaluating AI Competencies for Graduate Education Students. In R. Power and H. Hambrock (Eds), Artificial Intelligence for Seamless Education. International Research Network for Innovative Sustainable Seamless Education and Power Learning Solutions. ISBN 978-1-7390190-3-7. https://pressbooks.pub/aiforseamlesseducation/chapter/thechattproject/
Notebook LM Generated Podcast About the Chat-T Research Project:
(from Power, 2025)
Audio Transcript - Chat-T Podcast
File Size: 91 kb
File Type: pdf
Download File

Conclusions and Recommendations

Impacts on Self-Efficacy and Intention to Use AI Agents
(Power, 2024, p. 14)
​
These findings from this research "highlight the importance of including targeted professional development on the pedagogies of AI use to increase educators’ confidence and actual integration of such tools (Figure 3)" (Power, 2024b, p. 14).
This page is still under construction. Check back soon!
Figure 3: Using TAM (Davis, 1989) with Pedagogical Training to Support Educator Adoption of ChatGPT (Power, 2024b, p. 14)
Conclusions
(Power, 2024b, pp. 15-16)

Rapid growth in the number and variety of AI agents, combined with indications of their widespread use by students, necessitates the development of institutional guidelines. They also necessitate the preparation of educators with understandings of how such tools work, the implications of their use, and appropriate pedagogical strategies to leverage AI agents in their practice. This case study contributes to identified gaps in the body of research on preparing educators to effectively integrate AI tools by demonstrating how one group of graduate Education students explored the potentials and pitfalls of using the ChatGPT AI agent in academic writing. Results demonstrated a growth in their understanding of effective complimentary roles between AI tools and educators as described by Jeon and Lee (2023). The results also provided insights into the additional supports that the students may need to be better prepared to effectively integrate AI agents in teaching and learning. This case study also introduced the Chat-T research instrument. While further research is needed to verify the construct validity of the instrument, it did prove useful in identifying areas for strengthening the design of course activities. Further validation of the Chat-T instrument may result in a tool that can be beneficial as a tool for planning and evaluating AI-focused educator professional development. 
Recommendations for Practice
(Power, 2024b, p. 16)

​Educational institutions recognize the urgency for the development of policies and guidelines on the ethical use of AI agents (Bond et al, 2024; Cowan, 2023; D’Andrea, 2023; HESA, 2023). However, educators and students need training on the benefits and limitations of AI “so they can use AI-powered tools responsibly in the classroom and as everyday consumers of the technology” (Langreo, 2023a). Additionally, the integration of tools such as ChatGPT requires support for educators that goes “beyond a foundational understanding of AI and how to use it on a basic level” (MobileMind, 2024). One of the aims of this case study was to show how a group of graduate Education students went beyond understanding the foundational use of ChatGPT for academic writing to critically evaluate its strengths and limitations, so that they could gain confidence as educational leaders. However, while participants expressed eagerness to integrate AI agents in their practice, they also expressed trepidation about their limitations and a desire for more training and support on effective pedagogical approaches. Explicit pedagogical training is one of the essential elements of educator professional development highlighted by Langreo (2023b). Power and Kay (2023) show how gaining hands-on experience with novel technologies, combined with appropriate just-in-time supports, can help educators to move forward with Celik et al.’s (2022) call for them to be directly involved with “AI creation, development, and integration" (p. 617). Power and Kay (2023) also show how educators benefit from formal, institutionally-facilitated professional development focused on pedagogical practices. To promote the ethical and impactful integration of AI agents in teaching and learning practice, it is recommended that institutions look beyond establishing policies and towards the implementation of formal pedagogically-focused professional development opportunities. 
Recommendations for Future Research
(Power, 2024b, pp. 16-17)

​A second aim of this case study was to introduce a potential tool for evaluating the confidence and readiness of educators to use AI agents in their teaching practice, and identifying areas of focus for future professional development initiatives. The Chat-T instrument did provide insights into the impacts of hands-on tool use, and critical evaluation of ChatGPT generated academic writing, on the perceptions of self-efficacy among a group of graduate Education students. The net decrease in their expressions of confidence may be attributed to the lack of exposure to explicit pedagogical training, reiterating the importance of such support to promote the integration of AI agents in teaching practice. However, it will be necessary to evaluate the Chat-T instrument with a larger sample size to determine its construct validity in relation to the original TSES survey.

The Chat-T Survey Instrument

Using the Chat-T as a Research Tool

An online version of the Chat-T survey instrument is available using Google Forms. The form should be copied into your own Google Drive account. Simply copy the form, and update the settings so that it reports submissions to a new Google Sheets document. You'll then be able to download the data in common spreadsheet application formats for offline quantitative analysis.An online version of the mTSES survey instrument is available using Google Forms. The form should be copied into your own Google Drive account. Simply copy the form, and update the settings so that it reports submissions to a new Google Sheets document. You'll then be able to download the data in common spreadsheet application formats for offline quantitative analysis.

Research Team

Dr. Rob Power
Associate Professor, Education, Cape Breton University, Canada
Dr. Rob Power is currently an Associate Professor of Education at Cape Breton University. He has an extensive background in educational technology integration and instructional design for online and blended learning in K12, post-secondary, and workplace training contexts. Dr. Power has worked as an instructional design consultant and as the Leader of the Online Learning team with the Fraser Health Authority in British Columbia, Canada. He has also served in leadership roles with the International Association for Mobile Learning (IAmLearn) and the Canadian Network for Innovation in Education (CNIE). Dr. Power is also a founding member of the Pedagogy, Education and Technology Lab (PETL) and the International Research Network for Innovative Sustainable and Seamless Learning (IRN-ISSE), and president of Power Learning Solutions.

References

Bond, M., Khosravi, H., De Laat, M. et al. A meta systematic review of artificial intelligence in higher education: a call for increased ethics, collaboration, and rigour. International Journal of Educational Technology in Higher Education, 21, 4 (2024). https://doi.org/10.1186/s41239-023-00436-z

Celik, I., Dindar, M., Muukkonen, H., & Järvelä, s. (2022). The Promises and Challenges of Artificial Intelligence for Teachers: A Systematic Review of Research. TechTrends, 66, pp. 616-630. https://doi.org/10.1007/s11528-022-00715-y

Cowan, L. (2023, May). AI and the future of academic work. CAUT. https://www.caut.ca/bulletin/2023/05/ai-and-future-academic-work

D'Andrea, A. (2023, February 1). Canadian universities crafting ChatGPT policies as French school bans AI program. Global News. https://globalnews.ca/news/9451143/chatgpt-education-canadian-universities/

DeLaire, M. (2023, August 31). More than half of Canadian students over 18 use AI tools: survey. CTV News. https://www.ctvnews.ca/sci-tech/more-than-half-of-canadian-students-over-18-use-ai-tools-survey-1.6543380

Higher Education Strategy Associates (2023, August 18). HESA’s AI Observatory: What’s new in higher education (Aug. 18th, 2023). https://higheredstrategy.com/hesas-ai-observatory-whats-new-in-higher-education-aug-18th-2023/

Jeon, J., & Lee, S. (2023). Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT. Education and Information Technologies, 28(12), 15873–15892. https://doi.org/10.1007/s10639-023-11834-1

Langreo, L. (2023a, May 10). Teachers Need PD on Artificial Intelligence. What It Should Look Like. Education Week. https://www.edweek.org/leadership/teachers-need-pd-on-artificial-intelligence-what-it-should-look-like/2023/04

Langreo, L. (2023b, June 27). 7 Strategies to Prepare Educators to Teach With AI. Education Week. https://www.edweek.org/teaching-learning/7-strategies-to-prepare-educators-to-teach-with-ai/2023/06

MobileMind (2024). 6 Risks of Neglecting AI Professional Development. https://www.mobilemind.io/6-risks-of-neglecting-ai-professional-development

OpenAI (n.d.). ChatGPT. [Software application]. https://chat.openai.com   

Power, R. (2024a). Educator Confidence with AI: A Case Study and a New Research Tool. [Presentation File]. Invited presentation at Artificial Intelligence: The Balance of Innovation and Prevention, 21 March 2024, Cape Breton University, Sydney, NS, Canada. 

Power, R. (2024b). Evaluating Graduate Education Students’ Self-Efficacy with the Use of Artificial Intelligence Agents. Journal of Educational Informatics, 5(1), 3-19. ​https://journalofeducationalinformatics.ca/index.php/JEI/article/view/269

Power, R. (2025). Seamlessly Integrating and Evaluating AI Competencies for Graduate Education Students. In R. Power and H. Hambrock (Eds), Artificial Intelligence for Seamless Education. International Research Network for Innovative Sustainable Seamless Education and Power Learning Solutions. ISBN 978-1-7390190-3-7. https://pressbooks.pub/aiforseamlesseducation/chapter/thechattproject/​

​Power, R. and Hambrock, H. (2024, April 10). Seamless teaching and learning from an AI perspective. [Webinar]. AI for Education IRN-ISSE Webinar Series 2024. 

Power, R. & Kay, R. (2023). Higher Education Faculty Supports for the Transition to Online Teaching during the COVID-19 Pandemic. Journal of Educational Informatics, 4(1), 49-72. https://journalofeducationalinformatics.ca/index.php/JEI/article/view/191
Proudly powered by Weebly