Research Problem
A recent survey has shown that over half of post-secondary students have used AI agents to complete assignments or tests (DeLaire, 2023). D’Andrea (2023) quotes University of Saskatchewan educational ethics research Sarah Eaton who notes that while “[t]here are strong indications from Microsoft and Google that by the end of 2025, AI technologies will be fully integrated into Microsoft Office and the Google Suite of products.” But, educators are unprepared for the deep integration of such tools into student activity. Preparing educators to effectively leverage AI agents, and to discourage their misuse, requires targeted supports. This research investigated the use of an adapted version of the TSES, called the ChatGPT Teacher’s Sense of Efficacy Scale (Chat-T), to identify gaps in educators’ perceptions of efficacy with the use of AI agents such as ChatGPT in their teaching and learning practice.
Conceptual Framework
Research Questions
- What is the internal reliability and validity of the Chat-T research instrument?
- How does targeted training and practice impact educators’ perceptions of self-efficacy with the use of AI agents in teaching and learning practice?
- What additional targeted supports do educators need to increase their confidence with the use of AI agents in teaching and learning practice?
Significance of the Research
This research investigated the impacts of targeted training and a targeted hands-on experience with the use of AI agents, such as Chat GPT, on perceptions of self-efficacy with the use of AI agents in teaching and learning practice amongst graduate Education students. This research also aimed to establish the internal reliability and validity of the Chat-T research instrument, adapted from the TSES instrument, and its utility as a tool for gauging the effectiveness of professional development activities and the identification of gaps in confidence requiring further targeted supports. The results of this research provide information and a new tool that may be useful to administrators, policymakers, and others involved with planning for and supporting the integration, and effective and ethical use of AI agents in teaching and learning practice.
Results and Publications
Conference Presentations
Power, R. and Hambrock, H. (2024, April 10). Seamless teaching and learning from an AI perspective. [Webinar]. AI for Education IRN-ISSE Webinar Series 2024.
Power, R. (2024a). Educator Confidence with AI: A Case Study and a New Research Tool. [Presentation File]. Invited presentation at Artificial Intelligence: The Balance of Innovation and Prevention, 21 March 2024, Cape Breton University, Sydney, NS, Canada.
|
Publications
Power, R. (2024b). Evaluating Graduate Education Students’ Self-Efficacy with the Use of Artificial Intelligence Agents. Journal of Educational Informatics, 5(1), 3-19. https://journalofeducationalinformatics.ca/index.php/JEI/article/view/269
Power, R. (2025). Seamlessly Integrating and Evaluating AI Competencies for Graduate Education Students. In R. Power and H. Hambrock (Eds), Artificial Intelligence for Seamless Education. International Research Network for Innovative Sustainable Seamless Education and Power Learning Solutions. ISBN 978-1-7390190-3-7. https://pressbooks.pub/aiforseamlesseducation/chapter/thechattproject/
Notebook LM Generated Podcast About the Chat-T Research Project:
(from Power, 2025) ![]()
|
Conclusions and Recommendations
Impacts on Self-Efficacy and Intention to Use AI Agents |
Conclusions
(Power, 2024b, pp. 15-16)
Rapid growth in the number and variety of AI agents, combined with indications of their widespread use by students, necessitates the development of institutional guidelines. They also necessitate the preparation of educators with understandings of how such tools work, the implications of their use, and appropriate pedagogical strategies to leverage AI agents in their practice. This case study contributes to identified gaps in the body of research on preparing educators to effectively integrate AI tools by demonstrating how one group of graduate Education students explored the potentials and pitfalls of using the ChatGPT AI agent in academic writing. Results demonstrated a growth in their understanding of effective complimentary roles between AI tools and educators as described by Jeon and Lee (2023). The results also provided insights into the additional supports that the students may need to be better prepared to effectively integrate AI agents in teaching and learning. This case study also introduced the Chat-T research instrument. While further research is needed to verify the construct validity of the instrument, it did prove useful in identifying areas for strengthening the design of course activities. Further validation of the Chat-T instrument may result in a tool that can be beneficial as a tool for planning and evaluating AI-focused educator professional development.
Recommendations for Practice |
Recommendations for Future Research |
The Chat-T Survey Instrument
Using the Chat-T as a Research Tool
An online version of the Chat-T survey instrument is available using Google Forms. The form should be copied into your own Google Drive account. Simply copy the form, and update the settings so that it reports submissions to a new Google Sheets document. You'll then be able to download the data in common spreadsheet application formats for offline quantitative analysis.An online version of the mTSES survey instrument is available using Google Forms. The form should be copied into your own Google Drive account. Simply copy the form, and update the settings so that it reports submissions to a new Google Sheets document. You'll then be able to download the data in common spreadsheet application formats for offline quantitative analysis.
|
Research Team
Dr. Rob Power
Associate Professor, Education, Cape Breton University, Canada Dr. Rob Power is currently an Associate Professor of Education at Cape Breton University. He has an extensive background in educational technology integration and instructional design for online and blended learning in K12, post-secondary, and workplace training contexts. Dr. Power has worked as an instructional design consultant and as the Leader of the Online Learning team with the Fraser Health Authority in British Columbia, Canada. He has also served in leadership roles with the International Association for Mobile Learning (IAmLearn) and the Canadian Network for Innovation in Education (CNIE). Dr. Power is also a founding member of the Pedagogy, Education and Technology Lab (PETL) and the International Research Network for Innovative Sustainable and Seamless Learning (IRN-ISSE), and president of Power Learning Solutions. |
References
Bond, M., Khosravi, H., De Laat, M. et al. A meta systematic review of artificial intelligence in higher education: a call for increased ethics, collaboration, and rigour. International Journal of Educational Technology in Higher Education, 21, 4 (2024). https://doi.org/10.1186/s41239-023-00436-z
Celik, I., Dindar, M., Muukkonen, H., & Järvelä, s. (2022). The Promises and Challenges of Artificial Intelligence for Teachers: A Systematic Review of Research. TechTrends, 66, pp. 616-630. https://doi.org/10.1007/s11528-022-00715-y
Cowan, L. (2023, May). AI and the future of academic work. CAUT. https://www.caut.ca/bulletin/2023/05/ai-and-future-academic-work
D'Andrea, A. (2023, February 1). Canadian universities crafting ChatGPT policies as French school bans AI program. Global News. https://globalnews.ca/news/9451143/chatgpt-education-canadian-universities/
DeLaire, M. (2023, August 31). More than half of Canadian students over 18 use AI tools: survey. CTV News. https://www.ctvnews.ca/sci-tech/more-than-half-of-canadian-students-over-18-use-ai-tools-survey-1.6543380
Higher Education Strategy Associates (2023, August 18). HESA’s AI Observatory: What’s new in higher education (Aug. 18th, 2023). https://higheredstrategy.com/hesas-ai-observatory-whats-new-in-higher-education-aug-18th-2023/
Jeon, J., & Lee, S. (2023). Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT. Education and Information Technologies, 28(12), 15873–15892. https://doi.org/10.1007/s10639-023-11834-1
Langreo, L. (2023a, May 10). Teachers Need PD on Artificial Intelligence. What It Should Look Like. Education Week. https://www.edweek.org/leadership/teachers-need-pd-on-artificial-intelligence-what-it-should-look-like/2023/04
Langreo, L. (2023b, June 27). 7 Strategies to Prepare Educators to Teach With AI. Education Week. https://www.edweek.org/teaching-learning/7-strategies-to-prepare-educators-to-teach-with-ai/2023/06
MobileMind (2024). 6 Risks of Neglecting AI Professional Development. https://www.mobilemind.io/6-risks-of-neglecting-ai-professional-development
OpenAI (n.d.). ChatGPT. [Software application]. https://chat.openai.com
Power, R. (2024a). Educator Confidence with AI: A Case Study and a New Research Tool. [Presentation File]. Invited presentation at Artificial Intelligence: The Balance of Innovation and Prevention, 21 March 2024, Cape Breton University, Sydney, NS, Canada.
Power, R. (2024b). Evaluating Graduate Education Students’ Self-Efficacy with the Use of Artificial Intelligence Agents. Journal of Educational Informatics, 5(1), 3-19. https://journalofeducationalinformatics.ca/index.php/JEI/article/view/269
Power, R. (2025). Seamlessly Integrating and Evaluating AI Competencies for Graduate Education Students. In R. Power and H. Hambrock (Eds), Artificial Intelligence for Seamless Education. International Research Network for Innovative Sustainable Seamless Education and Power Learning Solutions. ISBN 978-1-7390190-3-7. https://pressbooks.pub/aiforseamlesseducation/chapter/thechattproject/
Power, R. and Hambrock, H. (2024, April 10). Seamless teaching and learning from an AI perspective. [Webinar]. AI for Education IRN-ISSE Webinar Series 2024.
Power, R. & Kay, R. (2023). Higher Education Faculty Supports for the Transition to Online Teaching during the COVID-19 Pandemic. Journal of Educational Informatics, 4(1), 49-72. https://journalofeducationalinformatics.ca/index.php/JEI/article/view/191
Celik, I., Dindar, M., Muukkonen, H., & Järvelä, s. (2022). The Promises and Challenges of Artificial Intelligence for Teachers: A Systematic Review of Research. TechTrends, 66, pp. 616-630. https://doi.org/10.1007/s11528-022-00715-y
Cowan, L. (2023, May). AI and the future of academic work. CAUT. https://www.caut.ca/bulletin/2023/05/ai-and-future-academic-work
D'Andrea, A. (2023, February 1). Canadian universities crafting ChatGPT policies as French school bans AI program. Global News. https://globalnews.ca/news/9451143/chatgpt-education-canadian-universities/
DeLaire, M. (2023, August 31). More than half of Canadian students over 18 use AI tools: survey. CTV News. https://www.ctvnews.ca/sci-tech/more-than-half-of-canadian-students-over-18-use-ai-tools-survey-1.6543380
Higher Education Strategy Associates (2023, August 18). HESA’s AI Observatory: What’s new in higher education (Aug. 18th, 2023). https://higheredstrategy.com/hesas-ai-observatory-whats-new-in-higher-education-aug-18th-2023/
Jeon, J., & Lee, S. (2023). Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT. Education and Information Technologies, 28(12), 15873–15892. https://doi.org/10.1007/s10639-023-11834-1
Langreo, L. (2023a, May 10). Teachers Need PD on Artificial Intelligence. What It Should Look Like. Education Week. https://www.edweek.org/leadership/teachers-need-pd-on-artificial-intelligence-what-it-should-look-like/2023/04
Langreo, L. (2023b, June 27). 7 Strategies to Prepare Educators to Teach With AI. Education Week. https://www.edweek.org/teaching-learning/7-strategies-to-prepare-educators-to-teach-with-ai/2023/06
MobileMind (2024). 6 Risks of Neglecting AI Professional Development. https://www.mobilemind.io/6-risks-of-neglecting-ai-professional-development
OpenAI (n.d.). ChatGPT. [Software application]. https://chat.openai.com
Power, R. (2024a). Educator Confidence with AI: A Case Study and a New Research Tool. [Presentation File]. Invited presentation at Artificial Intelligence: The Balance of Innovation and Prevention, 21 March 2024, Cape Breton University, Sydney, NS, Canada.
Power, R. (2024b). Evaluating Graduate Education Students’ Self-Efficacy with the Use of Artificial Intelligence Agents. Journal of Educational Informatics, 5(1), 3-19. https://journalofeducationalinformatics.ca/index.php/JEI/article/view/269
Power, R. (2025). Seamlessly Integrating and Evaluating AI Competencies for Graduate Education Students. In R. Power and H. Hambrock (Eds), Artificial Intelligence for Seamless Education. International Research Network for Innovative Sustainable Seamless Education and Power Learning Solutions. ISBN 978-1-7390190-3-7. https://pressbooks.pub/aiforseamlesseducation/chapter/thechattproject/
Power, R. and Hambrock, H. (2024, April 10). Seamless teaching and learning from an AI perspective. [Webinar]. AI for Education IRN-ISSE Webinar Series 2024.
Power, R. & Kay, R. (2023). Higher Education Faculty Supports for the Transition to Online Teaching during the COVID-19 Pandemic. Journal of Educational Informatics, 4(1), 49-72. https://journalofeducationalinformatics.ca/index.php/JEI/article/view/191