Back

Summary report of PLA3

PROFFORMANCE project - PLA 3 report

 

 

PLA overall concept

 

The PROFFORMANCE project aims to elaborate a joint criteria model for an assessment tool for teachers’ performance that may be customized to HEIs own needs at all EHEA countries.

The aim of the planned assessment tool is to define benchmarks and a framework in order to help HEIs and teachers to find strengths to base on and weaknesses to tackle. It may foster exchange of knowledge and best practices and encourage intra - and inter-institutional collaboration at national and international level.

The consortium planned 3 PLAs for the elaboration of the assessment tool. The PLA1 and PLA2 have been coordinated by the Croatian and Czech partners and experts. Daliborka Luketic (University of Zadar) and Jan Beseda (Center for Higher Education Studies) have taken the lead roles and the University of Zadar, Ministry of Education, Youth and Sport of Czech Republic and University of West Bohemia were responsible for organizational and logistical issues supported by the Project Coordinator from Hungary.

Peer learning is a powerful way of sharing knowledge. This learning involves individuals exchanging knowledge and experience with each other, and diffusing this learning back to their organizations to ensure an impact—at scale—on reform initiatives. That is why the consortium decided to gather and work together at PLAs, which beside the kick-off meeting results and benchmarking analysis contributes to the elaboration of the assessment tool.

Daliborka Luketic and Jan Beseda elaborated the background and the professional concept of PLA1 and PLA2, and the implementation has been planned together with Vilmos Vass and the Croatian, Czech and Hungarian partners. The PLA3 has been coordinated by Georgian Partner Organization and experts.

 

PLA 3

Preparation

After the end of the PLA2 and its follow-up activities, the Hungarian and Georgian partners started negotiations on technical, organizational and conceptual issues of the PLA3 and during the preparatory stage they had several online meetings, as well as intensive online communication via email. Co-leaders of the WGs have been also consulted in defining the concept of the PLA3. The online organizational and follow-up meetings with participation of Georgian and Hungarian colleagues have been held on: April 20th, April 23rd, 29th, May 5th, 12th, 18th, June 1st, 3rd,. and 29th. The Georgian partners had local organizational meetings (April 22nd, May, 26th) and preliminary meetings with the speakers of the PLA3 day 1 and day 2 on May 20th, 25th, June 4. The PLA3 issues have been discussed at CWG meeting on April 23rd, May 7th, 21st, June 14th, 21st. In parallel, Hungarian colleagues had several meetings.

It has been decided that PLA3 will be organized online, due to the world-wide spread of the 3rd wave of the Covid19 pandemic. Instead of a 2-day-long physical meeting, partners opted to split the process into two online events and follow-up activities in the framework of CWGs, considering the experience of the PLA1 and PLA2. The PLA3 was been planned for May 27th and June 7th, 2021.

Parties agreed on Irine Darchia (Ivane Javakhishvili Tbilisi State University) to take the lead role for PLA3 and the National Center for Educational Quality Enhancement (NCEQE), represented by Lali Giorgidze and Tornike Bakakuri to deal with the organizational issues, including hosting and providing the venue for the workshop, administrative and logistical tasks for contracting and reimbursing the PLA participants, communicating with external experts, etc. It has been decided that Daliborka Luketic (University of Zadar) and Jan Beseda (Center for Higher Education Studies), would channel experience of the PLA1 and PLA2 and Vilmos Vass (Metropolitan University, Hungary), Project Lead would be actively involved in the development of the concept and methodology for PLA3 for ensuring methodological and conceptual coherence of the PLAs.

The initial aim of the PLA3 formulated in the project proposal was to create Teachers’ Assessment Tool from students’ perspective, but this initial idea has been changed during the implementation of the project, as a result of benchmarking, development of teachers’ competence fields and statements and assigning them to different Thematic Areas, as well as after the development of vertical and horizontal aspects of teacher competences and 5 perspectives of their assessment (self-, student, peer, management, external stakeholder), i.e. the so-called 3D Matrix.

After the end of the PLA2 and before the start of the PLA3, Jan Beseda and Ida Dringo-Horvath developed 4 versions of assessment methodologies and based on them, the CWGs prepared the draft assessment tool for teachers’ self-assessment for 4 thematic areas (Curricula Development, Teaching and Learning Methodology, Students Assessment and Third Mission). Due to the many questions and unsolved issues concerning the 3D Matrix, different assessment perspectives and assessment methodology, that is, due to the existing “dilemmas”, it has been decided to ask external evaluators – assessment specialist, other HE experts and also representatives of different evaluator groups – to assess the work done by the CWGs and to provide their ideas, comments and suggestions for updating Thematic Areas, Competence Statements, 3D Matrix, assessment perspectives, draft assessment methodology, etc.

The concept and the agenda of the PLA3 have been revised several times in order to address the challenges in most pragmatic and efficient way. Therefore, during the preparatory stage, in close collaboration with the Project and PLA leads, as well as other Hungarian and Georgian colleagues, it has been decided to focus the PLA3 on “Quality Assurance”, on external assessment of the work already done and on finding solutions to the project “dilemmas”. In order to ensure more active involvement of the PLA participants in the discussion and to increase interactivity of the online event, the organizers planned Panel Discussions followed by Q/A sessions or Plenary Workshops instead of the Plenary Presentations and Workshops in three WGs.

The draft concepts for External Evaluators’ and Students’ Panel have been prepared by Irine Darchia and were finalized after getting feedback from the Project Management and CWG Leads. The Georgian Team (Irine Darchia, Lali Giorgidze, Tornike Bakakuri) organized three preparatory online meetings with the invited speakers – representatives of students, peers, management, external stakeholders - for both days of PLA3 (The meetings were held on May 20th, 24th, June 4th). In the second Day of the PLA3 most of the guest speakers have been selected from international experts’ pool of the NCEQE. 

 

Agreed methodology

Goals of PLA3:

1) External Evaluation of 3D Matrix, competence statements, assessment perspectives/evaluator groups, draft assessment methodology;

2) Updating of 3D Matrix and refining assessment perspectives/evaluator groups;

3) Preparing grounds for finding solutions to the existing „dilemmas” and making decisions.

Day 1 - May 27th

Input: External Evaluation of 3D Matrix, competence statements, draft assessment methodology by external experts; Methodological Guidance on Self-assessment Perspective; External Evaluation on Thematic Areas/Competence Fields to be assessed by students; Sharing the best practices of teachers’ assessment by students.

Goals: To identify the shortcomings of the 3D Matrix, competence statements, draft assessment methodology and to refine the Thematic Areas to be assessed by the Students.

Methodology:  Individual and group work; Synchronous and asynchronous work; WG and CWG meetings; Two Interactive Plenary Workshops to ensure common understanding of different issues and coherence in revising the 3D Matrix and draft assessment tool; Using online interaction APP Slido for plenary workshops and Q/A sessions.

Output: Identified shortcomings of the 3D Matrix, competence statements and draft assessment methodology in terms of usability and functionality; Identified challenges to be addressed after the PLA3; Revised competence areas to be assessed by the students; Methodological Guidance on teachers performance evaluation in terms of assesment scale, statements/indicators’ style and wording. 

Day 2 – June 7th 

Input: External Evaluation of 3D Matrix, Thematic Areas and Competence Statements by external experts; External View on Teachers’ Performance Assessment by external stakeholders, peers, management; Sharing the best practices of teachers’ assessment by peers; Results of Benchmarking Report and Best Practices.  

Goals: To identify the shortcomings of the 3D Matrix, Thematic Areas and Competence Statements; to refine the Assessment Perspectives/Thematic Areas to be assessed by external stakeholders, peers, management; to determine how to use the Results of Benchmarking Report and Best Practices.  

Methodology:  Individual and group work; Synchronous and asynchronous work; WG and meetings; Three Interactive Plenary Workshops to ensure common understanding of different issues and coherence in revising the 3D Matrix and refining the assessment perspectives/assessor groups; Using online interaction APPs Padlet and Mentimeter for plenary workshops and Q/A sessions.

Output: Identified shortcomings of the 3D Matrix, Competence Statements and Assessment Perspectives/Assessor Groups in terms of usability and functionality; Identified challenges to be addressed after the PLA3; Revised competence areas to be assessed by external stakeholders, peers, management; Decision on using the Results of Benchmarking Report and Best Practices in the Assessment Tool.

 

Implementation

 

1st DAY of PLA3 - External Assessment of Project Activities, Teachers’ Performance Assessment from Self-Assessment and Students’ Perspective

In the Introductory Part of the Day Vilmos Vass and Jan Beseda updated the audience about the PLA2 activities and results. Irine Darchia briefly presented the PLA3 goals and expected outcomes and outline the expectations from the first day of PLA3. She also presented the results of the CWG activities for development Assessment Tool before the first day of PLA3.

In the External Evaluators Panel Ida Dringo-Horvath (Károli Gáspár University of Reformed Church in Hungary), Ruadhri Neavyn (Higher Education Authority Ireland), Bernd Wachter (Academic Cooperation Association - ACA) made an overview of the work already done by CW groups before the first day of the PLA3 for the 3D Matrix, different Thematic Areas and Horizontal Aspects of teachers’ competences, as well as on the Assessment Tool and they shared with the project participants their comments, suggestions, recommendations in order to proceed in more effective and efficient way.

The following questions have been raised, discussed and answered:

  • What do the experts think about the teachers’ competences as outlined in the 3D Matrix?
  • What do the experts think about the Thematic Areas of the Matrix?
  • Is it possible to merge some Thematic Areas? (e.g. Supervising and Mentoring, Teaching and Learning  for Employability etc.)
  • How realistic and effective can be assessment of the teachers’ competencies by different stakeholders as indicated in the 3D Matrix?
  • What should be taken into consideration while developing the indicators for assessing teachers’ performance?
  • What should be taken into consideration while assessing teachers’ performance?

The External Evaluator of Assessment Tool, Uwe Brandenburg (Global Impact Institute, Czech Republic) presented the results of his assessment on the following issues: specificity of self-assessment, assessment scale, 360˚ approach, language and style of the tool, relevance and funcionability of the structure, purpusefulness as formative assessment tool, usability for all potential users, ability of the users to draw relevant results from the tool, style and wording of the statements/indicators to increase their relevance, integration of transversal competences, etc. His presentation was followed by the workshop moderated by him.

In the Students’ Panel and followed Workshop Jakub Grodecki (European Students’ Union), Ana Gvritishvili (Georgian Student Organizations Association - GSOA), Alexander Hedlund (Heriot-Watt University Student Union, UK), László Murai (National Union of Students in Hungary) participated and shared their observations on 3D Matrix and draft Assessment Tool, their ideas and insights on teachers’ performance assessment, experience in assessing teachers’ performance which had positive impact on teaching and learning. The Students Panel was focused on the following questions:

  • How realistic and effective can be assessment of the teachers’ competencies by students as indicated in the 3D Matrix;
  • What should be taken into consideration while assessing teachers’ performance?
  • Which are the good examples of Teachers’ Assessment by Students, etc.

 

2nd DAY of PLA3 - External Assessment of Project Activities, Teachers’ Performance Assessment from Peers’, Management, External Stakeholder Perspective

In the Introductory Part of the Day Vilmos Vass and Irine Darchia briefly summarized the first day of the PLA3 and presented the PLA3 goals and expected outcomes, outlined the expectations from the second day of PLA3. The Padlet Link with main “dilemmas” of the 3D Matrix and Assessment Tool  has been prepared by Szilvia Besze and shared with the PLA3 participants. They have been asked to reflect on the unsolved issues with their comments and suggestions (Padlet Link: https://padlet.com/profformanceproject/dilemmas).

 

In the Panel on Involvement of External Stakeholders in Teachers’ Assessment representatives of three external stakeholders – QA agency, Business, Professional Organization – participated: Andy Gibbs (Expert and Consultant on Higher Education Reform, UK), Tornike Guruli (Archi Group CEO, Employers’ representative expert of the NCEQE, Georgia), VaidotasViliunas (Policy and Project Officer, Lithuania, EURASHE). In the end of the discussion, the participants of the PLA3 voted using the mentimeter. 13 participats voted against assessing the teachers by external stakeholders and only 7 supported this idea, while 12 participants voted for involvement of external stakeholders in this process and 9 voted agaisnt it.

The following questions have been raised, discussed and answered during the Panel:

  • Which organizations can serve as external stakeholders?
  • Can QA, Professional Bodies/Organizations, Potential Employers assess individual teachers?
  • Which competences can be assessed by external stakeholders?
  • Which is an added value for involving the external stakeholders in the assessment of teachers’ performance?
  • Should we eliminate or minimise the external stakeholders perspective?

In the Panel on Teachers Assessment by Peers representatives of Academia Anca Greere (Babes-Bolyai University, Rumania, QAA expert, UK) and Anca Prisacariu (Senior Quality Assurance Consultant, Republic of Mauritius) participated and shared their experience from different national and institutional contexts.

The following questions have been raised, discussed and answered during the Panel and followed interactive Workshop:

  • Why peer assessment is considered sensitive?
  • Which is a communication strategy we need for peer assessment? 
  • Which thematic areas can be assessed by peer?
  • Which competences can be assessed by peer?
  • Which ways, which methods of assessment can be used?
  • Can classroom observation be effective?
  • How can we use the results of peer assessment?

In the Panel on Involvement of Management in Teachers’ Assessment representatives of three different levels of management (Vice-Rector, Dean, Head of Administrative Unit) have participated: István Vilmos Kovács (Vice-rector Metropolitan University, Hungary), Milan Pol (Dean of Faculty of Arts, Masaryk University, Czech Republic), Oliver Vettori (Director of Program Management and Teaching and Learning Affairs, Vienna University of Economics and Business, Austria).

The following questions have been raised, discussed and answered during the Panel and followed Workshop:

  • Who is superior for the teachers? Vice-rector? Dean? Head of Department/institute/chair? Programme Director? QA staff? Professional Development Staff? Lack of direct superior...
  • Do the managers themselves have proper competences to assess teachers’ performance?
  • How can the managers participate in the assessment?
  • How can the results of the assessment be used?

The speakers shared their observations on 3D Matrix, their ideas and insights on teachers’ performance assessment by external stakeholders, peers and management. During the workshops different aspects of 3D Matrix and revision of assessment perspectives have be discussed by the guest speakers and project participants in a very open and vivid discussion.

These three Panel Discussions and Workshops have been moderated by Irine Darchia (Tbilisi State University, Georgia) and have been summarized by Rapporteur Jan Beseda, (Centre for Higher Education Studies, Czech Republic).

In the final Panel on Benchmarking Report and Best Practices the authors of the benchmarking report participated. László Horváth Eötvös (Loránd University, Hungary) presented the Report results, main findings and best practices. The Presentation has been followed by the workshop moderated by Günter Wageneder (University of Salzburg, Austria) and has been summarized by Rapporteur Jan Beseda, (Centre for Higher Education Studies, Czech Republic).

The Discussion was mainly focused on the following questions:   

  • Which general conclusions and recommendations should be drawn from the benchmarking results (to be included in the report)?
  • Which conclusions & recommendations can be drawn from the results for the various PROFFORMANCE activities, especially for the current discussion of the assessment tool?

 

Difficulties, Challenges and Results:

During the Panel Discussions, Q/A Sessions and Workshops the following difficulties and challenges, so-called “dilemmas” have been identified which should be addressed based on the inputs and outputs of the PLA3: 

I. Thematic Areas – and Competence Statements – as indicators?

Decide on thematic areas – merging, renaming, adding “sustainability”.

To Merge:

Teaching and Supervision: combining Teaching and Learning Performance, Supervising and Mentoring;

Teaching Quality and Personal Development: combining Teaching Quality and Professional Development and adding the social skills;

Instrumentalising and Linking Research and Social Engagement for and with teaching: combining Research and Innovation, Third Mission.

To Rename:

Curriculum Design and Development > Curriculum Development

Assessing Student Learning and Learning Outcomes > Assessment and Evaluation

To Focus:

Assess research and social engagement only related to teaching activities.    

To agree on:

Phrasing the statements – which statements to keep/skip/rephrase:

  • Self-assessment: I know, I engage, I am familiar...
  • My colleague, my teacher, my staff member...

Wording, style of competence formulations

  • The teacher knows…
  • The teacher is able…
  • The teacher is aware…

Update 3D Matrix, refine the assessment perspectives after getting feedback from student, peers, external stakeholders, management and finalize the 3D Matrix while working on the assessment tool;

 

II. Assessment Tool Form and Methodology

Self-motivation or 360 degree?

Choose among 5 versions proposed;

Focus on self-assessment?

10 point likert scale?

How to reinforce formative aspect of the tool?

Complexity vs simplicity

Policy requirements vs. reality at HEIs

 

III. Horizontal aspects

Separately or integrated?

Universal design or inclusion and diversity?

Sustainability – to be added?

 

IV. Working Methodology of WGs

Eliminate the separate working groups?

Distribute 6 thematic areas to 6 co-leaders of working groups and proceed with one “core” group with 6 members?

 

V. Assessor Groups

Eliminate or minimise the external stakeholders and managers perspective;

Identify which managers and external stakeholders will be involved in teachers’ assessment; how they will be involved; how the results of the assessment will be used;

Find ways to motivate the assessors – follow-up planning – collection of best practices etc.

 

Main outputs of PLA3

  1. Identified shortcomings of the 3D Matrix, Thematic Areas, Competence Statements, Assessment Perspectives/Assessor Groups and Draft Assessment Methodology in terms of usability and functionality;
  2. Methodological Guidance on teachers performance evaluation in terms of assesment scale, statements/indicators’ style and wording, etc.;
  3. Revised competence areas to be assessed by students, external stakeholders, peers, management;
  4. Decision on using the Results of Benchmarking Report and Best Practices in the Assessment Tool;
  5. Identified challenges to be addressed after the PLA3;
  6. Preparing the grounds for making decisions on existing challengs and dilemmas.

 

Next Steps and PLA3 Follow-up Activities:

  1. Merging Thematic Areas;  
  2. Renaming Thematic Areas;
  3. Deciding on the number and names of Horizontal Aspects;
  4. Agreement on Teachers’ Competence Formulation;
  5. Revising Competence Fields and Statements;
  6. Revising the Assessment Perspectives/Assessor Groups for merged and renamed Thematic Areas;
  7. Updating 3D Matrix;
  8. Agreement on Assessment Methodology;
  9. Deciding on Assessment Scale;
  10. Deciding on Assessment Tool Statements style and wording;
  11. Development of assessment indicators/survey questions from different assessment perspectives and for all Thematic Areas.