Senior Director, Assessment

The PA Education Association (PAEA) is seeking a Senior Director, Assessment to lead, develop, and grow a high-performing assessment team while guiding the organization's assessment strategy. The ideal candidate is an experienced people leader with a proven record of building, mentoring, and expanding teams that deliver excellence, collaboration, and continuous improvement. This leader also brings a strong foundation in psychometrics and educational measurement and a commitment to innovation and mission-driven impact.

The Senior Director, Assessment provides strategic and operational leadership for PAEA's assessment portfolio, translating organizational goals into assessment strategy and operational plans and ensuring that high-quality assessment programs and services are developed, delivered, evaluated, and continuously improved. Working closely with the Chief Data Officer and Principal Psychometrician, this position oversees assessment operations, psychometric and measurement activities, exam development, innovation initiatives, and external partnerships and vendors, ensuring that PAEA assessments remain valid, reliable, fair, defensible, and aligned with organizational strategy. The Senior Director also supports the development of future assessment products and services and represents PAEA within the broader assessment, measurement, and health professions education communities.

Please include your resume and cover letter when applying for this position.

This position may be exposed to sensitive and confidential information daily, this individual is responsible for maintaining the confidentiality that knowledge of such warrants. In this position, employees must respect and protect the privacy, confidentiality, and security of all personal information to which they have access in the performance of their duties for PAEA, in accordance with PAEA’s Privacy Policy and the PAEA Employee Handbook that all employees must sign. 

Essential Functions:

Reasonable accommodations may be made to enable individuals with disabilities to perform these essential functions.

Assessment Strategy and Innovation

• Lead the strategy, planning, implementation, evaluation, and continuous improvement of current and future assessment products and services, aligning short- and long-term priorities with organizational goals and member needs.
• Monitor developments in educational measurement, psychometrics, assessment technology, artificial intelligence, and health professions education to identify opportunities to improve assessment design, delivery, reporting, and product development.
• Collaborate with organizational leaders on assessment initiatives, business planning, and product development opportunities.

Assessment Program and Measurement Oversight

• Oversee the full assessment lifecycle, including blueprint development and evaluation, item banking, test construction, administration and delivery, scoring, scaling, equating, standard setting, reporting, quality assurance, and technical documentation.
• Ensure assessments are valid, reliable, fair, accessible, secure, and legally defensible, and comply with testing standards, industry best practices, organizational policies, and governance requirements. Establish and maintain related assessment policies and procedures.
• Lead psychometric analyses and measurement research, including item and test performance monitoring, scoring methodology, standard setting, applied research, corrective action, and continuous evaluation, to support evidence-based decision-making.
• Guide the development and use of data tools, analytical solutions, and measurement technologies, and communicate technical findings, score interpretation guidance, and assessment information to technical and nontechnical audiences.

Team Leadership and Development

• Supervise, coach, and mentor Assessment staff; manage performance; and foster alignment across psychometric, operational, and strategic functions.
• Support staff development and participation in professional organizations, conferences, publications, and research, while strengthening organizational capacity through cross-training, knowledge sharing, and succession planning.

Operations and Partnerships

• Oversee exam development activities, production schedules, budgets, resource allocation, volunteer engagement, and vendor performance.
• Coordinate the work of assessment staff, subject matter experts, vendors, and external partners, and cultivate relationships that advance PAEA’s assessment strategy.
• Implement process improvements that enhance quality, efficiency, scalability, sustainability, and the stakeholder experience, and support assessment-related meetings, summits, workshops, and professional development activities.

External and Professional Leadership

• Represent PAEA through presentations, publications, workshops, consultation, and professional service within the testing, psychometrics, educational measurement, credentialing, and health professions education communities.
• Strengthen members’ assessment literacy through educational programming, consultation, score interpretation resources, and the dissemination of best practices.

Security and Compliance

• Protect the confidentiality, integrity, and security of assessment materials, data, and processes, and maintain related policies, procedures, and risk controls.
• Coordinate incident response, risk mitigation, and compliance activities with organizational leaders, legal counsel, and other relevant stakeholders.

• Doctoral degree (PhD, EdD, or equivalent) in Psychometrics, Educational Measurement, Quantitative Psychology, Educational Psychology, Statistics, Research Methodology, or a closely related field.
• Minimum of 8 years of progressively responsible experience in assessment, psychometrics, educational measurement, testing, or related fields.
• Minimum of 3 years of supervisory or leadership experience managing professional staff and complex projects.
• Deep knowledge of Classical Test Theory, Item Response Theory, Rasch measurement, standard setting, test development, validation, scoring, equating, and assessment reporting.
• Demonstrated experience leading assessment programs, initiatives, or portfolios.
• Experience managing external vendors, consultants, and stakeholder relationships.
• Strong strategic planning, organizational leadership, and project management skills.
• Excellent written and verbal communication skills, including the ability to explain technical concepts to non-technical audiences.
• Ability to manage multiple priorities and lead complex cross-functional initiatives.
• Strong analytical, problem-solving, and decision-making skills.
• Demonstrated commitment to assessment quality, fairness, accessibility, and continuous improvement.

Preferred:

• Experience in health professions education and assessment.
• Experience developing innovative assessment products, technologies, or services.
• Record of professional presentations, publications, or contributions to the assessment and measurement field.
• Experience managing assessment security programs and policies.
• Experience with R, data visualization tools, assessment platforms, and modern analytical technologies.
• Active involvement in professional measurement or testing organizations.