This evaluation and data collection plan was designed to accompany a change initiative where an organization was working to become more entrepreneurial and competitive. This was a government agency so there was significant resistance to this change. However, the change initiative was critical to the organization’s survival. This page includes an overview of the evaluation framework and the specific data collection points and techniques.
The transformation initiative that we are embarking upon – to become a self-sufficient government agency with entrepreneurial skills – is a whole system change that has four objectives:
As we examine these four objectives, we can see that training is an important component of each one and so the Kirkpatrick model can be an effective framework for assessing the changes that are taking place within the organization. In fact, applying the Kirkpatrick model during the change initiative will provide excellent data on leading measures, which gives us an opportunity to course correct as necessary. A new and different model will be needed to assess lagging measures of whether we have reached our overall goal of self-sufficiency.
For this case study, we will apply the Kirkpatrick Model in reverse order, starting with the results we want to see, working backward to the reaction employees had to the training. This allows us to connect the evaluation more closely to initiative goals (Petrone, 2017).
Results:
There are a variety of measures that will tell us if the training components of the change initiative are effective. How satisfied are our customers? How many patents are in the pipeline? What is the win rate of proposals? The evaluation tools we will use to assess results include:
1. Conduct a customer satisfaction survey
2. Collect data on invention disclosures, patent applications, and patent awards
3. Assess win rate on proposals submitted and then track how many dollars are in the pipeline
4. Track safety metrics on machine operation
Behavior:
To assess behavior, we will incorporate relevant metrics into all annual evaluations, including those of the leaders. Additionally, we will require senior leaders to coach their management team to ensure they are modeling the desired behaviors, including adoption of new processes
Learning:
There are a wide variety of new skills and competencies that will be needed by employees for a successful transformation. Customer service, strategic thinking, sales, navigating government bureaucracy, and other skills will be taught. We will assess these new skills using informal quizzes, hands-on assignments, and direct observation by supervisors. Absence of skills will be addressed through individualized training.
Reaction:
Immediately after a training we will administer a survey that assess the content and delivery of the training. Additionally, we will conduct follow-up focus groups with trainees to reinforce the learning and to assess retention.
Jones, M. C., & Rothwell, W. J. (2018). Evaluating Organization Development: How to Ensure and Sustain the Successful Transformation (1st ed.). Productivity Press.
Kirkpatrick’s Four-Level Training Evaluation Model: Analyzing Training Effectiveness. (n.d.). Mind Tools. Retrieved February 13, 2021, from https://www.mindtools.com/pages/article/kirkpatrick.htm
Kirkpatrick, D. (2013, February 27). The Kirkpatrick Model. Training Industry. https://trainingindustry.com/wiki/measurement-and-analytics/the-kirkpatrick-model/.
Petrone, P. (2017). The best way to use the kirkpatrick model, the most common way to measure learning. https://www.linkedin.com/business/learning/blog/learning-and-development/the-best-way-to-use-the-kirkpatrick-model-the-most-common-way.
GOAL |
Success Indicators |
Type of Data |
Data gathering method |
Frequency of Data collection |
Who owns the data? |
Who will collect the data? |
Resources |
How display and share the data? |
Dashboard? |
Organization achieves self-sufficiency and preserves core capabilities |
Annual revenue: Gross laboratory revenue during each fiscal year, measured by quarter
|
Quantitative, lagging |
CFO runs report on gross revenue from financial system |
Quarterly |
CFO |
Technical Director, who delivers it to consultant |
Online financial systems |
As dollar value, combined with projected revenue from proposal pipeline. Included in the Quarterly Report submitted to the Director. |
Yes |
Projected revenue: Gross revenue in the sales pipeline with greater than 50% chance of win |
Quantitative, lagging |
Team leads report up to Division. Chiefs, who consolidate data for division |
Quarterly |
Division Chiefs |
Team leads, division chiefs, delivered to consultant |
Balanced scorecard tracking software |
As dollar value, combined with actual gross revenue. Included in the Quarterly In Process Review. |
Yes – folded into annual revenue |
|
Number of employees: Number of full-time government employees and number of labor hours provided by contractual employees |
Quantitative, lagging |
“Snapshot in time” of total employees, as indicated in payroll system. Contractual labor provided by contracting officers |
Annually |
Human Resources and Contracting firms |
Human Resources and Contracting Officers |
Standard definitions of employees. Sharepoint site accessible by external parties. |
Employee headcount will be depicted as a whole number. Number of contracting labor hours will be displayed in fractions of full-time equivalents. Included in the Annual Report. |
Yes |
|
Utilization of equipment: Run time on laboratory equipment as a percentage of total potential run time. |
Quantitative, lagging |
Usage of equipment, such as hoods, NMRs, and other lab equipment will be tracked by operators |
Collected daily |
Laboratory team leads |
Laboratory technicians |
Paper system that can be scanned into electronic system and merged with data from each team |
In numbers of hours as ratio of total potential run time. Submitted as part of the Quarterly In Process Review. |
Yes |
|
Number of days without accidents: Time between accidents or leaks |
Quantitative, lagging |
Existing accident reporting system |
Daily |
Safety and Occupational Health Office |
Team leads |
Safety tracking system |
In numbers of days per year. Submitted in the Quarterly In Process Review. |
Yes |
|
OBJECTIVES |
Success Indicators |
Type of Data |
Data gathering method |
Frequency of Data collection |
Who owns the data? |
Who will collect the data? |
Resources |
How display the data? |
Dashboard? |
Build and maintain a focus on customer service and a culture of entrepreneurship
|
Customer satisfaction scores
|
Qualitative and quantitative, leading |
1.Survey using Likert scale 2.Customer interviews |
Annual |
Business Management Team |
PIs will distribute to clients and Business Management Team will collate data and conduct interviews |
Survey instrument developed and piloted. Focus group protocol created and interviewers taught how to conduct focus groups |
Excel spread sheet showing data and customer responses. Reviewed at the Annual Directors’ Strategy Retreat. |
No |
Number of invention disclosures, patent applications, partnership agreements |
Quantitative, leading |
Report from technology transfer office database |
Quarterly |
Technology Transfer Office |
Technology Transfer Office |
Database |
Lists created and submitted for the Annual Report. |
Yes |
|
Examine and redesign our business processes to support an agency that is 95% customer reimbursable
|
Process improvement initiatives: Employees look for opportunities to streamline processes |
Qualitative, leading |
Team Meetings |
Monthly |
Employees |
Team leads |
Awards program to reward people who achieve efficiencies by eliminating unnecessary processes
|
Report narratives included in each Quarterly In Process Review |
No |
Number of days to complete a transaction: Repetitive processes can be automated and streamlined
|
Quantitative, leading |
Identify processes, such as document review. Gather data using existing tracking systems |
Quarterly |
Employees |
Team leads |
Tracking systems, employee surveys and individual dialogue |
Report narratives included in Quarterly In Process Review |
No |
|
Ensure that our employees have access to training and learning opportunities so they can develop the skills necessary to move the organization toward its goal
|
Percentage of employees accessing customer service training |
Quantitative, leading |
Training database review |
Quarterly |
Human Resources |
Team leads |
Training database
|
Displayed as percentage in Quarterly In Process Review |
Yes |
Percentage of employees with active IDPs |
Quantitative, leading |
Employee database |
Quarterly |
Team leads |
Human Resource Office |
Employee database and IDP form |
Displayed as percentage in Quarterly In Process Review |
Yes |
|
Employee satisfaction |
Qualitative, leading |
Employee survey |
Annually |
Office of Personnel Management |
Office of Personnel Management |
Data collected from OPM |
Report included in Annual Report and data compared to prior year |
No |
|
Upgrade physical plant facilities and equipment to meet needs of customers
|
Projects outsourced due to facilities, equipment issues
|
Quantitative, leading |
Monitoring and tracking of expenditures |
Annually |
Contracting officers |
Contracting officers |
Input from team leads to evaluate reason for outsourcing |
As a list of equipment outsourced |
No |