Evaluation & Data – Transformation GPS

Evaluation Plan

This evaluation and data collection plan was designed to accompany a change initiative where an organization was working to become more entrepreneurial and competitive. This was a government agency so there was significant resistance to this change. However, the change initiative was critical to the organization’s survival. This page includes an overview of the evaluation framework and the specific data collection points and techniques.

The transformation initiative that we are embarking upon – to become a self-sufficient government agency with entrepreneurial skills – is a whole system change that has four objectives:

Build and maintain a focus on customer service and a culture of entrepreneurship
Examine and redesign our business processes to support an agency that is 95% customer reimbursable
Ensure that our employees have access to training and learning opportunities so they can develop the skills necessary to move the organization toward its goal
Upgrade physical plant facilities and equipment to meet needs of customers

As we examine these four objectives, we can see that training is an important component of each one and so the Kirkpatrick model can be an effective framework for assessing the changes that are taking place within the organization. In fact, applying the Kirkpatrick model during the change initiative will provide excellent data on leading measures, which gives us an opportunity to course correct as necessary. A new and different model will be needed to assess lagging measures of whether we have reached our overall goal of self-sufficiency.

For this case study, we will apply the Kirkpatrick Model in reverse order, starting with the results we want to see, working backward to the reaction employees had to the training. This allows us to connect the evaluation more closely to initiative goals (Petrone, 2017).

Results:

There are a variety of measures that will tell us if the training components of the change initiative are effective. How satisfied are our customers? How many patents are in the pipeline? What is the win rate of proposals? The evaluation tools we will use to assess results include:

1. Conduct a customer satisfaction survey

2. Collect data on invention disclosures, patent applications, and patent awards

3. Assess win rate on proposals submitted and then track how many dollars are in the pipeline

4. Track safety metrics on machine operation

Behavior:

To assess behavior, we will incorporate relevant metrics into all annual evaluations, including those of the leaders. Additionally, we will require senior leaders to coach their management team to ensure they are modeling the desired behaviors, including adoption of new processes

Learning:

There are a wide variety of new skills and competencies that will be needed by employees for a successful transformation. Customer service, strategic thinking, sales, navigating government bureaucracy, and other skills will be taught. We will assess these new skills using informal quizzes, hands-on assignments, and direct observation by supervisors. Absence of skills will be addressed through individualized training.

Reaction:

Immediately after a training we will administer a survey that assess the content and delivery of the training. Additionally, we will conduct follow-up focus groups with trainees to reinforce the learning and to assess retention.

Jones, M. C., & Rothwell, W. J. (2018). Evaluating Organization Development: How to Ensure and Sustain the Successful Transformation (1st ed.). Productivity Press.

Kirkpatrick’s Four-Level Training Evaluation Model: Analyzing Training Effectiveness. (n.d.). Mind Tools. Retrieved February 13, 2021, from https://www.mindtools.com/pages/article/kirkpatrick.htm

Kirkpatrick, D. (2013, February 27). The Kirkpatrick Model. Training Industry. https://trainingindustry.com/wiki/measurement-and-analytics/the-kirkpatrick-model/.

Petrone, P. (2017). The best way to use the kirkpatrick model, the most common way to measure learning. https://www.linkedin.com/business/learning/blog/learning-and-development/the-best-way-to-use-the-kirkpatrick-model-the-most-common-way.

Data Collection Plan

GOAL	Success Indicators	Type of Data	Data gathering method	Frequency of Data collection	Who owns the data?	Who will collect the data?	Resources	How display and share the data?	Dashboard?
Organization achieves self-sufficiency and preserves core capabilities	Annual revenue: Gross laboratory revenue during each fiscal year, measured by quarter	Quantitative, lagging	CFO runs report on gross revenue from financial system	Quarterly	CFO	Technical Director, who delivers it to consultant	Online financial systems	As dollar value, combined with projected revenue from proposal pipeline. Included in the Quarterly Report submitted to the Director.	Yes
	Projected revenue: Gross revenue in the sales pipeline with greater than 50% chance of win	Quantitative, lagging	Team leads report up to Division. Chiefs, who consolidate data for division	Quarterly	Division Chiefs	Team leads, division chiefs, delivered to consultant	Balanced scorecard tracking software	As dollar value, combined with actual gross revenue. Included in the Quarterly In Process Review.	Yes – folded into annual revenue
	Number of employees: Number of full-time government employees and number of labor hours provided by contractual employees	Quantitative, lagging	“Snapshot in time” of total employees, as indicated in payroll system. Contractual labor provided by contracting officers	Annually	Human Resources and Contracting firms	Human Resources and Contracting Officers	Standard definitions of employees. Sharepoint site accessible by external parties.	Employee headcount will be depicted as a whole number. Number of contracting labor hours will be displayed in fractions of full-time equivalents. Included in the Annual Report.	Yes
	Utilization of equipment: Run time on laboratory equipment as a percentage of total potential run time.	Quantitative, lagging	Usage of equipment, such as hoods, NMRs, and other lab equipment will be tracked by operators	Collected daily	Laboratory team leads	Laboratory technicians	Paper system that can be scanned into electronic system and merged with data from each team	In numbers of hours as ratio of total potential run time. Submitted as part of the Quarterly In Process Review.	Yes
	Number of days without accidents: Time between accidents or leaks	Quantitative, lagging	Existing accident reporting system	Daily	Safety and Occupational Health Office	Team leads	Safety tracking system	In numbers of days per year. Submitted in the Quarterly In Process Review.	Yes
OBJECTIVES	Success Indicators	Type of Data	Data gathering method	Frequency of Data collection	Who owns the data?	Who will collect the data?	Resources	How display the data?	Dashboard?
Build and maintain a focus on customer service and a culture of entrepreneurship	Customer satisfaction scores	Qualitative and quantitative, leading	1.Survey using Likert scale 2.Customer interviews	Annual	Business Management Team	PIs will distribute to clients and Business Management Team will collate data and conduct interviews	Survey instrument developed and piloted. Focus group protocol created and interviewers taught how to conduct focus groups	Excel spread sheet showing data and customer responses. Reviewed at the Annual Directors’ Strategy Retreat.	No
	Number of invention disclosures, patent applications, partnership agreements	Quantitative, leading	Report from technology transfer office database	Quarterly	Technology Transfer Office	Technology Transfer Office	Database	Lists created and submitted for the Annual Report.	Yes
Examine and redesign our business processes to support an agency that is 95% customer reimbursable	Process improvement initiatives: Employees look for opportunities to streamline processes	Qualitative, leading	Team Meetings	Monthly	Employees	Team leads	Awards program to reward people who achieve efficiencies by eliminating unnecessary processes	Report narratives included in each Quarterly In Process Review	No
	Number of days to complete a transaction: Repetitive processes can be automated and streamlined	Quantitative, leading	Identify processes, such as document review. Gather data using existing tracking systems	Quarterly	Employees	Team leads	Tracking systems, employee surveys and individual dialogue	Report narratives included in Quarterly In Process Review	No
Ensure that our employees have access to training and learning opportunities so they can develop the skills necessary to move the organization toward its goal	Percentage of employees accessing customer service training	Quantitative, leading	Training database review	Quarterly	Human Resources	Team leads	Training database	Displayed as percentage in Quarterly In Process Review	Yes
	Percentage of employees with active IDPs	Quantitative, leading	Employee database	Quarterly	Team leads	Human Resource Office	Employee database and IDP form	Displayed as percentage in Quarterly In Process Review	Yes
	Employee satisfaction	Qualitative, leading	Employee survey	Annually	Office of Personnel Management	Office of Personnel Management	Data collected from OPM	Report included in Annual Report and data compared to prior year	No
Upgrade physical plant facilities and equipment to meet needs of customers	Projects outsourced due to facilities, equipment issues	Quantitative, leading	Monitoring and tracking of expenditures	Annually	Contracting officers	Contracting officers	Input from team leads to evaluate reason for outsourcing	As a list of equipment outsourced	No