Evaluation & Data

Evaluation Plan

This evaluation and data collection plan was designed to accompany a change initiative where an organization was working to become more entrepreneurial and competitive. This was a government agency so there was significant resistance to this change. However, the change initiative was critical to the organization’s survival. This page includes an overview of the evaluation framework and the specific data collection points and techniques.

 

The transformation initiative that we are embarking upon – to become a self-sufficient government agency with entrepreneurial skills – is a whole system change that has four objectives:

  1. Build and maintain a focus on customer service and a culture of entrepreneurship
  2. Examine and redesign our business processes to support an agency that is 95% customer reimbursable
  3. Ensure that our employees have access to training and learning opportunities so they can develop the skills necessary to move the organization toward its goal
  4. Upgrade physical plant facilities and equipment to meet needs of customers

As we examine these four objectives, we can see that training is an important component of each one and so the Kirkpatrick model can be an effective framework for assessing the changes that are taking place within the organization. In fact, applying the Kirkpatrick model during the change initiative will provide excellent data on leading measures, which gives us an opportunity to course correct as necessary. A new and different model will be needed to assess lagging measures of whether we have reached our overall goal of self-sufficiency.

For this case study, we will apply the Kirkpatrick Model in reverse order, starting with the results we want to see, working backward to the reaction employees had to the training. This allows us to connect the evaluation more closely to initiative goals (Petrone, 2017).

Results:

There are a variety of measures that will tell us if the training components of the change initiative are effective. How satisfied are our customers? How many patents are in the pipeline? What is the win rate of proposals? The evaluation tools we will use to assess results include:

1. Conduct a customer satisfaction survey

2. Collect data on invention disclosures, patent applications, and patent awards

3. Assess win rate on proposals submitted and then track how many dollars are in the pipeline

4. Track safety metrics on machine operation

Behavior:

To assess behavior, we will incorporate relevant metrics into all annual evaluations, including those of the leaders. Additionally, we will require senior leaders to coach their management team to ensure they are modeling the desired behaviors, including adoption of new processes

Learning:

There are a wide variety of new skills and competencies that will be needed by employees for a successful transformation. Customer service, strategic thinking, sales, navigating government bureaucracy, and other skills will be taught. We will assess these new skills using informal quizzes, hands-on assignments, and direct observation by supervisors. Absence of skills will be addressed through individualized training.

Reaction:

Immediately after a training we will administer a survey that assess the content and delivery of the training. Additionally, we will conduct follow-up focus groups with trainees to reinforce the learning and to assess retention.

Jones, M. C., & Rothwell, W. J. (2018). Evaluating Organization Development: How to Ensure and Sustain the Successful Transformation (1st ed.). Productivity Press.

Kirkpatrick’s Four-Level Training Evaluation Model: Analyzing Training Effectiveness. (n.d.). Mind Tools. Retrieved February 13, 2021, from https://www.mindtools.com/pages/article/kirkpatrick.htm

Kirkpatrick, D. (2013, February 27). The Kirkpatrick Model. Training Industry. https://trainingindustry.com/wiki/measurement-and-analytics/the-kirkpatrick-model/.

Petrone, P. (2017). The best way to use the kirkpatrick model, the most common way to measure learning. https://www.linkedin.com/business/learning/blog/learning-and-development/the-best-way-to-use-the-kirkpatrick-model-the-most-common-way.

 

Data Collection Plan

GOAL

Success Indicators

Type of Data

Data gathering method

Frequency of Data collection

Who owns the data?

Who will collect the data?

Resources

How display and share the data?

Dashboard?

Organization achieves self-sufficiency and preserves core capabilities

Annual revenue: Gross laboratory revenue during each fiscal year, measured by quarter

 

Quantitative, lagging

CFO runs report on gross revenue from financial system

Quarterly

CFO

Technical Director, who delivers it to consultant

Online financial systems

As dollar value, combined with projected revenue from proposal pipeline. Included in the Quarterly Report submitted to the Director.

Yes

Projected revenue: Gross revenue in the sales pipeline with greater than 50% chance of win

Quantitative, lagging

Team leads report up to Division. Chiefs, who consolidate data for division

Quarterly

Division Chiefs

Team leads, division chiefs, delivered to consultant

Balanced scorecard tracking software

As dollar value, combined with actual gross revenue. Included in the Quarterly In Process Review.

Yes – folded into annual revenue

Number of employees: Number of full-time government employees and number of labor hours provided by contractual employees

Quantitative, lagging

“Snapshot in time” of total employees, as indicated in payroll system. Contractual labor provided by contracting officers

Annually

Human Resources and Contracting firms

Human Resources and Contracting Officers

Standard definitions of employees. Sharepoint site accessible by external parties.

Employee headcount will be depicted as a whole number. Number of contracting labor hours will be displayed in fractions of full-time equivalents. Included in the Annual Report.

Yes

Utilization of equipment: Run time on laboratory equipment as a percentage of total potential run time.

Quantitative, lagging

Usage of equipment, such as hoods, NMRs, and other lab equipment will be tracked by operators

Collected daily

Laboratory team leads

Laboratory technicians

Paper system that can be scanned into electronic system and merged with data from each team

In numbers of hours as ratio of total potential run time. Submitted as part of the Quarterly In Process Review.

Yes

Number of days without accidents: Time between accidents or leaks

Quantitative, lagging

Existing accident reporting system

Daily

Safety and Occupational Health Office

Team leads

Safety tracking system

In numbers of days per year. Submitted in the Quarterly In Process Review.

Yes

OBJECTIVES

Success Indicators

Type of Data

Data gathering method

Frequency of Data collection

Who owns the data?

Who will collect the data?

Resources

How display the data?

Dashboard?

Build and maintain a focus on customer service and a culture of entrepreneurship

 

Customer satisfaction scores

 

Qualitative and quantitative, leading

1.Survey using Likert scale

2.Customer interviews

Annual

Business Management Team

PIs will distribute to clients and Business Management Team will collate data and conduct interviews

Survey instrument developed and piloted. Focus group protocol created and interviewers taught how to conduct focus groups

Excel spread sheet showing data and customer responses. Reviewed at the Annual Directors’ Strategy Retreat.

No

Number of invention disclosures, patent applications, partnership agreements

Quantitative, leading

Report from technology transfer office database

Quarterly

Technology Transfer Office

Technology Transfer Office

Database

Lists created and submitted for the Annual Report.

Yes

Examine and redesign our business processes to support an agency that is 95% customer reimbursable

 

Process improvement initiatives: Employees look for opportunities to streamline processes

Qualitative, leading

Team Meetings

Monthly

Employees

Team leads

Awards program to reward people who achieve efficiencies by eliminating unnecessary processes

 

Report narratives included in each Quarterly In Process Review

No

Number of days to complete a transaction: Repetitive processes can be automated and streamlined

 

Quantitative, leading

Identify processes, such as document review. Gather data using existing tracking systems

Quarterly

Employees

Team leads

Tracking systems, employee surveys and individual dialogue

Report narratives included in Quarterly In Process Review

No

Ensure that our employees have access to training and learning opportunities so they can develop the skills necessary to move the organization toward its goal

 

Percentage of employees accessing customer service training

Quantitative, leading

Training database review

Quarterly

Human Resources

Team leads

Training database

 

Displayed as percentage in Quarterly In Process Review

Yes

Percentage of employees with active IDPs

Quantitative, leading

Employee database

Quarterly

Team leads

Human Resource Office

Employee database and IDP form

Displayed as percentage in Quarterly In Process Review

Yes

Employee satisfaction

Qualitative, leading

Employee survey

Annually

Office of Personnel Management

Office of Personnel Management

Data collected from OPM

Report included in Annual Report and data compared to prior year

No

Upgrade physical plant facilities and equipment to meet needs of customers

 

Projects outsourced due to facilities, equipment issues

 

 

Quantitative, leading

Monitoring and tracking of expenditures

Annually

Contracting officers

Contracting officers

Input from team leads to evaluate reason for outsourcing

As a list of equipment outsourced

No