5
Steps to calculate ROI
1. Obtain data to demonstrate the changes in behavior, e.g., that
gathered through surveys, questionnaires, on-the-job observations,
post-program interviews, focus groups, performance monitoring.
2. Isolate the effect of training, e.g., through the use of control groups,
trend lines, forecasting models.
3. Convert the data to monetary value by focusing on a unit of
measure, determining a value for that unit, calculating the change
in performance data, determining the annual amount for the
change, and calculating the total value of the improvement.
4. Tabulate the program costs: this is the value of the cost of taking
people away from their jobs for the training, including salary and
benefits.
5. Calculate the return on investment by dividing the net benefits
by the costs times 100 percent
Ekta
From India, Ahmadabad
1. Obtain data to demonstrate the changes in behavior, e.g., that
gathered through surveys, questionnaires, on-the-job observations,
post-program interviews, focus groups, performance monitoring.
2. Isolate the effect of training, e.g., through the use of control groups,
trend lines, forecasting models.
3. Convert the data to monetary value by focusing on a unit of
measure, determining a value for that unit, calculating the change
in performance data, determining the annual amount for the
change, and calculating the total value of the improvement.
4. Tabulate the program costs: this is the value of the cost of taking
people away from their jobs for the training, including salary and
benefits.
5. Calculate the return on investment by dividing the net benefits
by the costs times 100 percent
Ekta
From India, Ahmadabad
Ekta neatly summarises the essence of Jack Phillips' ROI approach. Thank you Ekta!
The Phillips approach has a number of useful characteristics but if you don't appreciate the subtleties of some of them you can create problems if you don't watch out for certain danger spots. Let me explain further...
1. Obtain data to demonstrate the changes in behavior, e.g., that
gathered through surveys, questionnaires, on-the-job observations,
post-program interviews, focus groups, performance monitoring.
The potential danger to watch out for with Step #1 is root cause - you need to KNOW what the root causes are for the changes in behaviour - just observing the changes, getting survey and interview data, esp. after the training, will only show CORRELATION, not CAUSATION. Get this bit right and you SHOULD have a number of measures of performance that are readily convertable in to financial values, which will be useful later on.
2. Isolate the effect of training, e.g., through the use of control groups,
trend lines, forecasting models.
There are a number of dangers to watch out for in Step #2 as well. First, it may not be appropriate to isolate the effects of training. If the training is part of a project to address performance, as is almost always the case, and if this training is a 'must have' part of the project, i.e. no training means no required improvement in performance, then it is meaningless to isolate the effects of training, you have to look at the project as a whole - and neither can you necessarily claim 100% of the benefits of the project simply because without the training the project will fail to deliver the required performance improvement. That said, if you do the root cause analysis properly you will get a solid measure of how much of the problem can be addressed by the training aspect - so there will be times when isolating the effect of training does make sense - just be sure you know what you are doing and why - there are plenty of CEOs and CFOs that will shoot your ROI claims down in flames because you have not been rigorous enough in this respect.
Secondly, trend lines again only show CORRELATION, not CAUSATION - unless you have done a thorough root cause analysis and your plots of trends are on the right data. Further more. trend line plots may not show easily problems in the processes that deliver the performance in the first place - they are not the best tools for showing if the process is 'in control'. If the process is not 'in control' then no amount of forecasting trends and extrapolating will get you credible data - where forecasts appear to be accurate will be out of sheer luck - nothing more.
Control groups are fine to use PROVIDED you account for all the major factors. It is almost always the case that comparing one group that has had the training with another that has not had the training will tell you NOTHING other than the fact that there may be a difference in performance between the two. It CANNOT tell you WHY the difference is there because you have not taken steps to consider and robustly discount other factors that could affect performance.
3. Convert the data to monetary value by focusing on a unit of
measure, determining a value for that unit, calculating the change
in performance data, determining the annual amount for the
change, and calculating the total value of the improvement.
If you recall my comments from Step #1, if you have a decent root cause analysis you SHOULD have some measures of performance that are readily convertable to financial values.
4. Tabulate the program costs: this is the value of the cost of taking
people away from their jobs for the training, including salary and
benefits.
Yes - and also the development and delivery costs of the training course, and any accommodation and travel costs that go with it.
5. Calculate the return on investment by dividing the net benefits
by the costs times 100 percent
Because these figures are based on a comprehensive root cause analysis, and robut use of control groups for example, and supported if you wish by estimates of contribution by operators and other experts impacted by the training, you can rest easy in the knowledge that there isn't a CEO or CFO alive that can shoot your claims down - except to fulfil a hidden agenda, in which case normal reasoning goes out the window and it's all down to politics!!
I speak from some experience - I followed the Phillips approach and used control groups and trend lines and observations of behaviour - presented them to the Board of Directors, who then promptly shot holes in the claims I was making - because I couldn't PROVE my claims. Along with 85 other people I lost my job during the next difficult trading period about 3 months later because I couldn't PROVE the contribution of the training function - just the fact that we were spending a lot of money on training.
Use the Phillips ROI approach, but like all methods (including my own 5 Boxes) use it with care and from a position of intimate knowledge.
Good luck!
Regards
Martin
From United Kingdom,
The Phillips approach has a number of useful characteristics but if you don't appreciate the subtleties of some of them you can create problems if you don't watch out for certain danger spots. Let me explain further...
1. Obtain data to demonstrate the changes in behavior, e.g., that
gathered through surveys, questionnaires, on-the-job observations,
post-program interviews, focus groups, performance monitoring.
The potential danger to watch out for with Step #1 is root cause - you need to KNOW what the root causes are for the changes in behaviour - just observing the changes, getting survey and interview data, esp. after the training, will only show CORRELATION, not CAUSATION. Get this bit right and you SHOULD have a number of measures of performance that are readily convertable in to financial values, which will be useful later on.
2. Isolate the effect of training, e.g., through the use of control groups,
trend lines, forecasting models.
There are a number of dangers to watch out for in Step #2 as well. First, it may not be appropriate to isolate the effects of training. If the training is part of a project to address performance, as is almost always the case, and if this training is a 'must have' part of the project, i.e. no training means no required improvement in performance, then it is meaningless to isolate the effects of training, you have to look at the project as a whole - and neither can you necessarily claim 100% of the benefits of the project simply because without the training the project will fail to deliver the required performance improvement. That said, if you do the root cause analysis properly you will get a solid measure of how much of the problem can be addressed by the training aspect - so there will be times when isolating the effect of training does make sense - just be sure you know what you are doing and why - there are plenty of CEOs and CFOs that will shoot your ROI claims down in flames because you have not been rigorous enough in this respect.
Secondly, trend lines again only show CORRELATION, not CAUSATION - unless you have done a thorough root cause analysis and your plots of trends are on the right data. Further more. trend line plots may not show easily problems in the processes that deliver the performance in the first place - they are not the best tools for showing if the process is 'in control'. If the process is not 'in control' then no amount of forecasting trends and extrapolating will get you credible data - where forecasts appear to be accurate will be out of sheer luck - nothing more.
Control groups are fine to use PROVIDED you account for all the major factors. It is almost always the case that comparing one group that has had the training with another that has not had the training will tell you NOTHING other than the fact that there may be a difference in performance between the two. It CANNOT tell you WHY the difference is there because you have not taken steps to consider and robustly discount other factors that could affect performance.
3. Convert the data to monetary value by focusing on a unit of
measure, determining a value for that unit, calculating the change
in performance data, determining the annual amount for the
change, and calculating the total value of the improvement.
If you recall my comments from Step #1, if you have a decent root cause analysis you SHOULD have some measures of performance that are readily convertable to financial values.
4. Tabulate the program costs: this is the value of the cost of taking
people away from their jobs for the training, including salary and
benefits.
Yes - and also the development and delivery costs of the training course, and any accommodation and travel costs that go with it.
5. Calculate the return on investment by dividing the net benefits
by the costs times 100 percent
Because these figures are based on a comprehensive root cause analysis, and robut use of control groups for example, and supported if you wish by estimates of contribution by operators and other experts impacted by the training, you can rest easy in the knowledge that there isn't a CEO or CFO alive that can shoot your claims down - except to fulfil a hidden agenda, in which case normal reasoning goes out the window and it's all down to politics!!
I speak from some experience - I followed the Phillips approach and used control groups and trend lines and observations of behaviour - presented them to the Board of Directors, who then promptly shot holes in the claims I was making - because I couldn't PROVE my claims. Along with 85 other people I lost my job during the next difficult trading period about 3 months later because I couldn't PROVE the contribution of the training function - just the fact that we were spending a lot of money on training.
Use the Phillips ROI approach, but like all methods (including my own 5 Boxes) use it with care and from a position of intimate knowledge.
Good luck!
Regards
Martin
From United Kingdom,
Thanks Mr.Martin for your valuable inputs.
But i've certain constraints in calculation of ROI of training.
The first and foremost constraint which i'm facing is that i'm here in this organization for approximately 15days more since i'm doing summer training here and the program for which ROI needs to be calculated is currently in process.
Secondly, I don't have much control in creation of control groups etc. The people to whom training has to be imparted has already been decided. In fact they are currently getting the training.
I wanted to know that is there any way with which I can calculate short-term ROI i.e. short term returns of a training program with respect to the investment made in it.
Thanks for your help.
Reena
From India, Mumbai
But i've certain constraints in calculation of ROI of training.
The first and foremost constraint which i'm facing is that i'm here in this organization for approximately 15days more since i'm doing summer training here and the program for which ROI needs to be calculated is currently in process.
Secondly, I don't have much control in creation of control groups etc. The people to whom training has to be imparted has already been decided. In fact they are currently getting the training.
I wanted to know that is there any way with which I can calculate short-term ROI i.e. short term returns of a training program with respect to the investment made in it.
Thanks for your help.
Reena
From India, Mumbai
Hi Mr.Martin,
The training program of which I'm supposed to calculate ROI, is on SDH Basics. SDH stands for Synchronous Digital Hierarchy. It is a technical program.
There are 4 measures on which we are planning to evaluate how much the employee has learned after training. These are: -
1. Installation & Commisioning
2. Problem Diagnosis Skills
3. Testing with Customer
4. Equipment Handling
Since i'm not a technical person, so I don't have a knowledge of all this. But this training program is real important for the organization since the company for which i'm doing the project is in telecom sector.
Waiting for your suggestions.
Thanks a lot.
Reena
From India, Mumbai
The training program of which I'm supposed to calculate ROI, is on SDH Basics. SDH stands for Synchronous Digital Hierarchy. It is a technical program.
There are 4 measures on which we are planning to evaluate how much the employee has learned after training. These are: -
1. Installation & Commisioning
2. Problem Diagnosis Skills
3. Testing with Customer
4. Equipment Handling
Since i'm not a technical person, so I don't have a knowledge of all this. But this training program is real important for the organization since the company for which i'm doing the project is in telecom sector.
Waiting for your suggestions.
Thanks a lot.
Reena
From India, Mumbai
Hi Reena
Thanks for sharing that.
If I understand what you are saying, the SDH programme is about achieving one of your employer's strategic objectives - building an SDH capability.
As such there may not be an immediate KPI linked to this such as profitability. This is quite normal for a project of this type.
It also makes the ROI issue an interesting one! The financial benefits of SDH may not become apparent for some time to come. What we can do though is to ask senior managers for their considered views, informed estimates, forecasts etc - call it what you will, the important thing is it will be THEIR numbers, and use this to predict the likely ROI. We won't claim all of the benefits of SDH, but through a root cause analysis we can determine the probable contribution of the training and estimate the ROI on that basis.
In situations like this where time is short, and/or data is short, estimates are entirely reasonable data to use, so long as assumptions underlying the estimates are made clear to all.
I'll be back in a few hours with my specific suggestions.
Regards
Martin
From United Kingdom,
Thanks for sharing that.
If I understand what you are saying, the SDH programme is about achieving one of your employer's strategic objectives - building an SDH capability.
As such there may not be an immediate KPI linked to this such as profitability. This is quite normal for a project of this type.
It also makes the ROI issue an interesting one! The financial benefits of SDH may not become apparent for some time to come. What we can do though is to ask senior managers for their considered views, informed estimates, forecasts etc - call it what you will, the important thing is it will be THEIR numbers, and use this to predict the likely ROI. We won't claim all of the benefits of SDH, but through a root cause analysis we can determine the probable contribution of the training and estimate the ROI on that basis.
In situations like this where time is short, and/or data is short, estimates are entirely reasonable data to use, so long as assumptions underlying the estimates are made clear to all.
I'll be back in a few hours with my specific suggestions.
Regards
Martin
From United Kingdom,
Hello Mr.Martin,
This is not the first time when SDH Basics training is being imparted to employees in our organization.
As i've also told you previously, that the company for which i'm working is in telecom sector, Airtel to be more precise, so for transmission of data through communication lines, the best technology available is of SDH.
So, this training is at the heart of our organization since new employees(engineers to be more precise) need to be taught the details of this technology.
In Airtel, till now, training evaluation was used to be done through feedback forms only i.e. at the reaction level. But now we are planning to go for proper ROI calculations.
I hope I have cleared a few more details.
Thanks for your interest.
Waiting for your valued suggestions.
Reena
From India, Mumbai
This is not the first time when SDH Basics training is being imparted to employees in our organization.
As i've also told you previously, that the company for which i'm working is in telecom sector, Airtel to be more precise, so for transmission of data through communication lines, the best technology available is of SDH.
So, this training is at the heart of our organization since new employees(engineers to be more precise) need to be taught the details of this technology.
In Airtel, till now, training evaluation was used to be done through feedback forms only i.e. at the reaction level. But now we are planning to go for proper ROI calculations.
I hope I have cleared a few more details.
Thanks for your interest.
Waiting for your valued suggestions.
Reena
From India, Mumbai
Thanks for that update.
An immediate suggestion I'd like to offer is this: try to find out the expectation of senior management with respect to evaluating at ROI level - what do they want to do with this information? Is it to determine if the training is being done as cheaply as possible yet still remain effective? Is it to decide if the training should be cancelled? ROI cannot always help in these respects. It may be unkind to suggest that senior management have recently discovered the concept of training ROI seeing as it seems to be getting lots of 'air time' at the moment, and that they have 'jumped on the band wagon' without appreciating the limitations of this approach, but I can't totally dismiss this notion from my mind - I've certainly seen it in many other organisations!
It seems that SDH is core to the engineering operations of this telco. As such this SDH training that we are discussing is like safety and other regulatory training - it has to be done.
The benefits of doing this training therefore need to be viewed in terms of what it prevents, i.e. it provides engineers with the skills and knowledge needed to prevent them making mistakes through inappropriate actions and decision making. Assuming the training is effective, any mistakes they do make after the training will more likely be due to other factors, such as problems with equipment, the working culture and practices in the workplace, standards of supervision, and perhaps also issues with customer and team relations - all of these will be a factor, some larger than others.
The maximum benefit this training will deliver is therefore expressable in terms of preventing the worst case scenario - so this scenario needs to be identified and costed. This is reality is likely to require the involvement of some experienced engineers for example, aided by a reasonably detailed process map of all the activities these engineers do, and for each of these activities, all the possible errors they can make, and for each error, the liklihood or probability of that error happening, and the costs/consequences of each. In the interests of time this need not be exhaustive, but should cover perhaps 80% of the factors.
One tool to help manage this is to use the relationship between incident rates of major errors to minor errors, based on work in the safety industry. Typically for every major error, or say a fatality, there are between 10 and 30 (nearer 30) lesser errors or lesser injuries, and for each of these lesser incidents there are between 10 and 30 (nearer 30) minor errors or incidents.
It is also the case that newly trained operators are less likely to make mistakes in operations IF the training they have had is effective and if the workplace culture is reasonable. So the real ROI can perhaps only be actually calculated for a particular course and group of engineers after one year, where you identify all the possible opportunities they had to make mistakes, from this calculate the maximum possible benefit that COULD be gained by not having any errors/mistakes, and then subtracting the costs of the mistakes that actually happened. Assuming you also have the delivery and amortised development costs of the training, an estimate of the ROI can be made. BUT, you still need to account for the effects of other factors such as poor supervision - which implies you probably need some form of error/incident investigation system to enable each error to be investigated for root cause etc - this is all beginning to get very complex and you only have a few weeks!!!
I'll ponder this a little more and get back to you today!
From United Kingdom,
An immediate suggestion I'd like to offer is this: try to find out the expectation of senior management with respect to evaluating at ROI level - what do they want to do with this information? Is it to determine if the training is being done as cheaply as possible yet still remain effective? Is it to decide if the training should be cancelled? ROI cannot always help in these respects. It may be unkind to suggest that senior management have recently discovered the concept of training ROI seeing as it seems to be getting lots of 'air time' at the moment, and that they have 'jumped on the band wagon' without appreciating the limitations of this approach, but I can't totally dismiss this notion from my mind - I've certainly seen it in many other organisations!
It seems that SDH is core to the engineering operations of this telco. As such this SDH training that we are discussing is like safety and other regulatory training - it has to be done.
The benefits of doing this training therefore need to be viewed in terms of what it prevents, i.e. it provides engineers with the skills and knowledge needed to prevent them making mistakes through inappropriate actions and decision making. Assuming the training is effective, any mistakes they do make after the training will more likely be due to other factors, such as problems with equipment, the working culture and practices in the workplace, standards of supervision, and perhaps also issues with customer and team relations - all of these will be a factor, some larger than others.
The maximum benefit this training will deliver is therefore expressable in terms of preventing the worst case scenario - so this scenario needs to be identified and costed. This is reality is likely to require the involvement of some experienced engineers for example, aided by a reasonably detailed process map of all the activities these engineers do, and for each of these activities, all the possible errors they can make, and for each error, the liklihood or probability of that error happening, and the costs/consequences of each. In the interests of time this need not be exhaustive, but should cover perhaps 80% of the factors.
One tool to help manage this is to use the relationship between incident rates of major errors to minor errors, based on work in the safety industry. Typically for every major error, or say a fatality, there are between 10 and 30 (nearer 30) lesser errors or lesser injuries, and for each of these lesser incidents there are between 10 and 30 (nearer 30) minor errors or incidents.
It is also the case that newly trained operators are less likely to make mistakes in operations IF the training they have had is effective and if the workplace culture is reasonable. So the real ROI can perhaps only be actually calculated for a particular course and group of engineers after one year, where you identify all the possible opportunities they had to make mistakes, from this calculate the maximum possible benefit that COULD be gained by not having any errors/mistakes, and then subtracting the costs of the mistakes that actually happened. Assuming you also have the delivery and amortised development costs of the training, an estimate of the ROI can be made. BUT, you still need to account for the effects of other factors such as poor supervision - which implies you probably need some form of error/incident investigation system to enable each error to be investigated for root cause etc - this is all beginning to get very complex and you only have a few weeks!!!
I'll ponder this a little more and get back to you today!
From United Kingdom,
Sorry, Mr. Martin for such a late response...
Your articles were really informative.
Since, my training is about to end in a week, so i've completed some portion of the project n planned for the rest. Here is how I'm doing it...
Level1 - Reaction - With the help of feedback forms. Participants have already filled these forms. I'm in the process of analyzing them. Our feedback form consisted of a combination of both objective and subjective questions.
Level2 - Learning - With the help of quiz. Already taken by the trainer on the last day of the training and I have received the respective scores of the participants. I still have to analyze them.
Level3 - Behaviour - We identified 4 measures which would be impacted upon by the training program. Participants have already rated themselves on a 10-point scale with respect to these measures on the 1st day of the training. Similar response from their superiors is awaited.
Participants and their superiors were again be asked to rate themselves 3 months after training. Then the difference will be analyzed.
There are some limitations of this which we have identified. These are:-
1. Level 4 is not possible to measure since the training effects are not properly seperable. Moreover since we are calculating ROI for the first time, so I think its better to move in small steps.
2. ROI would be calculated in quantitative terms which would then be converted into percentage form. It won't be calculated in monetary terms.
3. This is my own limitation. Since I've to submit project by 15th of August, so I won't be able to get post-data of Level 3. Although I would be showing the process in my project report but there won't be any post data.
Waiting for your valued feedback....
Regards,
Reena
From India, Mumbai
Your articles were really informative.
Since, my training is about to end in a week, so i've completed some portion of the project n planned for the rest. Here is how I'm doing it...
Level1 - Reaction - With the help of feedback forms. Participants have already filled these forms. I'm in the process of analyzing them. Our feedback form consisted of a combination of both objective and subjective questions.
Level2 - Learning - With the help of quiz. Already taken by the trainer on the last day of the training and I have received the respective scores of the participants. I still have to analyze them.
Level3 - Behaviour - We identified 4 measures which would be impacted upon by the training program. Participants have already rated themselves on a 10-point scale with respect to these measures on the 1st day of the training. Similar response from their superiors is awaited.
Participants and their superiors were again be asked to rate themselves 3 months after training. Then the difference will be analyzed.
There are some limitations of this which we have identified. These are:-
1. Level 4 is not possible to measure since the training effects are not properly seperable. Moreover since we are calculating ROI for the first time, so I think its better to move in small steps.
2. ROI would be calculated in quantitative terms which would then be converted into percentage form. It won't be calculated in monetary terms.
3. This is my own limitation. Since I've to submit project by 15th of August, so I won't be able to get post-data of Level 3. Although I would be showing the process in my project report but there won't be any post data.
Waiting for your valued feedback....
Regards,
Reena
From India, Mumbai
Community Support and Knowledge-base on business, career and organisational prospects and issues - Register and Log In to CiteHR and post your query, download formats and be part of a fostered community of professionals.
Innovative Model to Calculate Return on Training Investment.pdf
Annual Return form 21 under F. act.doc