Category Archives: Derisking

Project Risk Management and High Uncertainty

Leave a reply

Risk management for novel development, engineering development, high complexity, new product development, new ventures, and large capital projects demands new and more robust methods to effectively deal with high uncertainty. Prior posts explored varieties of uncertainty and design strategies, facing innovation uncertainty, and technical risk assessments in new product development.

Since most development and innovation work are managed as projects or as a subproject within a large project new methods to manage high uncertainty that integrate within existing project management and project risk management approaches would be useful. This post reviews a book Managing The Unknown: A New Approach to Managing Uncertainty and Risk in Projects written by Christoph Loch, Arnoud DeMeyer, and Michael Pich in 2006.

Project Risk Management Framework

The authors present an integrated project risk management framework that assists project management to select project risk management methods based on the sources of uncertainty and complexity as shown here.

Project Uncertainty Framework

This framework is useful because it integrates existing and well understood project risk management methods with new methods for managing uncertainty like learning and selectionism. The source of uncertainty axis maps well into the uncertainty continuum concept of a previous post. The authors use an uncertainty continuum breaking uncertainty into foreseeable and unforeseeable uncertainty.  Foreseeable uncertainty includes variation and foreseeable events while unforeseeable uncertainty includes unknown unknowns. These distinctions are enough to provide a decision tool to select appropriate risk management methods.

Measuring Project Complexity

The project risk management framework requires project management to determine project complexity. Another useful approach described in the book is a method to measure the project complexity. The authors base their approach on the number of interactions between major parts of the project in three domains: system domain; project task domain; and organizational domain. The authors apply the Design Structure Matrix (DSM) tool to model and visualize how major project pieces interact. Methods are proposed to simplify the measurement process to give quick / high level results for large projects. Complexity is then computed based on the sum of all project elements multiplied by the sum of all interactions. No comparative complexity measure results are presented for project types, industries, etc leaving each firm to develop their own benchmarks from amongst their own project portfolio.

Uncovering Hidden Unknown Unknowns

The authors propose a method to uncover hidden unknown unknowns at the beginning of a project with the goal of enabling a systematic process for project managers. The process involves asking: “What do I know and what do I not know? and where are the major knowledge gaps?” where the knowledge gaps identify where unknown unknowns may emerge. The process steps summarized here are:

  1. Identify the problem structure.
  2. Break the overall problem into pieces.
  3. For each piece, perform risk identification, identify knowledge gaps by probing assumptions in an iterative way.
  4. Estimate the complexity of each project piece and the overall project.
  5. Manage pieces in parallel pieces according to the project risk management framework above.

To visualize the uncertainty and complexity of each major piece the authors propose a pie chart for each piece with a bar for: variation, foreseeable uncertainty, unforeseeable uncertainty, and complexity. These pie charts can be used to decide where additional attention is required according to the project pieces and can be used to capture trends over the project life.

Project Risk Management Methods

The book also describes a broad set of project risk management methods with an excellent set of examples and case studies from many industries. The project risk management methods are summarized here.

Project Uncertainty Management Methods

Beyond the well-known project risk management methods for managing variation, foreseeable events, and residual risk the book goes into great detail describing learning and selectionism as two main methods to manage unforeseeable uncertainty or unknown unknowns.  The authors propose the following definitions:

  1. Learning – “Learning in projects is the flexible adjustment of the project approach to the changing environment as it occurs; these adjustments are based on new information obtained during the project and on developing new – that is, not previously planned – solutions during the course of the project.”
  2. Selection – “In the face of uncertainty, one launches several solution attempts or sub-projects, each with a different solution strategy to the problem in hand. If the solution strategies are sufficiently different, one would hope that one of them will succeed and lead to a successful outcome. Success depends on generating enough variation so that ex post, we obtain desirable results.”

In the case of a learning approach the authors explore improvisational learning and experimentation.  For selection the authors explore what makes selectionism work and offer set based design as an example of how to implement it in practice. Guidance is provided on how to implement these methods and good case examples are provided.

Overall, this book provides a useful project risk management framework and decision tool that can be applied in practice for a more robust project management and risk management system when high uncertainty is present. Effort would need to be planned for and budgeted for in the project which suggests that the approaches would need to be applied on several projects to collect historical basis of estimates. The authors provide a number of methods to address unforeseeable uncertainty but additional methods could be added to this decision tool such as real options, big data, adaption, and creaction reported in prior posts. The book provides a practical method to measure project complexity which is often difficult to measure and benchmark. Anyone assigned accountability for projects with novelty and complexity should read this book.

Innovation Investment Decision Risk Aversion

1 Reply

If overcoming uncertainty and risk is the largest barrier to innovation regardless of the type of firm it is important to understand how human biases and tendencies may be influencing innovation investment decision making. Extreme leadership risk aversion can cause firms to miss good opportunities and harm long run performance and growth prospects. Extreme leadership risk taking on the other hand can bet the farm jeopardizing the entire business. Healthy leadership risk management requires a balance suitable for a firm’s external, internal environment, and business cycle. Canadian business leaders appear to be tending towards extreme risk aversion.

What are some people biases and tendencies that may be driving extreme leadership risk aversion when making innovation investment decisions?

Innovation Investment Risk Sources and Assessment

To understand the people biases and tendencies impacting innovation investment decision risks assessments we first need to quickly review innovation risks assessment.

The return on innovation investments targeting new or improved ways of doing business or products are measured in terms of new value creation or improved value capture. In pure financial terms returns would need to exceed the corporate hurdle rate within the business risk tolerance. Risk assessments for innovation investment therefore consider what could impact achieving the return on innovation investment. Although risk sources tend to be viewed as negative equal balance should be given to upside as well as down side risks.

In general business risk sources that may impact innovation investments can cover operations, market, financial, or political risks including:

  1. Market Risks – Risks impacting revenue forecast and demand for the innovation particularly assumptions related to customer’s need and willingness to pay for the innovation.
  2. Adoption Risks – Risks impacting the adoption of the innovation in the innovation ecosystem up to and including the end user.
  3. Technical Risks – Risks impacting the technical achievement of the innovation performance such as product performance, feasibility (readiness for commercialization), regulatory approval/compliance,
  4. Operational or Execution Risks – Risks impacting the operational delivery of the innovation including supply chain disruptions, procedural failure, cash flow for required work in process.
  5. Co-Innovation Risks – Risks impacting key supplier/partner innovation required to achieve  innovation or deliver the innovation to market.
  6. Schedule Risks – Risks impacting lateness of delivering innovation to capture the value.
  7. Cost Risks – Risks impacting the innovation investment costs that impact profitability.
  8. Quality Risks – Risks impacting the innovation investment customer satisfaction, conformance to specifications, reliability, durability, serviceability, and aesthetics factors.
  9. Financial Risks – Pricing, asset, currency, or liquidity risks impacting the innovation return.
  10. Reputation Risks – Risks impacting business reputation such as product failure, integrity, social systems,
  11. Legal Risks – Risks arising from the innovation that may drive liability torts, property damage, IP legal actions.
  12. Strategic Risks – Risks arising from the innovation that may create new threats from new market entries, strained partnerships, regulatory changes, random surprise events impacting innovation assumptions,
  13. Political Risks – Risks impacting market for the innovation arsing from government protectionism, freedom of trade and tariffs, labour markets, local capital markets, corruption, and openness in different countries.
  14. Business Model Risks – Risks that assumptions unpinning the innovation are valid.

The list of potential risks impacting innovation investments is long giving decision makers reason to pause.  Management competence & experience are required to judge the severity and probability of each risk. The context for innovation investment decision making is therefore complex, ambiguous, and subject to uncertainty.

Risk Aversion In Decision Making

How does the presence of risk affect decision making? According to Kahneman and Tversky most people are risk seeking when it comes to losses but risk averse when it comes to gains.  Instead of weighing decision alternatives impact on total value decision makers frame outcomes as either a gain or loss based on an arbitrary reference point.

For example framed as a gain if faced with two options:

Option A: Receiving $100,000 cash.

Option B: Playing a game that offers a 50% chance of winning $200,000 cash and a 50% chance of not winning anything.

Most decision makers would select the sure thing Option A (ie. risk aversion for a gain) even though the expected value of each is $100,000 (Option B expected value = $200,000 x 0.5 = $100,000). But if the decision was framed as a loss:

Option C: Paying $100,000 for an unexpected cost or expense.

Option D: Playing a game that offers a 50% chance of paying nothing and a 50% chance of paying $200,000.

If forced to decide individuals would choose Option D (ie. risk seeking for a loss) again even though the both have an expected value of $100,000.

This helps to explain why innovation decision makers will take risks on continuous improvement which they frame as loss but not on R&D projects or export trade options which they frame as a gain. Continuous improvement is seen as Option D whereas R&D projects/export trade options are seen as Option B. Canadian business leaders decisions tend towards Option A and D and avoid Options B and C. The ability to make decisions based on probability tends to be avoided unless decision makers have confidence in the business case numbers.

Individual Biases and Tendencies Influencing Innovation Investments

Participants in the innovation risk assessment process could include: idea generator, designer, engineer, project manager, support staff, marketer, sales, executive sponsor, intermediaries, partners, financiers, and finally the investment decision maker. What are some of the individual biases that can influence any of these participants and possibly cause them to be extremely risk averse:

Overconfidence Bias – Individuals tend to have unwarranted levels of confidence in their judgment, occurrence of positive events, and accuracy of forecasts and then the underestimates of the likelihood of negative events. This bias can cause innovators to identify hidden flaws in their assumptions, approaches, or estimates of cost or schedule. High prevalence of this bias may cause decision makers to discount claims from employees appearing as extreme risk averseness.

Status Quo Bias – The tendency for individuals to prefer to leave things as they are driven by an aversion to loss. (Samuelson & Zeckhauser) This bias can cause decision makers to prefer to continue to use the current process, existing product line or business model even though the competition or market is changing. This bias certainly drives extreme risk averse behaviour.

Availability heuristic – The more prevalent a category is judged to be the easier it is for individuals to bring instances of this category to mind. Recent public failures or events may be cited as possible reason not to proceed even though the connection to proposed innovation investment is not relevant. This tendency could undermine the credibility of innovators in the eyes of decision makers.

Base Rate Fallacy – People who consult their neighbors and friends will discount perfectly valid information and choose instead to rely on a vivid example. Flawed vivid examples could steer decision makers and innovators away from the best course of action or alternative. This tendency could undermine the credibility of innovators in the eyes of decision makers.

Herding Instinct – The tendency for individuals to follow the behaviour and opinions of others.(Belsky & Gilovich) Individuals may propose innovation initiatives that are similar to what competitors are doing rather than focusing on what the customer actually needs and wants. This tendency could cause decision makers to delay decisions if they see their competitors not following the herd which is not necessarily bad but if they only approve investments that follow the herd and the herd is not moving it could be interpreted as extreme risk averse behaviour.

Gambler’s Fallacy – The tendency to treat chance events as though they have a built-in evening-out mechanism even though each event is independently determined. Innovators may continue to experiment hoping for a desired end result rather than learning from the results that may point in a very different direction. This tendency could undermine the credibility of innovators in the eyes of decision makers.

Perseverance Effect – The tendency for people to continue to believe that something is true even though they are offered strong counter evidence that disproves or proves it to be false. (Ross & Lepper) This tendency could also blind innovators from learning from results and decision makers to avoid making investment decisions.

Illusory Correlation – Tendency to see invalid correlations between events.(Hamilton & Gifford) This tendency undermines the soundness of underlying rationale or logic of decisions. This tendency could undermine the credibility of innovators in the eyes of decision makers.

Anchor Bias – Tendency to make estimates on readily available evidence that is meaningless. This tendency could also lead innovators down the wrong path or blind them to possibilities. This tendency could undermine the credibility of innovators in the eyes of decision makers. This tendency could also constrain decision making.

Confirmation Bias – Tendency to use favourable information that supports a position and suppress information that contradicts the position. The underlying assumptions and data must be challenged and verified. This tendency could undermine the credibility of innovators in the eyes of decision makers.

Hindsight Bias – Tendency for individuals to infer a process once the outcome is known but unable to predict outcomes in advance…”I knew it all along”.(Fischhoff) The inability to connect the dots to create new value also constrains taking chances.

Functional Fixedness – Tendency to base a problem solution on familiar methods but hinders the development of strategies for new situations.(Adamson & Taylor) This tendency could lead decision makers to constrain their view of available courses of action.

Selective Recall – The tendency to recall only facts and experiences that support assumptions underpinning a position. This tendency could undermine the credibility of innovators in the eyes of decision makers.

Set Effect – Prior experience can have a negative effect on solving new problems by limiting an individuals view in breadth and generality. In the absence of experience and weak competition firms may avoid innovation investments based on limited prior experience.

Biased Interpretation – Individuals only hear what they want to hear particularly in the presence of ambiguity. This applies to misinformed, uninformed, or narrow thinking in strategy that constrains consideration of innovation before it has a chance to demonstrate its potential.

Curse of Knowledge – Individuals who are privy to information and knowledge that they know others are not continue to act as if the others have the information. Innovators seeking to persuade business leaders need to be aware that decision makers do not have the level of technical understanding.

Escalate Commitment – Tendency direct more resources to a failed course of action. This tendency is particularly prevalent in failed innovation projects where good money is thrown after bad based on continued results that contradict assumptions.

Mental Accounting – The tendency to treat money differently depending its source, where kept, and how the money is spent. Some money is spent freely while other money is highly scrutinized. (Thayer) This is an interesting tendency in decision makers particularly if they apply a very stringent standard on innovation investments but not other expenditures with lower potential return or expenditures susceptible to the status quo bias.

Organizational Biases Influencing Innovation Investments

Complex innovation investments often require teams or groups such as: innovation initiative team, product development team, production team, sales team, leadership team, and board. Organizational biases that can influence these teams or groups during innovation and to be extremely risk averse include:

False Consensus Effect – Most individuals think others agree with them more than the group actually does. This effect could give the impression that decision makers are risk averse but actually have better or different information from the team.

Groupthink – The tendency for group members not on side to fall in line and suppress their objections.(Janis) This tendency could undermine the credibility of team in the eyes of decision makers. In the case of a leadership team if members don’t state their views on potential opportunities then good opportunities may be missed.

Tunnel Vision – The tendency for a group to underestimate the number of feasible options available. Tunnel vision could lead to a course of inaction and risk aversion.

Uneven Participation – The tendency for a small number of strong willed individuals to do all the talking. Similar to groupthink.

Naïve Realism Principle – The tendency for people to expect others to hold views of the world similar to their own. This tendency can blind decision makers to good opportunities.

Dominant Bias On Innovation Investment Risk Aversion

Dominant bias and tendencies behind extreme risk aversion in Canada or in general may be the status quo bias, mental accounting, herding instinct, set effect, and biased interpretation. Certainly a predisposition to not take innovation ideas serious because of perceived credibility issues driven by some of the biases may also to blame.

Methods to address individual and organizational biases and tendencies when making innovation investment decisions will be the subject of future posts.

Facing and Overcoming Innovation Uncertainty and Risk

2 Replies

Business growth requires leaders to identify opportunities, evaluate their potential and risks, decide amongst the most promising, and executing for results. Growth strategy options include: organic growth, growth through acquisition, or growth through alliances.  Growth can be achieved through product / market choice (concentrated, vertical/horizontal, diversification), white space, or incremental/substantial/breakthrough innovation.

Uncertainty and risk associated with innovation opportunities are often cited as barriers to growth. The inability to overcome uncertainty and risk when innovating was cited as the leading reason for slow Canadian SME growth as confirmed by the 2009 SIBS study and recently by Deloitte.

The 2009 SIBS study reported uncertainty and risk as the largest obstacle (47% of firms) to innovation regardless of the type of firm. Steps taken to overcome uncertainty and risk as an obstacle were also reported to be one of the least effective (38% of firms reporting uncertainty as an obstacle).

The Deloitte study reported that “Canadian business leaders were substantially more risk averse than U.S. leaders, and more reliant on government assistance to pursue new projects” and that Canadian firms “seem unable to deal with these factors successfully” moreover “as Canadian firms mature, they become less likely to engage in the kind of activities that contribute to rapid growth”. The Deloitte study suggests that to deal with risk “firms have the power to mitigate these obstacles by hedging and compensation tactics”. The study also reported that low R&D spending, poor export intensity, lack of market diversification, low access to market diversification, and attitudinal preferences were also major inhibitors to growth.

Innovation though is a non-linear process where the value of success can be much higher than the cost of failure. Global markets are increasingly uncertain.  What approaches beyond hedging and compensation tactics can be used to deal with uncertainty to improve Canadian business leaders confidence?

Adaptability

An interesting view of how to grow in the face of uncertainty comes from Max McKeown who wrote Adaptability: The Art of Winning in an Age of Uncertainty. In this book the author identifies a number of rules for winning in the face of uncertainty.  Chief among these applicable to Canada is that stability is a dangerous illusion. He defines failure as the failure to adapt and success as successful adaptation to cope or win – defining degrees of adaptation outcomes being collapse, survival, thriving, and transcendence.

The author observes that there are three steps to adaptability:

  1. Recognizing The Need To Adapt – The ability to feel or know something is wrong, timeframe of change depends on the situation and can be long or extremely fast, and some find out too late by missing signals or are simply complacent.
  2. Understand The Adaptation Required – Observing that there is often no agreement on adaption requires, culture/rules/tradition can be barriers, learn what works from failure, imagination is needed to see alternatives.
  3. Do What Is Necessary To Adapt – Sometimes adaption must be provoked, strong action to overcome barriers, need to focus on changing the nature of the game, and build influence to make changes.

The rules for winning in the face of uncertainty organized by the three stages as suggested by the author are:

Recognize The Need to Adapt

  • Play your own game – If losing find a way to change the game – no one way to win
  • All failure is a failure to adapt – didn’t recognize it, didn’t understand what adaption required, did not do what was necessary to adapt.
  • Embrace unacceptable wisdom – speaking opposite to the prevailing wisdom creates opportunities.
  • Know when to break the rules – rules contain knowledge & experience, rules also contain prejudice or mistaken beliefs, rules may no longer be applicable.
  • Stability is a dangerous illusion.
  • Stupid survives until smart succeeds – ‘we were wrong’, biases to remain on course of action.

Understand Necessary Adaption

  • Learning fast is better than failing fast.
  • Plan B matters most – adaptability doesn’t kick in automatically.
  • Free radicals – radicals influence the group – stir the pot – counter complacency.
  • Think better together – collective support important.
  • Get a strong partner – diverse skills and talents increases adaption effectiveness.

Adapt as Necessary

  • Never Grow up – organizations get old, grow up, and lose edginess – remain curiosity driven.
  • Hierarchy is fossil fuel – Locks people in boxes, resists learning, and institutionalizes self-interested behaviours.
  • Keep the Ball – reduce the game down to its fundamental components that captures the most important features of the system to improve – compete outside the game.
  • Swerve and swarm – Combining swerving, avoiding dominance of the obvious idea, and swarming, to bring mass participation to finding non-obvious answers is powerful.
  • Get Ambition on – the future gives direction and unlimited energy to change – ambition is a way of seeing the future – ambition gets us started.
  • Always the beginning – Advantage from adapting first, winners acquire resources and knowledge.

Creaction Method

The creaction (short for creative action) method to move forward in the face of uncertainty was proposed by Leonard Schlesinger, Charles Kiefer, and Paul Brown in Just Start: Take Action, Embrace Uncertainty, and Create The Future. These authors observed that most business leaders have worked in a world where the world was predictable and that the future could be forecasted, plans made, resources gathered, and then execute the plan to make it happen.  The core assumptions being that the future will behave like the past so plans can extrapolate current reality moving forward. The authors suggest that the world that is changing fast will become increasingly unpredictable so business leaders need a new way of thinking to drive business growth.

Borrowing from entrepreneurial behaviours, the authors propose the creaction method to succeed when markets are unpredictable. The creaction method is:

  • Act (with a modest goal as a guide);
  • Learn (from the action); and
  • Build (off learning) and then act again.

Creaction starts with a desire to achieve a goal with a purpose no necessarily a passion. The authors suggest acting quickly with the resources have at hand and never more than you can afford to lose if things don’t work out defined as an acceptable loss. A small bet rather than betting the firm. When considering acceptable loss the authors suggest assets at risk are: money, time, professional reputation, personal reputation, and missed opportunities and bounding the investment so that if the option fails it fails cheaply. Enlist the support of other like minded people who share interest in the goal and purpose. When learning from the results of the small step the authors note that “creation is all about exploiting the contingencies and leveraging the uncertainty by treating unexpected events as an opportunity….treating surprises as a gift….running headlong into a problem and then solving it can give you a barrier to the competition”. The book provides useful implementation advice.

Implications For Canada

Max McKeown’s observations on adaptability are particularly poignant for Canada as many SMEs that have stopped growing may have not recognized the need to adapt given the predominance of the resource industries in the overall economy. The implication being that stability is not only a dangerous illusion but that the voices that oppose the prevailing wisdom are potentially being ignored. SME business leaders should stress test their business assumptions and consider potential avenues for change and adaptation. The creaction method provides a means for SME business leaders to take small steps, learn, and adapt to develop growth strategy. Future posts will explore other approaches to overcome risk and uncertainty.

Effective Technical Risk Assessments In New Product Development

Leave a reply

Technical risks are a common cause of new product development project cost and schedule overruns. Effective technical risk assessment is therefore on the mind of investment decision makers and all new product development project managers.

Effective Technical Risk Assessment

To be effective, technical risk assessment must be performed up front so that investment decision makers can clearly weight the risk/reward of continuing while enabling product developers to plan the realistic scope of effort along with tackling the biggest technical mitigation efforts immediately.

A technical risk assessment is typically based on lessons learned from previous new product development projects, experience of team members or advisors, and historical data from similar products developed either by the company or competitors. A project pre-mortem  is an effective tactic to identify ‘project killer’ issues up-front to ensure project team members and stakeholders are not feeling pressured to express their views.

Technical risk assessments should cover the feasibility of the product including building blocks (particularly mix of existing/new), how the product building blocks integrate amongst themselves, how the product integrates with the operating environment, ability to manufacture the product cost effectively, and feasibility of new manufacturing technologies.

A central theme in any technical risk assessment therefore is that new product development outcomes depend on the maturity of the underlying hardware, software, and integrated system. A key question facing new product developers then is: How can technology maturity be measured to determine the level of technical risk? 

Technology Maturity Assessment

Good technical risk assessments depend on an effective technology maturity assessment.  The percentage of unproven technology and level of integration in the new product determines the degree of technical risk that can impact project cost, schedule, and quality.

The technology readiness level scale provides a means to assess the maturity of the subsystem building blocks in a new system. First developed by NASA as illustrated below the Technology Readiness Level scale uses nine levels as described.

technology_TRLS

The NASA TRL scale illustrated is for products used in space but the scale can be easily adapted for any operating environments for which the new product is intended.  The assessment scale is also used by the US DoD and has been adopted by the Canadian government for use in the Canadian Innovation Commercialization Program.

Product developers should deconstruct their product building blocks and assess the technology maturity level of the individual building blocks using the technology readiness levels.  Product developers need to assess whether technology proposed is ready to be used in their project. Typically subsystems should not be used unless demonstrated at TRL 7 or higher. Subsystems with TRL below 7 are really research projects presenting too much uncertainty for accurate customer schedule commitments. R&D projects can use the TRL scale to measure progress towards commercialization. Product portfolio strategies and roadmaps need to plan how new technology subsystem R&D projects will be coordinated, prioritized, and sequenced with sufficient time to reach TRL 7.

Integrated System Maturity Assessments

Product developers should also assess the technology maturity level of the integrated system. Integration problems can be as serious as individual subsystem immaturity issues if not more when the scope of the new product/project is large and often go unrecognized until well into new product development project.

The level of complexity of the integrated system drives the scope of the system level maturity assessment. The TRL scale is not very effective as system complexity increases. It is also important to recognize that complex integration can be present in both new systems (new systems with existing and new subsystems) where we would expect them but also legacy system upgrading (old systems being upgraded with new subsystems).  Limitations with the technology readiness level scale reported for major/complex projects include:

  • Emphasis on subsystems;
  • Nonlinearity of the scale particularly the large leap from TRL 6 to 7;
  • Not accounting for system integration and manufacturing; and
  • Does not indicate the degree of risk of moving up the scale.

The Risk Identification, Integration, and Ilities (RI3) approach has been proposed to  augment the technology readiness level system for manufacturing readiness and systems engineering ‘ilities’ . Another approach to assess the system level readiness is the UK MoD System Readiness Level method. The advancement degree of difficulty (AD2) method has been proposed to address the degree of risk of moving up the scale.  The cost effectiveness of applying these approaches depend on the complexity of the integrated system and size of the new product development project.

Product developers need to apply a disciplined technology maturity assessment early in any new product development project to proactively mitigate technical risks. Investment decision makers should consider impartial technical feasibility assessments based on subsystem and system technology readiness levels described in this post to remove bias from technology maturity assessments.