Background

It has now taken two decades or more for the design of short-term wholesale markets to evolve to where they are today. The organizational and short-term wholesale market design challenges I addressed in my 1996 (Joskow, Reference Joskow1996) paper had largely been resolved reasonably successfully prior to the decarbonization transition, although market designs have evolved considerably over time even before decarbonization policies began to bind. Decarbonization commitments create new challenges for the design or short-term wholesale markets and the attributes of the generation ‘products’ supplied there. I have discussed these challenges in previous work (Joskow, Reference Joskow2019) and will not pursue them here.

The mechanisms for guiding investment in new generating capacity and retirements of existing generating capacity consistent with reliability criteria have been the subject of more controversy and more diversity of approaches than is the case for short-term market design even before decarbonization commitments came to dominate the sector. In an ideal Arrow–Debreu world, short-term markets should provide the price signals to guide ‘technology neutral’ investment in new and retirement of old generating capacity supported by efficient voluntary contracts to allocate and hedge market risks. Anyway, this was the assumption that was often made by short-term market designers which focused perhaps too much attention on short-term market design and too little attention on investment and retirement incentives. The historical and ongoing issues with investment incentives are now compounded by the policies requiring a rapid transition to decarbonized electricity sectors supported by very significant investments in wind and solar generation, investments in storage, and the retirement of most fossil-generating capacity.

‘Missing money’ and supplemental capacity remuneration mechanisms

It is well known (Joskow, Reference Joskow2008; Joskow and Tirole, Reference Joskow and Tirole2007; Léautier, Reference Léautier2019) that theoretically for a short-term electricity market to balance supply and demand in a way that is compatible with economics-based reliability criteria, when supplies are very tight, markets should be cleared ‘on the demand side’ by reductions in demand reflecting the diverse consumers' values of having their demand suddenly curtailed voluntarily or involuntarily. It is theoretically convenient to represent the demand side with assumptions about the value of lost load (VoLL).Footnote ⁵ This is referred to as ‘scarcity pricing’ where prices rise to reflect the value consumers place on avoiding an involuntary blackout by reducing demand.

However, short-term wholesale market designs rarely replicate these idealized conditions linking short-term market price incentives with efficient generation investment and retirement behavior. There is one or perhaps more missing markets for reliability, a limited ability for individual customers to express their individual values for avoiding supply curtailments, and barriers to adjusting retail prices faced by consumers to reflect variations in wholesale market prices and incenting voluntary demand adjustments (Joskow and Tirole, Reference Joskow and Tirole2007). To reflect the VoLL, short-term energy prices would have to rise, rarely, to levels on the order of 200 times higher than the average short-term energy price – e.g. $10,000/MWh or higher (Sullivan et al., Reference Sullivan, Schellenberg and Blundell2015).Footnote ⁶ However, short-term wholesale market designs do not have a real demand curve for reliability and initially did not include a scarcity pricing mechanism that allowed prices to rise to the VoLL. In the USA, short-term wholesale prices were initially capped at about $1,000/MWh or less.Footnote ⁷ The rationales for price caps include mitigation of market power, the absence of a demand-side market where consumers can express their reliability preferences, and the technological inability of system operators to ration demand by individual consumers (Joskow and Tirole, Reference Joskow and Tirole2007), leading to a vertical short-term demand function where prices could be unbounded when capacity is fully utilized.

In my view, despite the diversity in short-term wholesale market designs, perhaps the primary departures of short-term wholesale markets from the theoretical ideal that have emerged in the last 20 years occur under scarcity conditions when prices are too constrained by administrative price caps and as a result of out-of-market actionsFootnote ⁸ by the system operator to avoid reliability problems. The failure of prices to rise high enough under scarcity conditions, along with out-of-market actions by system operators, suppress short-term market prices, and distort long-term generator investment and retirement decisions, leading to inadequate or an inefficient mix of generating capacity to meet administrative supply security criteria. This has come to be called the ‘missing money’ from the energy market (Joskow, Reference Joskow2008). It is the ‘missing money’ problem that has been the primary source of security of supply concerns in markets designed around conventional dispatchable generation.

Let my note, however, that even if price caps and out-of-market actions could be removed and price sensitive demand represented in the wholesale market, it is unlikely that the concerns about adequate generating capacity to meet security of supply criteria would disappear. This is the case because administrative security of supply criteria has never been designed around credible estimates of the VoLL in most electric power systems. Rather, the reliability criteria are defined exogenously by regulators and typically have implied VoLLs that are many times the estimates of VoLL that appear in the literature. These legacy reliability criteria have simply been carried over to liberalized markets from the old regime.Footnote ⁹ Policymakers (and system operators) are quite averse to involuntary blackouts or ‘load shedding’ as it is called in the business. As a result, they will act opportunistically – do what they have to do – regardless of what the market signals may be, in order to keep the lights on because they do not want to be held responsible for blacking out consumers. After the blackouts in ERCOT in February 2021, all of the members of ERCOT's board and its CEO were fired. Thus, it is not surprising that human behavioral factors play an important role in these markets.

In response to security of supply concerns, many, but not all, regulators and system operators have introduced some form of an additional capacity remuneration mechanism tied to more or less precise forward looking reliability indicators: historical administrative reserve margins, probability of lost load, quantity of lost load, duration of lost load, etc., at times of peak system demand (typically the summer in the USA and the winter in Europe). These mechanisms typically provide supplementary compensation to generators to ensure that there is adequate generation with the right operating attributes to be available to balance supply and demand without involuntary demand curtailments under extreme supply and demand conditions. Bublitz et al. (Reference Bublitz, Keles, Zimmermann, Fraunholz and Fichtner2019) provides an excellent overview of alternative capacity remuneration mechanisms adopted in Europe that have been added to the short-term wholesale market designs.

Background

The current short-term wholesale market designs have been built around the fact that most electric power systems have relied and still do rely primarily upon ‘dispatchable’ generating capacity – fossil-fueled, nuclear, and in some countries stored hydroelectric capacity, to meet the demand for electricity (Joskow, Reference Joskow2019: 299–300). In traditional vertically integrated systems, increases and decreases in supply from dispatchable generating facilities are determined centrally by the system operator as it balances supply and demand continuously based on short-run marginal cost estimates. In a competitive short-term wholesale market, the market-based economic dispatch curve that the system operator relies upon to balance supply and demand is determined by the prices and quantities that clear supply and demand in day-ahead, day-of, and real-time wholesale auction markets. Since the stock of generating plants changes very slowly, the market-based economic dispatch curve is fairly stable, varying primarily with fossil fuel prices and maintenance schedules over longer periods of time,Footnote ¹⁰ whereas demand varies continuously during the day, from day to day, week to week, month to month, etc.

To oversimplify, the system operator moves up and down the bid-based economic dispatch curve as demand varies across the day to balance supply and demand and to determine generator dispatch schedules and associated wholesale market prices for each hour day-ahead, day-of adjustments, and real-time. The ability of the system operator to control which generators are called upon to operate to balance supply and demand continuously allows the system to operate both economically and reliably in real time, to produce good short-term price signals, except in most wholesale markets under scarcity conditions, and where necessary to take out-of-market actions to respond to reliability problems. These attributes have allowed wholesale market designs to combine auction market economics with traditional power system engineering principles effectively to replicate what used to be done within vertically integrated hierarchies but instead with short-term wholesale markets with better cost control incentives. Williamson would have appreciated the integration of power system engineering principles with economics, but probably not the sole reliance on neoclassical auction economics rather than a broader set of organizational, contractual, market, behavioral, and political complexities.

In decarbonized systems dominated by intermittent wind and solar generation, the traditional reasonably static economic dispatch curve (whether based on estimates of short-run marginal cost for regulated vertically integrated monopoly utilities or wholesale market price signals in liberalized markets) changes rather dramatically. Since, the marginal operating cost of wind and solar generating facilities is effectively zero, if wind and solar generators were dispatchable generators in the traditional sense they would appear at the bottom of the economic dispatch curve and would be called by the system operator all of the time up to their maximum capacity capabilities. However, because wind and solar generation is intermittent, the quantity of electricity they are able to supply varies, sometimes very quickly, with exogenous variations in wind and solar irradiation conditions, not through traditional economic dispatch decisions by the system operator. Basically, the system operator is now faced with an economic dispatch curve that moves inward and outward based on exogenous changes in wind speed and solar irradiation, and wind and solar irradiation conditions can change very quickly even within an hour during the day. Market prices may be zero (or negative)Footnote ¹¹ when wind and solar are abundant and on the margin, above zero as storage charges up, and then very high when storage discharges or when scarcity conditions emerge and the system clears at the price cap.

Figure 1 displays wind generation every 5 minutes in ERCOT (the ISO covering most of Texas) during August 2020 to provide a better sense for what intermittency means. Note the large intra-day and day-to-day variation in output during the month. It varies from almost 0 to almost 20,000 MW and there are large variations even within a single day. This variation is driven by changes in wind speeds and direction, not by the system operator managing a bid-based economic dispatch schedule. ERCOT is a summer peaking system so that intermittency is a potential security of supply challenge in August.Footnote ¹² At the present time, the ERCOT system operator relies on a large portfolio of gas-fueled generation to balance supply and demand in response to the variation in intermitted wind (and growing solar) generation. In a fully decarbonized system, this gas generation would no longer be available to balance swings in wind and solar generation. Storage would play this role instead.

Figure 1. Wind generation in ERCOT August 2020 produced using NRGStream Trader 8.1.

In an efficient fully decarbonized system, the short-term wholesale market price distribution will look very different from the wholesale price distributions that we see today, with a lot of near zero or negative price hours when wind or solar are on the margin and a small number of very high-price hours, at least compared to the upper tail of the price distributions we see today (Cole and Frazier, Reference Cole and Frazier2018, Figure 12; Joskow, Reference Joskow2019). When short-term prices are very low, the quasi-rents produced to cover the cost of investments are also very low. As a result, the net revenues that wind and solar generators earn become much more concentrated hours approaching ‘scarcity conditions’ which are exactly the hours when short-term market price signals are suppressed. The distribution of short-term energy prices also depends heavily on the mix of wind, solar, storage, and dispatchable generation on the system. This will vary from system to system reflecting the mix of wind and solar resources that are available to it. What these price distributions will look like over a 20-year investment horizon is very uncertain today, depending on uncertain future decarbonization policies, uncertain mix of wind, solar, storage, and remaining dispatchable generation, certain future wholesale market design changes, and uncertain future technological change. To be clear, it is not just that wholesale prices will be characterized by more variation than is the case today, but that the entire distribution of wholesale prices in a fully decarbonized system will change and how it will change is very uncertain today. This is a situation where bounded rationality and the absence of a full set of contingent claims contracts may result in behavioral limitations on the ‘rational’ consideration of all future states of nature (Williamson, Reference Williamson1975). The high incidence of very high prices also may lead to opportunistic interventions by system operators or regulators to cap market prices for political reasons. As Williamson (e.g. Williamson Reference Williamson1985) has argued many times, opportunism leads to organizational or contractual responses to mitigate its costs.

The challenges

Electricity policymakers around the world are asking a similar set of questions about short-run and long-run wholesale and retail market design in a world where decarbonized electric power systems are dominated by intermittent generating technologies and storage.

1. (1) What changes in the design of short-run wholesale energy and ancillary services markets and transmission enhancements need to be made to manage intermittency and storage on the grid so that traditional operating reliability criteria can be achieved?

2. (2) How can an efficient portfolio of investments in wind and solar generating capacity, and perhaps other future zero-carbon generating technologies, as well as storage, be stimulated to meet the ambitious decarbonization goals are met on schedule and at the lowest cost feasible given the constraints?

3. (3) How can deep decarbonization goals be achieved while achieving long-run supply security goals in systems with large quantities of intermittent generation?

4. (4) How can short-run wholesale market price signals be reflected in retail prices and demand response services to help to achieve ‘green’ electrification goals in other sectors – e.g. transportation, buildings, selected industrial applications?

I have discussed challenges for the design of short-term markets elsewhere (Joskow, Reference Joskow2019). Here, I will focus on the two investment-related questions identified above (2, 3), and responding to them in a cost-efficient way? Specifically, I will focus here on the mechanisms supporting investment in solar and wind generating capacity and storage to meet aggressive decarbonization goals, supply security goals, and cost efficiency goals.

Why are policymakers concerned that short-term wholesale price signals will not provide adequate incentives to meet these goals? First, we must recognize that adding deep decarbonization commitments – both the endpoint and the trajectory for getting from here to there – means that we need to add at least one additional policy instrument to existing wholesale market mechanisms. The social cost of greenhouse gas emissions is a global externality problem. The obvious candidate instrument for internalizing this externality is to establish a stable credible trajectory for carbon emissions prices using either a carbon tax or a cap and trade program consistent with decarbonization commitments and the speed with which these decarbonization commitments are supposed to be achieved. Carbon emissions prices would be easily reflected in short-term wholesale market prices and quantities. The practical behavioral problem here is that there is tremendous uncertainty about what the carbon emissions price trajectory needs to be to achieve the deep decarbonization quantity commitments made by governments. In some areas, like much of the USA, there is no carbon price at all and where it exists it is not alone adequate to achieve the decarbonization commitments made by policymakers. The EU has a cap and trade program for CO₂ emissions that establishes emissions prices but emissions prices have been volatile, a credible carbon price trajectory has not been established, and the emissions prices are not necessarily consistent with the EU countries' decarbonization commitments. Estimating the social cost of carbon upon which a Pigouvian carbon emissions tax might be based is a complex and uncertain undertaking (National Academy of Sciences, 2017). Moreover, and probably more importantly, many policymakers and climate scientists simply have more faith in working with quantity goals, targeted subsidies, and zero-carbon generating and storage resource procurement obligations, rather than relying on assumptions about the optimal carbon price trajectory. Accordingly, for the last decade or so, policymakers have instead relied heavily upon a rather ad hoc and inefficient set of renewable energy subsidies, feed-in tariffs, renewable or "clean" energy portfolio standards (RPS), and other mechanisms to support the transition to a decarbonized electricity sector.

A second concern results from the considerable uncertainty about how the distribution of short-term wholesale prices will evolve in the future. As noted above, there will be many more low or zero or even negative price hours and many more high price hours determined by the charging and discharging of storage and scarcity pricing as discussed earlier. Thus, the net revenues from investment in generation and storage will depend on getting the short-run prices exactly right in a much smaller number of hours. Moreover, the future distributions of short-term wholesale prices depend heavily on the ultimate mix and entry/exit trajectory of wind, solar, and storage, which are very uncertain today. Short-term market designs will change to manage systems dominated by intermittent generation, but how they will change is very uncertain today. The increased reliance on high prices in a small number of hours during the years also raises opportunism issues as system operators are guided by regulators, behavioral constraints, and politics. There is also growing evidence that these uncertainties increase the cost of capital for developers of wind and solar generating facilities which are not supported by long-term PPAs with credit worthy counterparties, including the government through the use of the regulatory system to guarantee cost recovery (Grubb and Newbery, Reference Grubb and Newbery2018).

A third concern is that technological change, especially learning by doing, will continue to play an important role driving the costs down of wind, solar, and storage, and stimulating innovation in new economical zero-carbon technologies during the transition as has been the case in the last decade. All of the cost components of wind, solar, and lithium-ion batteries, including so-called soft costs of construction and maintenance, have declined dramatically in the last 10 years as these technologies have continued to penetrate electricity systems (The Economist, January 9, 2021, pp. 5–7). If learning by doing is important, the full value of moving down the learning curve may not be internalized in competitive wholesale markets and associated price incentives. Learning by doing also creates potential hold-up or opportunism opportunities a la Williamson as investors and system operators look to supporting the newer lower cost technologies and ignore cost recovery for the first movers. Long-term PPAs with credit-worthy counterparties can largely eliminate this concern by shifting the ‘stranded cost’ risk to the buyer.

A fourth concern is that traditional approaches to measuring security of supply will not be effective in systems dominated by intermittent generation. In systems dominated by intermittent generation, the process for assessing a system's supply security attributes and designing mechanisms to ensure that supply security goals are achieved becomes quite different from the traditional approach for systems dominated by dispatchable generating capacity. In traditional dispatchable systems, resource adequacy or security of supply mechanisms are based on long-term forecasts of system peak demand, the variance in peak demand around these forecasts, assumptions about the associated probability of involuntary curtailment of demand at various levels of dispatchable generating capacity, and the VoLL. Historically, and to oversimplify, capacity obligations, whether for a whole ISO or for a traditional vertically integrated utility, are driven primarily by forecasts of peak demand on the system 10 or more years in the future, assuming that the dispatchable generating capacity will be available with high probability at the times of peak demand.

This traditional security of supply measurement protocol will eventually fail in systems dominated by wind and solar generating capacity. Since the generation from these resources is highly variable, driven by exogenous changes in wind speed and solar irradiation, it cannot be assumed to be available to supply at full capacity when system demand peaks. In solar-dominated systems, supply is likely to be abundant during the day when total demand is at its highest level but falls off quickly as the sun goes down. The ‘net demand’ faced by the system, that is total ‘gross’ demand by consumers minus the supply of intermittent generation, becomes the relevant driver of reliability and supply security evaluations.Footnote ¹³ Depending on the mix of wind and solar capacity and variations in wind speed and solar irradiation, the peak net demand that the system operator must balance could be at virtually any time and will be much more variable than gross demand, reflecting the netting out of variable wind and solar generation. Thus, evaluating whether security of supply criteria are being met requires integrating the stochastic attributes of wind speeds and solar irradiation, with wind, solar, and storage capacity assumptions, and demand forecasts for individual hours in the future. Defining the stochastic properties of the wind and solar resource drivers in turn requires many years of detailed weather data for key generation locations, data that are often unavailable for a long historical period. The methods for doing all of this are just being developed and applied in practice.Footnote ¹⁴

For example, Figure 2 displays the gross demand and the net demand for electricity within the California ISO (CAISO) on August 14 and 15, 2020. CAISO implemented rolling blackouts in the early evening on those days due to an imbalance between generation supplies and demand. As noted, the net demand on grid-based generation is the gross demand less ‘intermittent’ supplies from wind and solar capacity. On August 14, the peak net demand occurred 2 hours later than the peak system demand and it was at the time of the net peak demand that rolling blackouts had to be implemented due to a deficiency in generation supply to serve demand. On August 15, the net peak demand is an hour later than the gross peak demand and it is just after the peak net demand that rolling blackouts were implemented. Finally, note that since CAISO is a solar-dominated system, as the sun goes down the net demand increases rapidly and considerably in the early evening. Today, this ‘ramp’ is satisfied primarily with gas-fired generation and some imports from other states. California has committed to fully decarbonize its electricity sector, so in the future, meeting the late afternoon net demand ‘ramp’ will likely rely primarily on storage capacity that will have to be added to the system in the future or new technologies that are not yet economical, such hydrogen produced without carbon emissions.

Figure 2. Gross and net demand in CAISO August 14 and August 15, 2020. Source: California ISO, Final Root Cause Analysis: Mid-August 2020 Heat Wave, p. 45 (http://www.caiso.com/Documents/Final-Root-Cause-Analysis-Mid-August-2020-Extreme-Heat-Wave.pdf).

Thus, depending on relative penetrations of wind and solar, some intermittent technologies may be low-cost contributors to carbon emissions reduction but not particularly valuable for meeting security of supply criteria. For example, calculations of the contributions of different intermittent zero-carbon technologies to supply security (Electric Load Carrying Capability – ELCC) in California is forecast to be quite different as a result of differences in the stochastic production attributes of these technologies and their penetration into the system. Incremental supplies of solar capacity are relatively inexpensive options for reducing carbon emission by replacing fossil generation but have a very low value for meeting supply security criteria. This is the case because California has a system that already has a significant amount of grid-based and distributed solar photovoltaic (PV) capacity and the peak net demand has shifted to late afternoon or early evening when the sun goes down. Add storage to the solar PV and the supply security value increases significantly. Wind has a higher supply security value because it is more likely to supply during the net peak demand period. Again, add storage and the supply security value increases significantly (Astrapé Consulting, 2020).

Long-term PPAs, competitive procurement, and hybrid markets

How are policymakers responding to these concerns and how are they likely to perform? The earliest efforts to stimulate increased investment in wind and solar generating capacity relied on various combinations of direct and indirect subsidies in the form of production and investment tax credits in the USA and feed-in tariffs in Europe. In the USA, many U.S. states adopted RPS that obligated retail suppliers to purchase generation services with a specified percentage of renewable energy. RPS programs were often matched with renewable energy credits (RECs) which renewable energy suppliers could sell to retail suppliers to meet their RPS obligations. However, there was originally no obligation in most wholesale market designs for retail suppliers to enter into long-term contracts for resources that met RPS obligations, the market values of RECs could be quite volatile, and very dependent on changing regulatory definitions of various categories of RECs often with very different prices.

Regarding cost efficiency, these mechanisms are widely believed to be extremely inefficient ways to stimulate investment in wind and solar generating facilities. They typically do not distinguish between the carbon emissions mitigation and security of supply attributes of different technologies, do not distinguish between the costs of different technologies from either perspective, largely ignore storage, and do not do either in an overall systems context.

More recently, we are seeing much more engagement of government in planning decarbonization ‘pathways’ or integrated resource plans using optimal long-term planning and dispatch models that include solar, wind, storage, and other zero-carbon emissions technologies like nuclear power, to produce an indicative plan or pathway that meets both decarbonization commitments and security of supply criteria. These indicative plans are then integrated with procurement obligations and mechanisms to support new wind, solar, and storage projects consistent with the plan. That is, the reliance on long term planning and procurement obligations represents at least a partial rejection of the philosophy that the invisible hand alone should guide entry and decisions. Specifically, policymakers are using the indicative long-term plans to mandate that specified counterparties, typically retail suppliers, enter into long-term contracts – PPAs – with zero-carbon generators and storage developers consistent with the indicative plan.

Long-term PPAs transfer some or all of the market price risk, but not construction cost or performance risk, from the generators to the buyer counterparties, for periods of 10–30 years. They do so by agreeing to a price the buyer will pay for energy from the generator, as well as performance criteria, over the term of the contract. The generator still sells into the short-term wholesale markets but is compensated for the difference between the strike price and the market price if market prices are below the strike price and pays the buyer the difference between the market price and the strike price if market prices are above the strike price. This type of ‘contract for differences’ insulates the seller from all future wholesale market price risk by guaranteeing a fixed real price. Since the marginal cost of generation is typically far below the strike price in these PPAs, the generators have powerful incentives to maximize their output.Footnote ¹⁵ Alternative risk allocation structure that leave some market price risk with the seller are also available and are used in practice. Most importantly, governments are getting much more involved in long-term planning and procurement of wind, solar, and storage to meet decarbonization commitments and security of supply criteria. This is a partial recreation of the hierarchies that sector liberalization broke down. Although not the same as vertical integration, the long-term PPAs create a new contract-based hierarchy linking buyers and sellers in long-term relationships. These programs are taking account of and being coordinated with capacity remuneration mechanisms to achieve supply security goals as well as decarbonization goals. At least 50 countries have relied to some extent on long-term PPAs mediated through competitive procurement programs to acquire some or all of the wind, solar, nuclear, green natural gas, storage, etc. that they forecast they need to meet their decarbonization goals and, very recently to consider simultaneously their supply security goals (IRENA, 2019).

For example, the system in England and Wales has relied upon auctions for long-term contracts (PPAs) for offshore wind and new nuclear generating capacity for several years (Grubb and Newbery, Reference Grubb and Newbery2018) and has both a short-term and long-term capacity market mechanism to manage security of supply criteria. Spain recently initiated its first long-term procurement for wind, solar, and storage (Energy Storage News, January 19, 2021) with 12-year PPAs. Several other countries in Europe have also initiated auctions for zero-carbon generating capacity supported by long-term PPAs (http://aures2project.eu/project/#aures). They include Denmark, Germany, Italy, Ireland, and others. The EU RES Financing Mechanism (http://aures2project.eu/wp-content/uploads/2020/11/AURES_II_D6_3_EU.pdf) also relies on competitive procurement auctions for PPAs, while also facilitating cross-border trade. In the USA, several states have initiated competitive procurement programs for specified quantities of offshore wind, solar, and other zero- or low-carbon technologies. They include Maine, Massachusetts, Connecticut, New York, Maryland, and New Jersey. In California and some other states, load serving entities have renewable energy portfolio requirements, a large portion of which must be supported with PPAs of at least 10 years duration. In Virginia, where the major utility is still a vertically integrated regulated monopoly, a state law has been passed with detailed decarbonization requirements which the two vertically integrated utilities in the state can meet either through ownership of wind and solar generating facilities or through purchases from project developers under PPAs (https://lis.virginia.gov/cgi-bin/legp604.exe?201+sum+HB1526).

These developments reflect four important themes. First, governments have decided that they must get more involved in pushing their electric power systems to decarbonize more quickly, more deeply, and more efficiently than is emerging from the existing short-term wholesale markets and a wide array decarbonization incentive and mandate schemes. Second, they want to have more control over successfully achieving the decarbonization commitments they have made. Third, is the view that long-term PPAs reduce the cost of capital faced by developers of wind, solar and storage projects by transferring some market risk from project developers to credit-worthy counterparties, facilitating their development and lowering their costs to consumers. Fourth, is the reliance on competitive procurement auctions that require potential developers to compete for projects so that the lowest cost suppliers with the best cost containment commitments can be selected. Accordingly, what is emerging are what Roques and Finon (Reference Roques and Finon2017) refer to as ‘hybrid markets’ or what Williamson would categorize as a ‘hybrid form’ that includes a variety of contractual arrangements short of vertical integration.

The reliance on long-term PPA obligations placed on retail suppliers, regulated distribution or transmission companies or other counterparties (like the government) is not an abandonment of competition as some have argued. Rather, these developments partially replace the reliance on short-term wholesale prices and voluntary market driven hedging contracts of limited duration to bring forth the targeted quantities and types of wind and solar generating capacity and storage to meet decarbonization commitments with competitive procurement of zero-carbon energy and storage supported with long-term PPAs. As short-term markets evolve, as carbon pricing evolves, as the technologies evolve, as uncertainties about wholesale price distributions and future changes in short-term wholesale market design are reduced, and as voluntary hedging and contractual arrangements expand and mature, the competitive procurement program can itself can evolve to require developers to bear more and more short-term market risk, rely more on voluntary market-based hedges to reduce those risks and their cost of capital, and ultimately rely primarily on voluntary market mechanisms to determine the mix of generating and storage technologies.

What evidence is there that competitive bidding for long-term PPAs can be an ‘economizing’ approach to procuring wind, solar and storage resources to meet decarbonization commitments and supply security criteria? We have the evidence from the UK discussed by Grubb and Newbery (Reference Grubb and Newbery2018) and noted above. However, we also have supporting evidence from private companies and non-profit organizations that have begun to purchase zero-carbon resources to meet internal decarbonization goals. A number of companies, government entities, universities, and other organizations with excellent credit ratings have made voluntary commitments to decarbonize the electricity that they purchase. These voluntary commitments reflect the organizations' internal decarbonization goals which in turn reflect the preferences of boards, employees, and some customers to decarbonize the companies' activities, typically not regulatory requirements or carbon emissions prices. Although these organizations could have met their internal decarbonization commitments by buying RECs each year or in retail competition systems by asking their retailers to contract on their behalf with wind and solar generators relying on short-term contracts. Instead, many of these organizations have turned to relying on long-term PPAs with new wind and solar projects to meet their internal decarbonization commitments.Footnote ¹⁶

My point is not that we can rely on corporate PPAs alone to meet system-wide decarbonization and supply security goals – we cannot.Footnote ¹⁷ Rather, it is that the revealed behavior of these organizations provides useful information about the most efficient contractual arrangements to support achieving decarbonization commitments. Most importantly, these organizations have concluded that the cost of the carbon-free electricity they are purchasing is lower if they enter into long-term PPAs where some or all of the market risk is transferred from the electricity supplier to them. The kinds of organizations that have adopted internal decarbonization goals and rely on long-term PPAs to meet them typically have excellent credit ratings (think Google, Microsoft, Apple, Facebook, Walmart, University of California, MIT, etc.). The project developer can then finance the project at a lower cost using more debt, less costly debt, and less equity than otherwise, reducing their weighted average cost of capital and the overall cost of supplying electricity from these carbon-free generators. These organizations have also turned to organized auction-based competitive procurement programs to seek the least costly projects to meet their decarbonization commitments. The experience with contract design gained from these voluntary PPAs can also be useful in guiding contract design more generally.Footnote ¹⁸ Walmart's perspective is typical:

Other organizations and analysts have favored relying more on medium and long-term PPAs for similar reasons.Footnote ¹⁹

Pushback

Many reasonable questions have been raised about this movement to hybrid markets that rely on long-term indicative resource planning, competitive procurement for long-term contracts and supplemental capacity remuneration support mechanisms, to meet security of supply criteria and decarbonization commitments, and short-term markets to operate the resulting stock of generating and storage facilities efficiently and reliably. First, some have argued that these changes represent an abandonment of competition. I believe that this view is wrong. There are many types of competition and multiple market designs that can yield similar performance. Of particular relevance is Harold Demsetz's (Reference Demsetz1968) distinction between ‘competition in the market’ and ‘competition for the market’, in his case the latter referring to competition for long-term franchise contracts. Williamson (Reference Williamson1976) raised serious questions about the performance of competition for the market through franchise bidding for cable TV monopolies. However, Williamson was referring to competitive bidding for a complex dynamic natural monopoly. Here, the competitive procurement process is being applied to choose a portfolio of individual generation projects.

A related concern is that hybrid markets represent a return to ‘central planning’. This is not quite right although there are some potential concerns here. Supply security criteria and decarbonization commitments are established by governments and regulators. Well-designed competitive procurement programs can mitigate some of the inefficiencies of central planning. These mechanisms should have clear objectives, good auction designs and selection criteria, and in principle should be technology neutral, but not attribute neutral, allowing the competitive procurement process ultimately to determine which combination of competitive offers meets the security of supply criteria and decarbonization commitments most efficiently. Yet, many procurement programs today are not technology neutral, procuring wind, solar, storage, etc. in separate auctions with the quantities for which bids are sought established in ways that often appear to be arbitrary.

The challenge here is to develop auction and evaluation mechanisms that allow the available technologies to be compared and integrated into an entire portfolio of generating and storage projects taking account of the attributes of different technologies that contribute to decarbonization and/or supply security goals. Corneli (Reference Corneli2020) has outlined an interesting approach that relies on the use of transparent long-term optimal planning and dispatch models that integrate the expected costs and operating characteristics of all zero-carbon and storage technologies with decarbonization commitments, supply security criteria and forecasts of electricity demand. The long-term optimal planning and dispatch models define an initial configuration of wind, solar, and storage capacity. This defines the initial allocation to start the competitive procurement process. It is integrated with competitive procurement in a sequential iterative ‘reconfiguration’ auction framework that adjusts the target portfolio and ultimately the award of PPAs, to the additional information provided in the sequential competitive bidding process. The auction results would ultimately be technology neutral, but not attribute neutral, as the target portfolio is reconfigured through the sequential auction process. Since these auctions could be repeated every couple of years to meet incremental capacity needs consistent with supply security and decarbonization goals, the parameters for the different technologies used by the models can be updated frequently based on the outcome of the previous auctions as well as relevant general market information from other auctions.

There are still several basic design issues that need to be addressed for hybrid markets to perform well. First, auction designs and evaluation criteria need to be developed further, applied in practice, the results evaluated, and adjustments made. Second, contract designs should be compatible with efficient integration with short-term wholesale markets. On the one hand, PPAs with wind and solar generators with near zero short-run marginal cost should be designed in a way that they are ‘dispatched’ whenever they can supply energy and there is adequate demand and network operating support (e.g. local transmission capacity) to accept the supplies. On the other hand, in fully decarbonized systems, there will be times (e.g. windy, sunny days with relative low demand) when the system cannot accept all of the energy produced from intermittent resources and they must be curtailed. Contracts should be structured to support efficient curtailment mechanisms, but this has not always been the case to date. In particular, the standard contract for differences will lead to inefficient curtailment outcomes (Newbery, Reference Newbery2020).

Contracts for storage should look quite different from contracts for zero marginal cost wind and solar generation. For storage to operate efficiently it must face short-term market prices for energy and ancillary services since its efficient operation relies on arbitraging prices at different times – buying low and selling high – essential moving electricity supplies from one time period to another. The proper contract form is likely to be a fixed payment reflecting the power (MW) of the storage and its energy capacity (MWh). These contracts would be more like capacity payments and since storage is dispatchable with energy limits capacity markets can be adjusted to accommodate it.