Vol.:(0123456789) International Journal of Economic Policy Studies (2019) 13:3–26 https://doi.org/10.1007/s42495-018-0009-0 1 3 SPECIAL FEATURE: RESEARCH ARTICLE European energy market liberalization: experiences and challenges Guido Pepermans1   Received: 18 August 2018 / Accepted: 5 November 2018 / Published online: 3 December 2018 © Japan Economic Policy Association (JEPA) 2018 Abstract The European electricity market liberalization process started more than 20 years ago, one of its main purposes being to organize the provision of electricity and gas more efficiently by introducing competitive forces where possible and regulation where needed. Despite the fact that most Member States of the European Union (EU) have liberalized their electricity market, a European internal market for electricity—one of the ultimate goals of EU energy policy—has not been realized yet. For this to be realised a number of remaining obstacles need to be removed. This paper gives a brief overview of the EU electricity market liberalization process and surveys the main remaining issues and challenges. At the retail market level, additional efforts are required to increase retail competition and to remove regulated retail prices. Also, more effort is required to physically integrate existing regional electricity markets, by removing the existing barriers between regions and countries. Building new interconnection capacity and improving the allocation procedures for cross-border capacities can help achieving this. Furthermore, the EU should increase efforts to further harmonize and integrate electricity markets and electricity market policies and to encourage the demand-side of the market to participate more intensively through demand response. Keywords  Electricity market · Liberalisation · European Union · Challenges JEL Classification  D42 · E60 Energy Market Reform This paper is based on a presentation for the 2017 International JEPA conference in Okinawa, Japan (November 4, 2017). * Guido Pepermans guido.pepermans@kuleuven.be 1 KU Leuven, Faculty of Economics and Business, Centre for Economics and Corporate Sustainability, Warmoesberg 26, 1000 Brussels, Belgium 4 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 Introduction The European electricity market liberalization process started more than 20 years ago. Its main purpose was to organize the provision of electricity and gas more efficiently by introducing competitive forces where possible and by using regulation when and where needed. More competition creates incentives for firms to invest in cost reducing and innovative technologies and thus fosters cost-efficiency. Competition would thus force down prices, which by many politicians was seen as the ultimate goal of the liberalization process. Figure 1 illustrates the liberalization idea for the electricity sector. The left-hand side sketches the structure of the electricity sector as it was before the initiation of the liberalization process. Most Member States had one vertically integrated firm that dominated the sector.1 This vertically integrated firm took care of electricity generation, transmission and distribution and consumers had no choice as to where to buy electricity. The goal of liberalization was to move to a market structure as shown at the right-hand side in Fig. 1, which allows for competition in electricity generation and retail, while keeping transmission and distribution activities regulated. Economies of scale are typically quite important in transmission and distribution, which justifies regulation. Today, most Member States of the European Union (EU) have liberalized their electricity market. The United Kingdom was the first EU Member State to liberalize Fig. 1  Liberalising the electricity market 1  Some Member States had two of such vertically integrated firms, but then these forms would be regionally organized. 5 1 3 International Journal of Economic Policy Studies (2019) 13:3–26 its electricity market already more than 20  years ago, soon followed by Norway (which is not an EU Member State). Most other Member States initiated the liberalization process shortly after 1996 and ended the process about 10 to 15 years ago. However, although most Member States formally ended the liberalization process, a European internal market for electricity has not been realized yet. To complete the creation of a European internal electricity market—one of the ultimate goals of EU energy policy—a number of obstacles remain and need to be removed. The purpose of this paper is to give a brief overview of the EU electricity market liberalization process and to survey the main remaining issues and challenges. To do this, we will make intensive use of publicly available reports and studies, of which we will synthesize the main messages, findings, conclusions. The structure of the paper is as follows. “The European Union, some facts and figures” briefly surveys and describes the European Union, presents some facts and figures, and puts it into perspective from a more global point of view. This section also surveys the most important legislative initiatives in the context of EU energy policy making. “Experiences from liberalising the electricity market” then surveys some facts and figures on the energy market liberalization. In “Challenges”, we then describe the main challenges faced by EU policy makers, when creating an internal energy market. Finally, “Conclusion” concludes. The European Union, some facts and figures The foundations for the European Union (EU) as we know it today were laid in the aftermath of the Second World War. Already from the early start, the goal was to join efforts in economic cooperation, as it was felt that countries that intensively interact through trade and economic activity are less likely to get involved in conflicts. This resulted in the creation of the European Coal and Steel Community (ECSC), formally established in 1951 by the treaty of Paris. At the start, the ECSC counted six members: Belgium, France, Italy, Luxembourg, The Netherlands and West Germany. In 1957, soon after the installation of the ECSC, four other communities were created by the Treaty of Rome, the most important one being the European Economic Community (EEC). The Treaty entered into force on January 1st, 1958. About 60  years and some international treaties later, these institutions have merged into the EU as we know it today. The EU also has a much broader perspective than the ECSC or the EEC initially had. It spans policy areas such as climate, the environment, health, external relations, security, justice and migration. Over the past decades the number of Member States has steadily increased from 6 Member States in 1958 to 28 Member States in 2018 (see Fig. 2). To put the EU into perspective, its total size is about 4.5 million square kilometers, which is less than half of China’s size, about 12 times the size of Japan, and about half of the size of the United States. The EU has a population of just above 500 million people, which is about 200 million more than the US population and about 4 times the population 6 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 of Japan. In terms of GDP (estimates for 2017), the EU ranks second, after China and before the United States (3rd) and Japan (5th).2 In 2017, GDP per capita in the EU was $39,000 which is more than double the per capita GDP of China, about the same as Japan’s GDP per capita but far below GDP per capita in the United States. In terms of total electricity consumption, the EU ranks third, after China and the United States and before Japan on the fifth place. Expressed in ‘per capita’ terms, a US citizen consumes more than double the electricity consumed by a EU Fig. 2  EU Member States in 2018 Table 1  The EU relative to other countries in the world EU US China Japan World Land area (1000 km2 ) 4.476 9.148 9.326 365 148.940 Population, Mln (estimate 2017) 513 326 1.386 127 7.530 GDP per capita, PPP ($ estimate 2017) 41.327 59.532 16.807 43.876 16.961 Electricity consumption per capita, kWh (2014) 5.908 12.984 3.927 7.820 3.125 2   CIA World factbook: https​://www.cia.gov/libra​ry/publi​catio​ns/resou​rces/the-world​-factb​ook/. 7 1 3 International Journal of Economic Policy Studies (2019) 13:3–26 citizen. Also for Japan, electricity consumption per capita is about 40% above the per capita consumption level in the EU (Table 1). The EU has 24 official languages, which makes communication and law-making time consuming as all legal text have to be translated into these 24 official languages. EU institutions and policy making in the energy domain Before briefly describing the EU decision making process on energy matters, we first describe the most relevant institutions involved in the EU decision making process. Figure 3 illustrates the legislative, executive, and judicial bodies in the EU. Here, we focus on the legislative and executive bodies. The European Commission forms the executive branch and is the only institution having the right to initiate legislation. It has, however, no decisive say. The Commission can also be mandated by the EU Council to prepare legislation. The Council of Ministers is populated by the respective ministers (of Energy) of the Member States. Typically, a minister in the Council has a strong focus on the interests of the member State he or she represents. In the past, decision making had to be done unanimously, but the Lisbon Treaty has made it possible to take decisions with a qualified majority. Fig. 3  EU legislative, executive and judicial bodies 8 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 The European Parliament constitutes the legislative branch. The parliament has been given more power by the Lisbon treaty as now a simple majority is sufficient to take decisions. Finally, Member States are represented via their prime ministers or heads of state in the EU Council. Members of the European Council determine the general direction of energy policy. In fact, the EU can only act when given the competence to do so by the Member States, which is not always in the short term interest to do so. Policy making in the energy domain is a shared competence, i.e. it involves both the European Commission and the Member States [13]. Two exceptions exist: (1) a Member State has the right to determine the conditions for exploiting its energy resources and its energy mix, and (2) unanimity voting is required for measures which are of a fiscal nature. In almost all policy domains (including energy), the ‘Ordinary Legislative Procedure’ is used to make policy decisions. Under this procedure, EU legislation has to be approved on an equal footing by the (directly elected) European Parliament and the Council of Ministers. Energy policy in the EU The legal basis for the formation of a European Internal Energy Market (IEM) is provided by the treaty on the Functioning of the European Union [12]. The ultimate goal is to create a well-functioning European energy market, putting fair access and a high level of consumer protection in the center, while also guaranteeing adequate levels of generation and interconnection capacity. The first steps towards the IEM were actually set in the 1970s when the international oil crisis triggered some initiatives to start cooperating on a number of energy related issues. Initially, this cooperation was not very successful, but in the late 80s and early 90s the climate change problem came to the forefront and Member States increasingly became aware of the importance of developing coordinated policies on energy production and energy use if one wanted to do something about climate change [16]. This led to two electricity market liberalization waves. The first wave started in 1996, when the first European Directive concerning the liberalization of the electricity market was adopted. A similar initiative was taken for the gas sector and 2 years later, a Directive focusing on the European gas markets was adopted. These two directives, also known as the First Energy Package, initiated the European energy market liberalization process. The first electricity Directive defines the common rules for an internal electricity market and introduces the idea of competition in generation and so-called third party access, which means that all generators should have access to the electricity grid at an affordable and correct price in order to be able to transport their product to the end users. The idea of unbundling of generation, transmission and distribution was also initiated in the first Directive. In 2003, the Second Energy Package was adopted, including a second Directive on the electricity (and gas) market.3 The second electricity Directive allows suppliers to enter markets in other Member States’ and consumers to choose their supplier. 3   Directive 96/92/EC and Directive 2003/54/EC. Both Directives and other legal texts can be download from https​://eur-lex.europ​a.eu/. 9 1 3 International Journal of Economic Policy Studies (2019) 13:3–26 As of 2000, numerous other legislative initiatives were taken. One of the most important ones is the Energy Action Plan, dating from March 2007. The plan identifies three major challenges and places them at the center of European energy policy: sustainability, security of supply and competitiveness [13, 17]. Up to today, these challenges are still core for energy policy decision making. In 2009, the Third Energy Package was adopted. This package completes the integration of the European electricity and gas markets. Its main elements are ownership unbundling (separation of generation and retail activities from transmission activities), the obligation for all member States to establish a National Regulatory Authority and the installation of an Agency for the Cooperation of Energy Regulators (ACER), a forum for National Regulators to cooperate at the EU level. Another important initiative is the Climate and Energy Package 2020, dating from January 2008. The package sets out three important targets for EU policy making that go beyond, but with significant impact on, the creation of an internal electricity market: (1) reduce EU greenhouse gas emissions by 20% below the 1990 levels, (2) increase the share from renewables in EU energy consumption to 20%, and (3) improve energy efficiency by 20% relative to 2007 projections about energy consumption in 2020. Each of these goals has a potentially significant impact on the functioning of our internal energy market. In November 2016, the European Commission presented a new package of measures, aiming to take another significant step towards the realization of an Energy Union, of which a fully integrated Internal energy market is a cornerstone. Today, the Commission, the European Parliament and the Council are negotiating on the adoption of what became known as the ‘Clean Energy for All Europeans’ package. The package contains eight legislative proposals, two of which are about electricity market design. More specifically, the proposals aim to strengthen the role of consumers, to increase transparency and to further improve market design [9]. The next section describes the current state of affairs of the EU electricity market. Our focus will be on the competitive segments in the electricity sector, but one must keep in mind that generation and retail activities cannot be isolated from transmission and distribution activities. Moreover, many other policy areas in the EU, as well as international events and global challenges will also be of direct or indirect influence on generation and retail activities. Experiences from liberalising the electricity market One of the primary initial goals of the liberalization process was to introduce competition in the generation and retail segment of the electricity market. This section focuses on the question whether this goal has been reached. To answer this question, we will heavily draw on the market monitoring reports published by the newly established Agency for the Cooperation of Energy Regulators [2, 3, 4] as well as on 10 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 IEA [13]. In our analysis we will differentiate between the retail and wholesale markets, the latter being related to electricity generation activities. “The electricity retail market” focuses on the retail market and will describe the evolution of retail prices and its breakdown in different components. Moreover, we will look into the level of competition achieved in the retail markets of the Member States. Where possible, we will distinguish two groups of end-users: households and industry. “The wholesale market” then focuses on evolutions in the wholesale (electricity generation) market, again based on the market monitoring reports published by ACER. The electricity retail market The discussion of the retail market breaks down into two parts. Many politicians consider prices—and more specifically the evolution of price levels—as the ultimate touchstone for the success of the liberalization. The first part of the discussion therefore focuses on the evolution of electricity retail prices over the past decade. The second part then assesses the success of the electricity market reform from a wider economic perspective and explores a range of market indicators, using the wellknown structure-conduct-performance framework. Evolution of end‑user prices Figure 4 illustrates the evolution of average end-user prices in the EU, i.e. including taxes and all other types of levies, both for residential and industrial consumers. The lines show the evolution of an index of electricity prices (base year 2008) in the household and the industrial sector, while the bars show the end user electricity price levels (left-hand axis) in the these sectors. A first thing to note is the gap between the average price levels in both market segments, which is explained by the lower amount of taxes and contributions for renewables incorporated in the industrial market segment. For many firms, electricity is an important input in the production process and thus also an significant element of the cost structure. Therefore, Fig. 4  Trends in final electricity prices for household and industrial consumers in EU Member States and Norway—2008–2016 (euro cents/kWh and index change, 2008 = 100) 11 1 3 International Journal of Economic Policy Studies (2019) 13:3–26 policy makers tend to keep end-user prices lower in the industrial segment in order not to undermine the competitiveness of firms in their output market. Second, also note that the price evolution is different for both consumer groups. Prices for residential end-users show an increasing trend, at least up to 2015. On the other hand, prices for industrial consumers remain more stable and even started declining as of 2013. Compared to 2008, residential end-user prices have increased by more than 26% in 2016, while industrial prices have risen with almost 5.8% [3]. This is particularly interesting as the evolution of wholesale electricity prices is Fig. 5  Evolution of wholesale electricity prices in the EU Fig. 6  Breakdown of end-user prices in the incumbents’ standard offer for households consuming 3500 kWh per year in EU capitals 12 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 essentially downward sloping between 2012 and 2017, as illustrated in Fig. 5. This latter figure shows a decreasing trend for wholesale electricity prices, starting in 2012. However, looking at the price evolution in Fig. 4, we see an opposite price trend, in particular for residential consumers. This suggests that residential consumers are not fully exposed to the evolution of wholesale prices or that other cost components than energy play an increasingly important role (cost of renewables policies, energy efficiency, climate change…). For the residential segment, this is further explored in Fig. 6, which shows the evolution of the shares of the different components in the incumbents’ standard offer for households consuming 3500 kWh per year in EU capitals (and Olso, the capital of Norway). In line with the evolution of wholesale prices, the average energy share in the electricity bill shows a decreasing trend, from 41% in 2012 to 35% in 2016. Over the same period, electricity prices increased by 6.6%, from 19.26 €cent per kWh to 20.53 €cent per kWh (see Fig. 4). Note that this decreasing trend in the share of the energy component hides quite some variation over EU Member States, as illustrated for 2014 in Fig. 7. In Figs. 6 and 7, the light blue bar indicates the share of transmission and distribution costs. For a representative household with a standard contract, the share in the electricity bill of this component (Fig. 6) remains stable, but the geographical variation in the transmission and distribution component (see Fig. 7) is largely due to differences in the geographical structure and population density of the Member States. Fig. 7  Breakdown of end-user prices in the incumbents’ standard offer for households consuming 4000 kWh per year in EU capitals (2014) (Note that the household consumption profiles have been adapted, from 4000 kWh in 2014 to 3500 kWh in 2016. This change is beyond our control. Despite this change in consumption profile, we feel that the key messages derived from the figures remain unchanged) 13 1 3 International Journal of Economic Policy Studies (2019) 13:3–26 Also the shares of the other components vary considerably over Member States. Most likely, this has to do with the way costs of energy policies are being recovered, i.e. by passing through these costs to electricity prices—via a surcharge or a tax—or by using other channels, such as for example income taxes. The yellow bars represent the share of taxes in the bill, while the green bars represent the surcharge for renewables. Also note that over time the renewable share increases significantly, as illustrated in Fig. 6. In 2016, EU residential consumers pay on average 2.6 €cent per kWh for renewable policies, while in 2012 they only paid about 1.6 €cent per kWh. Level of competition From an economic perspective it makes more sense to follow a more comprehensive approach to assess the level of openness and competition in the retail electricity market. To do that, ACER developed the ACER Retail Competition Index (ARCI), a single composite index. The ARCI allows to assess the relative level of competition in the retail electricity markets for the household segment in each of the EU Member States and Norway. The analytical framework used to develop the ACRI is the structure-conduct-performance framework according to which the performance of a sector depends on the structure of that sector, but also on the conduct of suppliers and consumers in the sector [1, 2]. Many indicators exits to measure a market’s structure, conduct and performance. For the construction of its composite index, ACER decides to use nine sub-indicators, three of which measure structure, four of which measure conduct and two of which measure performance. After normalization and weighting, these sub-indicators are then combined into one composite index for each Member State and Nor- way.4 The indicator takes a value ranging between 0 and 9, with a higher score indicating a better market situation. We briefly summarize some of these nine indicators and discuss the main findings for the ACRI, based on different ACER Market Monitoring Reports [1, 2]. Structure  To assess the structure of a market, three indicators are being used: CR3 (which is a measure of market concentration), the number of suppliers with a market share above 5% (these are so-called main suppliers), and the ability to compare prices easily. Figure 8 combines two of these measures, CR3 and number of main suppliers. The figure illustrates market concentration in EU Member States and Norway. Each circle represents one country. The vertical axis measures the total market share of the three largest suppliers in that Member State (CR3). The horizontal axis counts the number of main active suppliers, i.e. suppliers with a market share at least equal to 5%. A market is considered more competitive when the number of main suppliers is larger and C3 is lower. A larger circle reflects more active, nationwide suppliers in the market. The figure shows that the distribution of market concentration levels 4   For a more detailed description, we refer to (IPA Advisory Limited [15]). This document can be downloaded from the ACER website. 14 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 over Member States is skewed. In 2015, the average concentration level was equal to 78%, with a median value of 85%. Only 8 out of 29 countries have a CR3 less than 70% (red horizontal dashed line), 12 out of 29 countries need only two suppliers to obtain a total market share of more than 70%, In summary, these data suggest that the issue of market power is omnipresent in most EU Member States (Norway is doing relatively well). Conduct  In the ACRI indicator, conduct is captured by four sub-indicators: average consumer switching activity, consumer inactivity, average net market entry, and the number of offers per supplier in the capital cities. Consumer switching includes switching to another supplier as well as switching to another tariff with the existing supplier. Consumer inactivity measures the share of consumers that did not switch away from the incumbent, although they might have switched to another tariff. While switching rates have increased over the past years, an EU-wide average switching rate of 6.4% consumer switching should be considered as moderate [2]. How can this be explained? A separate study performed by ACER learned that the monetary benefits from switching are perceived as insufficient to make the effort. Moreover, the perceived complexity of the switching procedure and a lack of trust in the new suppliers are also mentioned as important reasons not to switch. It should therefore not come as a surprise that consumer inactivity is high. In only 4 Member States more than 50% of households have switched to a non-incumbent electricity supplier: Portugal, The Netherlands, Great Britain and Belgium. However, maybe we need to be more patient. Figure  9 relates consumer switching (vertical axis) to the number of years since the electricity market was Fig. 8  C3, the number of main suppliers and the number of nationwide active suppliers in retail electricity markets for households (2015) 15 1 3 International Journal of Economic Policy Studies (2019) 13:3–26 (formally) liberalized (horizontal axis). The figure suggests higher switching rates in Member States that started liberalizing their electricity market a longer time ago. This suggests that consumers need some time to adapt to the new market conditions. Another indicator of conduct is the number of contract offers in the retail electricity market, which can be interpreted as a measure of product innovation. ACER [1] and ACER [2] find evidence that Member States that have started their liberalization process longer ago, tend to have a higher number of offers. As a measure of the number of different contract offers in a Member State, one uses the number contract offers in the capital city of a Member State. Contracts are differentiated mostly on the dimensions ‘fuel type’ (electricity only, gas only, or a combined electricity-gas contract), ‘type of pricing’ (fixed, variable, spot based…), ‘energy source’ (fossil of renewable) and ‘additional services’ (contract duration, billing method…). Performance  Finally, two sub-indicators are used to construct the ACRI indicator for market performance: consumer satisfaction and average annual mark-ups. The mark-up applied in a market is an important indicator of performance of that market. The mark-up is equal to the difference between the price charged to the consumer and the cost to supply the consumer. A high mark-up reflects less com- Fig. 9  Switching rates in relation to the number of years since market liberalization (2014) 16 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 petition. In theory, a high mark-up would trigger new entry and more price competition, which would make the mark-up vanish unless the market is insulated from competitive pressure in which case the mark-up will continue to exist. Although some correlation is observed between entry/exit activity and mark-up levels, there is little evidence for a clear pattern or relationship [1]. To measure the mark-up, ACER uses the difference between the wholesale energy cost and the energy component of retail prices as a proxy. Figure 10 shows average mark-ups as measured over the period 2008–2015 for a selection of EU Member States and Norway. The dots indicate the 2015 mark-up levels. ACER [2] also notes that the energy component in retail prices correlates well with wholesale electricity prices for two groups of countries: those countries with regulated retail prices (dark bars in Fig. 10, mostly at the right-hand side of the EU-average) and those countries where offers contain a direct reference to wholesale costs. Note that some countries had negative average mark-ups over the period 2008–2015. This is not sustainable in the long run, as it creates a barrier for entry and for investment in the sector. Consumer satisfaction is the second sub-indicator of performance in the ACRI indicator. This sub-indicator is constructed based on a consumer survey undertaken by the European Commission, Directorate-General Justice and Consumers and measures to what extent products and services on offer from different suppliers live up to what one wanted within the past year (scored on a scale from 0 to 10). Bringing everything together in the ACRI indicator  The 2014 and 2015 ACRI score for EU Member States and Norway is shown in Fig. 11.5 Higher scores indicate more competitive markets. Many Member States made progress, but two Member States— Latvia (LV) and Malta (MT)—catch the eye. Three Member States saw their ACRI score decrease: Greece (GR), Hungary (HU) and Slovakia (SK) [2]. According to Fig. 10  Mark-ups for a selection of EU Member States and Norway 5  Details on the assumptions behind and the construction of the ACRI indicator can be found in IPA Advisory Limited [15], ACER [1] and ACER [2]. 17 1 3 International Journal of Economic Policy Studies (2019) 13:3–26 ACER [2], the change in ACRI-scores is—for most Member States—driven by four sub-indicators: the number of main suppliers, average net entry, number of offers, and average annual mark-ups. For a more detailed discussion of the underlying reasons for the observed changes, we refer to ACER [2]. Fig. 11  The ACRI-indicator for the household segment in the retail electricity market Fig. 12  Relationship between ACRI-score and number of years since liberalisation 18 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 Figure 12 illustrates a clear positive relationship between the ACRI-score for a Member State and the number of years that have passed since the liberalization process in the retail electricity market started. In general, the more time that passed since the implementation of the liberalization, the better countries are performing in terms of achieved level of competition. This suggests that it takes time for markets to become more competitive. The figure also shows that markets that liberalized later (i.e. in 2007, which in Fig. 12 corresponds to 8 years since liberalization) show varying levels of competition. This suggests that other factors, such as the speed of implementing market reforms (e.g. privatization and unbundling of the industry, slow transposition/implementation of Energy Directives etc.) can also impact the level of competition. Finally, note that countries with regulated retail prices for some household subsegments also tend to have lower levels of retail competition as these regulated prices discourage entry and innovation, increase suppliers’ uncertainty regarding long term profitability levels and reduce consumers’ incentive to switch supplier. This is illustrated in Fig. 13. Although no statistical test was performed, the figure suggests that having regulated prices for at least some part of the household customers results in a lower ACRI-score. Summing up, a lot of progress has been made in increasing the level of competition in the retail electricity market, but a lot of work still needs to be done. Electricity retail markets are still highly concentrated in many Member States and essentially remain national markets. Many consumers do not actively participate in the market, which lowers competitive pressure, resulting in higher retail prices despite low or even decreasing wholesale prices. Many markets also show a weak link between wholesale prices and the cost of the energy component in retail prices for the household segment. Price regulation for (at least part of the) residential customers is still widespread, resulting in larger shares of inactive of consumers. But even Fig. 13  ACRI-scores for EU Member States and Norway with and without end-user price regulation 19 1 3 International Journal of Economic Policy Studies (2019) 13:3–26 in Member States where prices are unregulated, many consumers—for a number of reasons—are still reluctant to switch supplier. EU policy makers are aware of this and are ready to take initiatives on consumer protection empowerment [14]. The wholesale market Creating an European integrated electricity market is an ambitious objective and requires full integration of all regional energy markets, networks and systems. Looking at day-ahead wholesale electricity prices, we observe a decreasing price trend since 2011 (see Fig. 5). At the same time, traded volumes as well as cross-border trade flows continued to increase. Can we therefore conclude that integration was successful? Not necessarily, as (a combination of) other reasons might explain this declining trend in day-ahead prices. First, due to the economic crisis electricity demand in the EU dropped by 6.8% between 2008 and 2014. Second, as a result of its climate change and renewables policies, the EU is facing increased penetration levels of renewables, which creates a so-called merit-order effect and crowds out thermal plants with higher marginal costs. To assess progress in market integration, it makes more sense to look at the wholesale price convergence across different zones in the EU. Taking convergence of wholesale electricity prices as an indicator of market integration, ACER [4] concludes that—although different degrees of convergence were observed across Europe—significant progress has been made in 2016. This is illustrated in Fig. 14, which shows that progress was made especially in the Baltic region (Estonia, Latvia and Lithuania), the Core CWE region (Belgium, France, Germany, Austria, Luxembourg and the Netherlands) and the SWE region (France, Portugal and Spain). This improved price convergence was driven by additional interconnection capacity and the introduction of (flow based) market coupling. In the other regions price convergence remained stable.6 Fig. 14  Day-ahead price convergence in Europe by region (ranked)—2008–2016 6  For a description of each of these regions, we refer to ACER decision No 06/2016 of 17 November 2016, to be found on the website of ACER. 20 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 Further improving price convergence requires a further optimisation of the amount of tradable cross-zonal capacity, especially in those regions where a lower degree of price convergence was observed. This not only requires additional investment in cross-border transmission capacity, but also an optimisation and better coordination across Transmission System Operators of the capacity calculation methods for cross-border capacities available for Trade [8]. Submarkets The wholesale electricity market consists of a variety of interrelated submarkets, such as the forward markets, the day-ahead market, the intraday market and the balancing market. A full integration of all these markets is needed to end up with a well-functioning Integrated Electricity Market. ACER concludes that progress has been made in all of these markets, but also notes that ample room remains for further improvements, both in terms of integration and coordination [4]. Generation adequacy As mentioned in Sect. 2, security of supply is one of the key challenges of European energy policy and therefore also one of the main motivations for European Electricity Market Integration. Interconnecting neighbouring countries not only allows for international trade, but also allows countries to access generation capacity abroad when local generation capacity falls short. In the context of electricity markets, adequacy is understood as the ability to meet changes in present and future aggregate power requirements through timely investments in transmission and generation infrastructure and demand response [13]. Today, in most Member States and the EU as a whole, generation capacity levels are adequate. To a large extent, this is due to the drop in the demand for electricity after the 2008 economic crisis. However, this situation may come to an end in the nearby future for a number of reasons. First, the EU power generation fleet is ageing [13] and needs to be replaced. Second, increased penetration rates of generation plants using intermittent renewable energy create a merit order effect, pushing down prices of electricity on the wholesale electricity market and making investment projects in generation capacity less profitable [5]. Third, climate change policies make fossil fuelled generation plants less profitable to use or to build. While, in theory, a well-functioning competitive electricity market would provide the price signals needed to encourage investment in new generation capacity, most Member States are reluctant to rely on these market signals and have therefore adopted a so-called capacity mechanism to foster local investment in generation capacity [5]. Moreover, these countries tend to do this with a national focus, ignoring—at least partially—available generation capacity in the neighboring countries. As shown in Fig. 15, this has resulted in a patchwork of different capacity mechanisms being applied in the EU. 21 1 3 International Journal of Economic Policy Studies (2019) 13:3–26 Challenges Looking back at the performance of the EU wholesale and retail markets, as described in “Experiences from liberalising the electricity market”, a set of key challenges can be identified that need to be tackled in order to move closer to the objective of an European Integrated Electricity Market. This section identifies and briefly discusses these challenges. Accommodate an increasing share of renewables‑based electricity generation… In “The electricity retail market”, it was mentioned that the increased penetration of renewables is putting a burden on electricity bills as the cost of these renewables subsidies is in most Member States being recovered through a fixed or variable surcharge on the electricity bill. Furthermore, the merit order effect created by this increased penetration of renewables undermines the profitability of existing fossil fuel power plants (“The wholesale market”). Renewable technologies allow electricity generation at very low marginal cost, thereby shifting the merit order to the right. As a consequence, situations of high levels of electricity generation based on renewables, will result in smaller price increases in periods of high demand compared to periods of similar demand when there is no or much less electricity generation from renewables. This is illustrated in Fig. 16. The bold convex line represents Fig. 15  Capacity Mechanisms applied in EU Member States in 2016 22 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 the merit order curve in case of electricity generation based on conventional as well as on renewable resources. The dashed convex line represents the merit order curve when electricity generation from renewables is zero. The two vertical lines represent (inelastic) electricity demand in periods of low and high demand. Clearly, the equilibrium price for electricity is impacted by the availability of electricity generation from renewables. Having a situation where electricity generation is not sufficient to meet electricity demand cannot be accepted. Therefore, as most renewable technologies are intermittent, the need for (flexible) conventional backup capacity remains in order to be able to also cover demand in periods of peak demand and low availability of electricity generation based on renewables. This results in a paradox: the need for flexible (peaking) generation capacity remains, while the profitability of the existing plants can be questioned. The business case for these plants was built upon the anticipated occurrence of sufficiently high prices in periods of peak demand. In general, any electricity generation firm will be willing to invest in new generation capacity only when anticipated revenues are sufficiently high to cover both variable and fixed costs. This requires sufficiently high price levels, but also a sufficiently large number of running hours. Both the price level and the number of running hours are influenced by the merit order effect. …while guaranteeing adequate generation capacity Today, many Member States have overcapacity in the generation equipment. However, this overcapacity is expected to vanish soon as a large share of conventional capacity is close to end-of-life and because the profitability of (mid) peak plants is at risk due to the merit order effect. Fig. 16  Increasing the share of Renewables in electricity generation 23 1 3 International Journal of Economic Policy Studies (2019) 13:3–26 In theory, well-functioning markets signal scarcity in capacity through price increases, which trigger the appropriate level of investment. In practice, however, this price signal does not play its role in the wholesale electricity market as not all assumptions underlying a well-functioning (competitive) market are fulfilled. For one thing, prices do not rise (fast) enough to clear the market. Electricity demand is typically very inelastic, i.e. demand response to price changes is too small (see later). So large price increases are needed to restore equilibrium in the market in cases of high demand or low generation. Furthermore, increased penetration of renewables and the resulting merit order effect also tends to increase price volatility. Typically, these price spikes and price volatility increase the risk of regulatory intervention. Market interventions can take place in different ways. System operators will intervene by using operating reserves, by using non-price rationing or by installing administrative (in most cases, too low) price caps. In most cases, the motivation for these interventions is to protect consumers from high prices. However, at times of scarcity high prices are needed in a market to trigger new investments, so these interventions are detrimental for future investments in generation capacity. In order to solve this problem of too low or missing incentives to invest in generation capacity, many Member States have installed a capacity mechanism. Many different types of capacity mechanisms exist and this has resulted in a wide variety of mechanisms in the EU. This brings us to the next challenge. Harmonize energy policies across Member States The two challenges discussed above have resulted in a patchwork of uncoordinated actions and decisions by the Member States with respect to the capacity mechanism they installed. Also with regard to the choice of the incentive scheme used to meet its renewables target, Member States have taken uncoordinated decisions. Most countries use feed-in-tariffs to foster the use of renewables for electricity generation. A feed-in-tariff is a price based policy instrument. Other countries use tradable green certificates—which is a quantity based instrument—as their main policy instrument. Others still use tendering or a combination of these instruments. For most countries, the chosen policy instrument does not allow for interaction with incentive schemes in other countries. Clearly, this increases the cost for a Member State and for the EU as a whole of meeting its renewables target. But other examples exist where coordination between Member States would be cost efficient and beneficial from a welfare perspective [11]. For example, the organization of intra-day and balancing markets, the decision on the generation mix, or the mechanism to calculate and allocate available cross-border generation capacity. This lack of harmonization and coordination creates inefficiencies, i.e. the same outcome could be achieved at a lower cost. The challenge for the EU is to motivate Member States to look beyond their own stakes and to coordinate policy decisions and actions with the aim of reducing costs and optimizing welfare for the EU as a whole. 24 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 Reinforce cross‑border interconnections As argued in “The wholesale market”, together with harmonizing energy policies, reinforcing grid infrastructure is a prerequisite for a European Integrated Electricity Market. Sufficient cross-border capacity would contribute to solving generation adequacy problems more efficiently, would allow for more cross-border trade (at the generation as well as at the retail level) and would allow to more easily accommodate unscheduled flows, resulting from unexpected variations in electricity production from renewables. The challenge is to motivate Member States to make the proper investments in cross-border transmission capacity and to coordinate efforts on procedures to increase and allocate available cross-border capacities. Increase demand response and introduce more flexibility at the consumer side Finally, most member States have simple (regulated) tariff structures for the household segment and for ‘small business’ segment. However, these single tariff or timeof-use tariff schemes do not reflect wholesale electricity prices [7]. These latter prices can vary significantly hour by hour, but the (at best time-of-use) end-user tariffs do not depend on the time of day, the day, or the season to which they apply. As a consequence, this lack of variation in retail prices makes demand appear inelastic. Demand is not responsive to price changes because prices don’t change. This is an important issue for at least two reasons. First, prices cannot signal scarcity and stimulate efficient consumption. Second, a well functioning price system would make that less capacity is needed to cover peak demand (peak demand decreases). The challenge faced by the EU and its Member States is to increase demand response. One way to do this is by installing smart metering infrastructure in combination with pricing mechanisms closer to real-time pricing. The roll-out of smart metering equipment is on its way. It is expected that by 2020 about 70% of EU citizens will have a smart meter installed [13]. Here also, some challenges remain, as experience in markets that are at the forefront of the roll-out learns that there is a need for independent data and information management, for guaranteed privacy, for an accurate definition of the roles of suppliers and distribution operators, for the transparency of market data for all consumers, and for tariff schemes close to real-time pricing. The benefits of using real-time-pricing have been shown by Borenstein and Holland [6]. In addition, a paradigm shift is needed in the sense that the future energy system will have to move away from a conventional, more centralized energy system towards a more decentralized system that not only allows for demand response, but also for distributed generation and storage [19]. This will create many challenges and issues on its own, as discussed in Pepermans et al. [18]. 25 1 3 International Journal of Economic Policy Studies (2019) 13:3–26 Conclusion The liberalization of the electricity market was initiated more than 20 years ago. Since then one has moved a long way and made a lot of progress, also in areas that were not discussed in depth in this paper, such as transmission and regulation. However, today, the EU is still facing some important challenges. At the level of retail markets, many Member States need to make additional efforts to increase retail competition and to remove regulated retail prices. Moving closer to an Integrated European Electricity Market also requires more focus on the physical integration of existing regional electricity markets, by removing the existing barriers between regions and countries. Building new interconnection capacity and improving the allocation procedures for cross-border capacities can help achieving this. Furthermore, the EU should work on the harmonization and integration of several existing electricity markets. The EU should also encourage the demand-side of the market to participate more intensively through demand response. Further harmonization is also needed in other areas, such as fostering renewables and generation adequacy. References 1. ACER (2015). Annual report on the results of monitoring the internal electricity and natural gas markets in 2014 (p. 329). Luxembourg: Agency for the Cooperation of Energy Regulators. 2. ACER (2016). Annual report on the results of monitoring the internal electricity and gas markets in 2015. Retail Markets (p. 83). Luxembourg: Agency for the Cooperation of Energy Regulators. 3. ACER (2017a). Annual report on the results of monitoring the internal electricity and gas markets in 2016. Electricity and gas retail markets volume (p. 24). Luxembourg: Agency for the Cooperation of Energy Regulators. 4. ACER (2017b). Annual report on the results of monitoring the internal electricity and gas markets in 2016. Electricity wholesale markets volume (p. 78). Luxembourg: Agency for the Cooperation of Energy Regulators. 5. Auer, H. and Burgholzer, B. (2015). Opportunities, challenges and risks for res-E deployment in a fully integrated european electricity market: Market4RES (p. 71). 6. Borenstein, S., & Holland, S. (2005). On the efficiency of competitive electricity markets with timeinvariant retail prices. The Rand Journal of Economics, 36(3), 469–493. 7. Crampes, C., & Waddams, C. (2017). Empowering electricity consumers in retail and wholesale markets (p. 44): CERRE Report. 8. De Jong, J., Hassel, A., Jansen, J., Egenhofer, C., & Xu, Z. (2017). Improving the market for flexibility in the electricity sector. Brussels: Centre for European Policy Studies. 9. European Commission. (2016). Clean energy for all Europeans. Brussels: COM(2016) 860 final. 10. European Commission. (2016). Energy prices and costs in Europe (p. 20). Brussels: COM(2016) 769 final: European Commission. 11. Glachant, J.-M., & Ruester, S. (2014). The Eu internal electricity market: done forever? Utilities Policy, 31, 221–228. 12. Gouardères, F., Mcwatt, V., & Fleuret, L. (2018). Internal energy market (p. 5). Brussels: European Parliament. 13. IEA. (2014). Energy Policies of Iea Countries: European Union: Organisation for economic cooperation and development/organisation for economic co-operation and development (p. 312). 14. Ioannidou, M. (2018). Effective paths for consumer empowerment and protection in retail energy markets. Journal of Consumer Policy, 41(2), 135–157. 26 International Journal of Economic Policy Studies (2019) 13:3–26 1 3 15. IPA Advisory LIMITED. (2015). Ranking the competitiveness of retail electricity and gas markets: A proposed methodology (p. 95). London: IPA Advisory LIMITED. 16. Kanellakis, M., Martinopoulos, G., & Zachariadis, T. (2013). European energy policy—A review. Energy Policy, 62(Supplement C), 1020–1030. 17. Langsdorf, S. (2011). Eu energy policy: From the Ecsc to the energy roadmap 2050 (p. 9). Brussels: Green European Foundation. 18. Pepermans, G., Driesen, J., Haeseldonckx, D., Belmans, R., & D’Haeseleer, W. (2005). Distributed generation: Definition, benefits and issues. Energy Policy, 33(6), 787–798. 19. Poudineh, R., & Jamasb, T. (2014). Distributed generation, storage, demand response and energy efficiency as alternatives to grid capacity enhancement. Energy Policy, 67, 222–231.