FJL Boerman Blog

Year in review 2023: Flowbased CORE Individual Validations

Tue, 06 Feb 2024 00:00:00 +0000

Another year is over, and I wanted to write a ‘year in review’ series about the electricy system in 2023! First up is a look at how different TSOs adjust the Flowbased Domain for grid security reasons, the Individual Validations or IVA for short.

But first, a quick shout out to the sponsor of my database server: Code Yellow! Code Yellow has provided the server that supports my flowbased domain database. Without them the analysis in this post would be a lot more work. Many thanks!

Please note that this post is quite technical and about a specific capacity calculation topic. I tried to explain it to non experts as well, but you do need a basic idea of the flowbased domain.
This post is also specifically about the validation results of Flowbased CORE. The Nordic Flowbased has yet to implement a validation step.

What are virtual capacities and individual validations

Let’s start with an introduction to the topic. Virtual capacity is the capacity that TSOs release on top of the capacity calculated in the normal grid calculation. This is done to reach the regulatory minimum capacity requirement. This requirement is also informally called the “70% rule”.

Ever since the introduction of these virtual capacities, the challenge arose on how to ensure grid security for this virtual part. To tackle this, the validation step was implemented into the Flowbased Capacity Calculation methodology.

Validation allows TSOs to reduce capacity on elements (CNECs) in the domain if and only if TSO’s demonstrate that the capacity calculated cannot be secured even when all available remedial actions (RA) are activated. RA’s are usually forms of redispatch.

The adjustment of a CNEC is called Individual Validation Adjustment (IVA). The idea behind this adjustment is that the non-virtual part you get for free. However, if the market uses the virtual part, it incurs a TSO (and thus societal) cost. This then spurs grid investments to make sure you can reach the target capacity with as little virtual capacity as possible.

Process wise, the validation step is one of the last steps in Flowbased. The last being the addition of LTnom flows. The validation is called individual and thus is in principle run by local tooling of individual TSOs. The exception to this will be explained later.
Stepwise the process to the final capacity then becomes as follows:

Calculate real capacity ($RAM_{init}$)
Add virtual capacity until regulatory target ($AMR$)
Execute validation and subtract IVAs if needed ($IVA$)

So, the final capacity then becomes: $RAM_{final}=RAM_{init}+AMR-IVA$
Please note that this is a simplified picture of the whole flowbased process to explain the IVA flow.

With the rising of regulatory targets, this validation of virtual capacities is becoming increasingly important. For more information about these rising targets, you can read for example the Dutch action plan about this here.

Remarks on the data

Dataset used includes the entire of 2023, with a total of 8760 hours. On two days in 2023, Elia had an exceptional unintended fallback causing a very distorted picture on IVAs. For this reason, Elia IVAs for the full day of 2023-04-19, as well as, for hour 11, 12 and 13 of 2023-08-21 have been filtered out. More information about the first day can be found in my post here.

Now that the topic and data remarks are introduced, let’s get into the analysis.

IVAs in 2023

To start, figure 1 below shows how many hours were impacted by IVAs in 2023.

Figure 1: Overview 2023 IVAs

Here we can see that in only 36% of the hours, no IVA was applied on presolved or active constraints. This percentage of hours seems high. However, there are big differences between how much IVA and on how many constraints they were applied in those hours. Furthermore, not every TSO applies the same methodology, since its a local responsibility. Let us have a look at differences between TSOs.

Differences between TSOs: methodologies

There are some important differences in methodologies applied. First off the most confusing part of this story: not all Individual Validations are actually done Individual. The TSOs of Germany, Austria and the Netherlands decided to work together, using one tool to calculate IVAs for all three countries, this tool is called DAVinCy. This pooling allows the application of cross border IVAs. For example, an IVA on German CNECs to solve an overload in NL. From a social welfare perspective this is more efficient.
In fact pooling the whole region would be more efficient and this is also forseen in the methodology, the so called Common Validation Adjustment or CVA. This is not implemented yet however, but is actively being worked on.

As mentioned before, each TSO is responsible for their own validation process. Except for the common DAVinCy group, all CORE TSOs have developed their own methodologies. Explaining each of them is outside the scope of this post, instead, we focus here on the trend differences among TSOs.
Detailed explanations of these methodologies per TSO (including the DAVinCy group) can be found in the appendix of CORE Consultative Group slides of 2023-03-28, starting from slide 41 here.

IVA trends between TSOs: number of occurrences

First, we determine the number of hours in which a TSO had at least one active IVA in the presolved domain. Figure 2 below shows both the number of hours and the percentage of those hours in 2023.

Figure 2: Number of hours with IVA per TSO for presolved domain of 2023

Here we can see that there is a clear trend difference between the top 3 TSOs, namely RTE, SEPS and Transelectrica, and the other TSOs. These three TSOs applied IVAs in many more hours during 2023, with RTE applying in almost 1/3 of all hours.

In addition, a second trend is noticeable when IVAs per hours are split by months (figure 3 below).

Figure 3: Number of hours with IVA per TSO per month of 2023 (presolved domain)

Here we can see that Transelectrica applies more IVAs at the first half of the year, while RTE applied more in the second half of the year. All the other TSOs have fewer clear trends, with variations throughout the year.

IVA trends between TSOs: sizes

The same analysis can be made with sizes of IVAs. To make the comparison between CNECs fair, results are expressed as a % of the maximum thermal flow on an element (Fmax). Please note that due to negative reference flows it is possible to have an IVA which is > 100% of Fmax. This is a fairly uncommon case. In 2023 only the DAVinCy group applied such IVAs.
All presolved domain IVAs are shown in a boxplot in figure 4 below. Each dot next to the boxplot is a datapoint in the set.

Figure 4: Average IVA as % of Fmax per TSO over whole of 2023 (presolved domain) Hover to see quartiles

Results show a clear differences on percentage of IVAs applied. CEPS, Mavir, DAVinCy and RTE apply fairly large IVA’s. CEPS has a median of almost 40% of Fmax. However, interpretation requires caution since the median is based on very few applications. Mavir, DAVinCy and RTE have a median of around 20%, with DAVinCy showing large spread outside of their box. Elia shows this spread too. However, their median is low. We see again that there is a group of TSOs which are scoring higher than the rest on this indicator.

This indicator too can be split out per month, as shown in figure 5 below.

Figure 5: Average IVA as % of Fmax per TSO per month of 2023 (presolved domain)

Here no clear trend can be seen. The only thing that stands out, is that the high spread of Elia is apparently caused by IVA spikes in February and August.

Conclusion of trends

Is there an overall trend to be concluded?
Results clearly showed that there are major differences on when and how much IVAs are applied across TSOs. Their statistics are summarized in table 1 below. This is the table form of figure 2 and 4.

TSO	#CNECs	mean IVA (%Fmax)	min IVA (%Fmax)	median IVA (%Fmax)	max IVA (%Fmax)	#hours	% hours of year
CEPS	10.0	40.23	27.09	38.92	51.66	10	0.11
DAVinCy	1733.0	28.45	0.05	19.96	142.62	218	2.49
ELES	198.0	5.18	0.09	4.46	20.56	155	1.77
ELIA	397.0	17.10	0.60	6.17	99.06	277	3.16
HOPS	194.0	4.95	0.08	3.65	30.21	181	2.07
MAVIR	408.0	16.75	0.29	16.46	55.16	249	2.84
PSE	700.0	10.30	0.07	6.21	36.19	339	3.87
RTE	2949.0	22.24	0.03	21.72	55.38	2450	27.97
SEPS	3330.0	6.43	0.07	5.28	30.01	2232	25.48
TRANSELECTRICA	3602.0	10.51	0.08	8.75	73.40	1804	20.59

Table 1: statistics overview of CORE Flowbased IVAs in 2023 per TSO

From table 1, the TSOs can be categorized along the axis of number hours and the median IVA (expressed as % of Fmax). This is plotted in figure 6 below. CEPS is excluded from this graph due to its outlier character, showing few hours and a high median IVA %.

Figure 6: Summary of TSO trends for IVAs in 2023 (CEPS excluded as outlier)

From figure 6 four catagories of trends can be concluded:

RTE has a high number of hours with also a high median IVA value
Transelectrica and SEPS have a high number of hours but low median IVA value
ELES, HOPS, Elia and PSE have lower number of hours and low median IVA value
DAVinCy group, MAVIR and CEPS (last one not shown) have low number of hours but high median IVA

There are a couple of important things to note from this trend conclusion:

there is no right or wrong here. Individual validation is a tailored process to each grid. There are multiple ways to handle the validation and it is the individual responsibility of each TSO to apply those.
there may a bias on the dataset that benefits larger TSOs. Larger TSOs have more CNECs and have thus a larger chance of being presolved and needing to apply IVAs.

This concludes the first post on the “year in review” series. More to follow later!

If you have any questions or suggestions feel free to reach out to me on linkedin, twitter (on either search with my real name) or via email at frank@fboerman.nl.

Disclaimer: all information described in this post is meant for informational purpose and written by an expert working on the topic. No information neither any conclusion written in this post can be taken as an official opinion or position of my employer TenneT in any way, shape or form.

Flowbased: update 2 about the the Nordic Flowbased parallel run results

Mon, 16 Oct 2023 00:00:00 +0000

Recently, it was announced that the Nordic Flowbased go-live was delayed due to IT issues. While this is obviously a pity to hear, other positive things have been announced.
The evaluation report from NRAs was published. It was evaluated and found sufficient by the NRAs, although with some extra remarks.
A very interesting summary containing issues found and lessons learned, was published alongside the report. In there a point-by-point analysis is done for specific issues that were encountered during the parallel run. A very interesting read for experts!

I want to highlight one important issue, the modelling between SE2 and SE3. The price spread on that border was puzzling high when compared to the NTC historical results in the beginning of the parallel run. This was apparently explained by a modelling mistake, which was presented on march 27th in the stakeholder meeting. Slides can be found online.
In figure 1 below a slide from that meeting explaining this issue is shown.

Figure 1: slide 8 stakeholder meeting 2023-03-27 showing the modelling issues on SE-SE3 border

In summary: this modelling mistake made less capacity on the SE2-SE3 border available in Flowbased, when compared to the historical situation of NTC. This was rectified from week 9 (2023-02-27) onwards, which should decrease the price spread on this border in the parallel run.

Remarks on the dataset

In this post I will look at the simulated market results. As a recap, market results are resulting from simulations with historical orders. However this is done with the new Flowbased domain from the parallel run. From these simulations, the prices and net positions are then published.
I have chosen the starting point of the dataset for this post to be 2023-02-27, as this is when the SE2-SE3 mistake was fixed. This cut-off point still leaves a large set with both winter and summer days. There are 5039 hours between 2023-02-27 and the last simulated day in the external parallel run, which is 2023-09-24 (as of the time of writing this post).
The simulation results can be downloaded from Nordic RCC. This dataset contains 5093 hours, therefor all hours have been successfully simulated.

Market Simulation Results: Prices

So, let’s look at the main indicators, starting with prices.
We begin with the simplest visualization: the average shift in prices per bidding zone. Negative values mean a lower price in the simulation compared to the historical result. On the contrary, positive values indicate higher prices. This is shown in a map form in figure 2 below.

Figure 2: Average difference between simulated prices and production prices between 2023-02-27 and 2023-09-24,over all hours. Negative number means a lower price on average in the simulations. All in EUR/MWh (figure is interactive)

Now, while using absolute prices is interesting for showing which prices goes up and down, it is important to interpret those prices in context. In figure 3, the same data is shown. However, this time, percentages of the mean price per zone are used.

Figure 3: Average difference between simulated prices and production prices between 2023-02-27 and 2023-09-24, over all hours, as a percentage of the mean price of the zone. (figure is interactive)

Here you can see that an impact of a couple of euros on SE2 is relatively much higher then the couple of euros euros change on NO2.
From both figures, it can be seen that the price in Sweden and northen Norway has mainly moved up, while the price in South Norway and Denmark has dropped. Especially within Norway, there is a sharp divide. In general, it seems that northern zones have increased prices while southern ones have dropping prices.
Why is that the case?

Market Simulation Results: Net Positions

To answer this, we will take a look at a different key indicator: the net positions.

A net position is the summed import/export over all borders of a zone. Here, conventionally, negative NP means import, and positive NP means export.
In figure 4 below, the average change in hourly NP per zone is plotted, as percentage of the mean of historical NP. In table 1 the data underlying this map is shown.

Figure 4: Average difference between simulated Net Position and production Net Position between 2023-02-27 and 2023-09-24 over all hours, as a percentage of the mean NP of the zone. Negative number means shift into import direction. (figure is interactive)

Here we can see a correlation between the price shifts in figure 3 and the NP shifts in figure 4. Cheaper zones import more, while more expensive zones export more.
This correlation, between prices and netto position shifts, makes sense because Flowbased has made it possible to shift more power from a historical cheap zone (thus, increasing the prices and the export) to a historical more expensive zone (thus, decreasing prices and increasing import). This is one of the core goals of improved capacity calculation: decreasing of price spreads. In effect, the prices “move” towards eachother. Figure 3&4 thus show that Flowbased is doing what it is designed for!

The data also shows extreme percentages on there. For instance 200% for Finland and 100% for south of Norway. Why are these so extreme?

Zone	NP simulation - historical	mean NP historical	difference in %
FI	97.69	-49.19	198.61
NO1	14.67	-544.50	2.69
NO2	-1605.06	1539.74	-104.24
NO3	184.29	-332.19	55.48
NO4	156.56	890.78	17.58
NO5	-223.13	1373.13	-16.25
DK1	-2.59	456.14	-0.57
DK2	2.84	-376.82	0.75
SE1	284.52	1329.79	21.40
SE2	183.04	3387.44	5.40
SE3	95.37	-485.42	19.65
SE4	-1.02	-1250.66	-0.08

Table 1: Change in Net Position per zone tabulated

Here we can see that for extreme zones, such as FI and NO2, the NP has a large difference compared with historical results, while the zone had historically a small net position. So Finland for example became 100MW more exporting, while at first it was importing on average 50MW. This results in a large relative number.

Price Spreads

Now, we go back to the last indicator: price spreads. There are two ways of looking at it: the price spread on a specific border or the price spread between the highest and lowest price within the region, both per hour.

We start with the borders. The average price spread change per border is shown in table 2. Here, a positive price means price spread went up and a negative price means price spread went down.

border	change avg EUR/MWh	border	change avg EUR/MWh
DK1-SE3	-12.77	DK2-SE4	1.25
NO1-SE3	-11.11	FI-NO4	1.35
SE3-SE4	-10.32	DK1-NL	1.79
NO3-NO5	-10.17	FI-SE3	3.38
NO1-NO3	-8.52	SE1-SE2	3.46
NO1-NO2	-6.90	SE1-NO4	4.04
DK1-DK2	-3.33	SE2-SE3	5.28
FI-SE1	-2.08	NO4-SE2	5.34
NO2-NO5	-1.50	NO1-NO5	6.79
		NO3-NO4	8.97
		NO3-SE2	9.55

Table 2: Change in price spread per border in EUR/MWh

Here we see a different picture than in the previous update post. The shifting spreads seem smaller but most importantly the spread shift on SE2-SE3 is now a lot smaller. This confirms that the change in net modelling had a positive impact. There is still an increase in price spread. However, this is to be expected when flows get attracted by the south of Norway and Denmark, where a bigger welfare impact can be realized.

Lastly we will look at the price spread in the region. Price spread in the region is defined as the difference between the maximum and minimum price for the whole region per hour. In figure 5 this price spread in the region is shown, together with a comparison with production value. The blue bars signal the difference of simulation vs production. Thus, a negative value represents a decrease in the spread for that hour.
For an interactive dashboard see here

Figure 5: Price spread in Nordic region for both simulated and production values, for the set used in this post. All in EUR/MWh (live dashboard embed, interactive)

On average, the regional price spread drops by 2.38 EUR/MWh when taking the whole dataset into account. This corresponds to a drop of 3.24% relative to the average price spread in the historical data set. In 57% of all hours the price spread has decreased compared to the historical data. This result comes from a rather large dataset of 5039 hours (or almost 60% of an entire year) which has seen various checks and fixes already. Overall I would call this a good and encouraging result for the project.

As a last point we could ask if these results are consistent throughout all hours of a business day.To check this the average change in relative price spread is plotted in figure 6 below.

Figure 6: Relative change in regional price spread, averaged by hour of the day over the whole dataset (figure is interactive)

From this graph we can conclude that, while not evenly distributed, every hour of the day sees, on average, some decrease in price spread. A good result for the improved capacity calculation!

What is next

What is next for Nordic Flowbased? There were some recommendations from the NRAs, to which the TSOs will probably respond. And the new go-live schedule is set to be released somewhere in November 2023.
I hope the go live will be soon, so we can also see the effect of these changes on trader’s behaviors! Once there is sufficient new information, I will probably write a new post.

DA market: the benefits of the Day-Ahead market design for NL

Mon, 24 Jul 2023 00:00:00 +0000

Recently a well known participant in the dutch energy debate, lector Martien Visser, wrote tweets and a column setting out his view that the current day ahead market design is not a good business case for NL anymore. He is not the only one, this sentiment is widespread on social media. While I do agree that in a RES dominated system it could be possible to have a better tweaked market mechanism, I disagree that the current one is a very bad one, especially for NL. In this post I present some figures to support that.

SDAC net revenue NL

A couple months ago, my colleague Joost Greunsven proposed a nice way to visualize the revenue of all SDAC border exchanges for NL detailed here. This is a scatter plot in which each hour of the year is represented as a dot on a scale of import/export vs the revenue (sum of all border flows x DA price). I created a webapp to display such figures for more zones which can be found here. In figure 1 below, the figure of Joost is repeated for NL for the first half of 2023.

Figure 1: the net revenue for NL for all hours in first half of 2023

This plot can be broken down in four quadrants. The top left (quadrant I) is import with negative price, the top right (quadrant II) is export with positive price, the bottom left (quadrant III) is import with positive price and the bottom right (quadrant IV) export with negative price. Quadrant II and III is what most people would feel as “normal” and are the ones that are by far the most common. However, quadrant I and IV can also happen due to the fact that the market optimizes for the whole region and not only a single zone. For example, it can be beneficial for the whole region if some zones export with negative prices.

The core argument from opponents of the current market design is that with high renewables infeed quadrant IV is getting so large it’s a net loss for the Netherlands. When power is exported for a negative price, NL pays for its power export, effectively subsidizing its neighbours. This tends to happen when large amounts of subsidized power floods on the market. In NL that is mostly solar power. Subsidies push the power price to be negative and if there is a large surplus this is then exported.

The title of figure 1 already shows that the overall business case for NL is still quite positive. The sum of net revenue is a positive 434.4 Million Euros. In figure 2 below a breakdown per quadrant is shown in a waterfall chart.

Figure 2: Breakdown of net revenue of SDAC for NL in first half of 2023

Here we can clearly see that quadrant IV is actually quite small compared to the big export and import with positive price quadrants. In fact when expressed in relative numbers quadrant IV is only 2.7% of the total absolute sum of all revenue points. This constitutes only 2.9% of all hours in the first half of 2023.
This is by no means a historical fluke. In fact, this is an all time high since it started occuring in the dutch market in 2020, but it is still very small. In figure 3 the past 5 years net revenue is plotted with the total revenue of exporting with negative price next to it.

Figure 3: Total net revenue vs Total revenue export with negative price for NL over the past 5 years

Here it can clearly be seen that this part has always been small. In addition, for the past four years NL has seen a positive net revenue from SDAC interactions. The year 2022 is exceptionally high here due to the extreme high prices that year with the gas crisis. Based on this, I would argue that the current market design is still very beneficial for the dutch generators.

Now the argument is sometimes made that this revenue loss is mainly on the important hours of solar PV and that NL mainly makes a profit with its gas turbines at night. Figure 4 below shows the summed revenue in NL per hour of the day.

Figure 4: Total net revenue per hour of the day for NL in first half of 2023

Figure 4 clearly shows that NL actually makes the most positive revenue around midday when solar power peaks. There is even no hour which has a negative sum.

SDAC revenue abroad

All that export also means that some zones are net importers. A good example of this is our big neighbour Germany. It used to have a positive net revenue, but this year it is actually importing more. This results in the breakdown shown in figure 5. But even for DE it is sometimes optimal that it exports with negative prices. Quadrant IV is however relatively smaller then NL, representing 0.47% of total absolute revenue.

More on SDAC revenue for other zones in a later post!

Figure 5: Breakdown of net revenue of SDAC for DE in first half of 2023

Flowbased: One year of CORE: A look at the price convergence since go live

Thu, 08 Jun 2023 00:00:00 +0000

A full year ago the Flowbased capacity calculation methodology went live in the CORE region. To mark this milestone and celebrate the anniversary, this post will look at the process stability and price convergence of Flowbased.
Minimizing the price spread, or in other words maximizing the price convergence, is one of the goals of a more efficient capacity calculation in the region. At the time that full price convergence (e.g. a price spread of zero) is achieved, the market did not hit any constraint in the capacity domain.
The go live of FB CORE at business day 2022-06-09 coincided with an ongoing energy crisis in Europe, one of the worst in recent memory. I thought it would be interesting to take a look at how the price spread in the region held up in those times.

Definitions and data remarks

Lets start off with a definition: the “hourly price spread” is defined as the difference between the maximum and minimum price within one hour within a certain region on the Day Ahead market.
In this post I focus on the CORE region, however I have excluded Poland from all analysis. The reason for this is that Poland has been applying heavy additional restrictions on its import and export of electricity in the past months. The polish TSO PSE says they must conserve coal fuel due to fuel shortages and need these restrictions for operational security. This has resulted in vastly different prices compared to the rest of the region. As you can imagine this is not related to flowbased. For more data on this please view my dashboard on it.

Excluding Poland, the CORE region left is thus defined as the bidding zones of: Austria (AT), Belgium (BE), Czech Republic (CZ), Germany/Luxemburg (DE_LU), France (FR), Croatia (HR), Hungary (HU), Netherlands (NL), Romenia (RO), Slovenia (SI) and Slovakia (SK).
All data is based on the day ahead prices published on the ENTSOE transparency platform. I have used data from business day 2022-06-09 up to 2023-05-31. The dataset constitutes 8568 timestamps, which means there are no hours missing in that period.

Process Stability

The Flowbased process transparently publishes whether the calculations were successful for all hours of a given business day. When a calculation fails there are two backup options to deal with it: Spanning and Default Parameters.
Spanning means that a mathematical average is chosen between the results of the hours before and after the failure. This then bridges the data gap. This is only applicable if that average is feasible and the gap is not too many consecutive hours.
Default parameters are a set of minimum capacity values that are safe to be released to the market for trading, in terms of net security. These are a kind of fallback to apply when all other efforts failed.
Since go live, only 13 hours failed in their final calculation. Of those 13, eight resulted in default parameters and 5 were spanning.
That is 0.15% of the total number of hours. A very good result for process stability after its first year!

DA Prices

The DA prices in this period were rather extreme, both in their height and volatility. In figure 1 below the prices for CORE are plotted. This is from my generic live dashboard which does include Poland.

Figure 1: DA prices for CORE region, including Poland (live dashboard embed, interactive)

The figure is very busy, thus lets zoom out a bit. In figure 2 below, the mean of the whole region (excluding Poland) is shown per month. Especially in July, August and September of 2022 and again half way through December and early January, the prices exploded. This is explained by the European gas crisis, something that has already been extensively covered by others.

Figure 2: Average DA price per month in CORE region, excluding Poland (figure is interactive)

Still, sometimes the prices also plummeted in some zones. This is most likely due to high infeed of cheap renewables. In 3.3% of all timestamps (282 hours) at least one zone had a price below 1 EUR/MWh and in 2.3% of all timestamps (200 hours) it was even negative in at least one zone!
Luckily, the prices have dropped due to the easing of the gas crisis. Nonetheless, historically the prices keep being high. As a reminder, in the first five months of 2021 the average prices were 56, 49, 52, 60 and 57 EUR/MWh respectively, compared to 136, 142, 110, 103 and 83 EUR/MWh respectively for the first five months of 2023.

Price Spreads

Prices in and of itself are interesting, however they are not really relevant to flowbased. What is more relevant, is the hourly price spread. As explained earlier, price spread is defined as the delta between the max and min price within an hour, within a region.
First lets take the monthly average again. This time, for absolute price spreads in EUR/MWh. This is displayed in figure 4 below.

Figure 4: Average DA price spread per month in CORE region, excluding Poland (figure is interactive)

The month of July has the largest absolute spread on average, while the months after it are in the 70-90 EUR/MWh spread range. The last three months are even in the 20-30 range. This is interesting since the mean of the overall prices swung much more. This can probably partly be explained by renewable infeed. July is generally more sunny and therefore cheap solar power is available in large abundance. However the installed generation capacity of renewables is not equally distributed across all zones. For example, the Netherlands, which has much more installed solar generation capacity than other CORE countries. The grid becomes strained trying to move all that solar energy to other zones, thus increasing price spreads.
This theory is reinforced by Figure 5, which shows a heat map of the price spread. To keep the figure readable the extremes are cut off at 200 EUR/MWh of price spread.

Figure 5: Price spread in CORE region as heat map

Here we can see that in July 2022 (but also a bit later) the highest price spreads are in the middle of the day. These are typically the hours that solar energy is the most active. Also, in the last two months of 2022 the afternoon hours have higher price spread when the sun is picking up again. These do have a smaller magnitude due to lower prices overall compared to summer of ‘22.

Price Spreads Performance

So how do these price spreads play a role in the bigger picture? In figure 6 below, the absolute hourly price spread is binned in categories. Each category shows the percentage of timestamps that fall in it, out of the total set.

Figure 6: Absolute hourly price spreads in CORE region excluding Poland, binned in categories.

Here we can see that in 26.98% of the hours there was full price convergence (bar A), so a price spread of 0. For a large area such as the CORE one, in crisis times, that’s quite impressive!
In almost half of the time, 49.35%, the price spread was less than 100 EUR/MWh (split out over bar B, C and D). Although that doesn’t sound that much in these times, it is quite context dependent. A 100 EUR/MWh spread between 10 and 110 is much more impactful than, for instance, between 700 and 800 EUR/MWh.

To better look at this indicator in this context, we introduce a new indicator C%. This indicator is then defined as the “hourly price spread as expressed as a percentage of the mean price in a determined hour”. Or more formally:

$$ C_{\%}(t) = \dfrac{\overline{P_{spread}(t)}}{\overline{P(t)}}\cdot 100[\%] $$ in which the $\overline{bar}$ denotes the average over all CORE zones excluding PL at timestamp $t$

Now lets revisit the above example for a simple situation of two zones. A 10-110 spread is then expressed as a C% of 167%, while for the 700-800 spread the C% comes down to a mere 13.3%.

Next, we will bin the resulting numbers over the whole dataset. The result is shown in figure 7 below.

Figure 7: Hourly price spreads in CORE region excluding Poland, relative to hourly mean price, binned in categories.

Full convergence (or $C\%=0$) is still 26.98% of the total time. However, now we can also see that in 28.59% of the total time the price spread is between 0 and 25% of the mean price in an hour! Combined this means that in a little over half of the time (55.57%) the price spreads are less than a quarter of the mean price. Within the context of the extreme market conditions, this seems a quite good result!

An even better benchmark would be to compare this with a non flowbased approach. Unfortunately, this is not possible because this data does not exist anymore in the CORE region. Comparing with another flowbased region is also not possible yet, the nordic region is implementing flowbased but this is not done yet. Perhaps in the future it would be interesting to see how flowbased performs in terms of price spreads, in this different region and context. But for that we may have to wait at least another year.

So within the data we can see now, the system seems to perform very well. It will be interesting to see how the full (or almost full) price convergence share holds up the longer flowbased CORE marketcoupling runs.

Bonus Annex

As a bonus, to check whether our earlier assumption about price spreads going up in sunny hours holds, we can also look at the C% indicator, averaged per hour of the day. This is shown in figure 8 below.

Figure 8: Hourly price spreads relative to hourly mean price, binned per hour of the day

Here we can see a very large peak at hours 13 and 14, which typically is the peak solar hour of the day. The effect on hour 12 is less strong, but still visible. This indeed seems to confirm the theory that price spread is influenced by solar infeed.

Aten: pausing a wind park for bird migrations, a worlds first in the Netherlands

Mon, 15 May 2023 00:00:00 +0000

On the 11th of May a new kind of operational message was published on the TenneT NL feed. Normally this feed is quite boring, announcing transport restrictions or grid status for example. But on Friday it announced a special type of restriction for offshore windparks. To help large scale bird migrations the offshore windparks were restricted to very slow turning of turbines.
The discussion whether (sea)birds get killed en masse by (offshore) wind turbines is a long one. As far as I know there is no hard scientific evidence either way, but out of extra caution the Netherlands has implemented a process to help birds anyway.
It tries to detect and predict large bird migrations in the spring and fall over the north sea and then schdules a stop of the turbines in a controlled manner for a period of time. That way everybody involved, including TenneT for grid stability, can prepare for it.

On monday an official message from the ministry was published (dutch only) and some dutch media picked it up as well.

So can we see the effects of the restriction in some public data? Fortunately yes. In figure 1 below the output of Wind Offshore for the Netherlands is shown for the night of the restriction. All data is from co2monitor.nl.

Figure 1: Wind Offshore output NL showing the effect of the bird curtailment process

The restriction was scheduled between 00:00 and 04:00 on the night of Saturday 2023-05-13 local time. In figure 1 we can indeed see a clear dip between those hours. The wind speed in this period was pretty constant and the day ahead market prices was as well, so this is most probably fully caused by the restriction.
Please note that the restriction is to lower the speed of the turbines a very slow rate, but not zero. So there is still a bit of generation from the parks affected, as well as some from smaller offshore parks not affected.

To see the effect even better the change for each 15 min datapoint is shown in figure 2 below.

Figure 2: Wind Offshore output NL delta per 15 min, showing the effect of the bird curtailment process

Here we can see the largest deltas at 00:00 and 04:00, with a quite large delta as well. This shows even more clearly the restriction during its cheduled time period.

Day Ahead Market adventures: capacity restrictions and huge negative prices for business day 2023-04-19

Sun, 23 Apr 2023 00:00:00 +0000

The day ahead market has fascinated me since the very first day that I started working in the sector. Many components working together to form a safe and efficient market for electricy trading. On the 18th of April this year in some of these components rare events happened. In this post, I will offer an explaination on what happened.

The Day Ahead market: a description and some definitions

In the Day Ahead market every day electricity is traded. The traded energy is then delivered the day after trading. This explains the name, you trade one day ahead. We call this first day the trading day and the second day the business day. For the case with the extreme negative prices in this post 2023-04-18 is thus the trading day and 2023-04-19 is the business day.
In the EU trading is optimized over a large area called the Single Day-ahead Coupling or SDAC, more info on this can be found on the entsoe website. The market algorithm which determines the prices looks at this whole region to optimize trades. Prices from the Day Ahead (DA) market are generally refered to as the dynamic wholesale prices and differ per hour. Normally household consumers are not exposed directly to these prices, but instead are insulated through various financial instruments. However recently having a dynamic tarrif in your contract based on DA prices has become more popular. Whether that is a good idea is situation dependant, my colleague Jesse van Elteren has written about this recently.
The SDAC area is divided in bidding zones, usually one per country, in which eletricity can be freely traded. So the interactions that are determined by the algorithm are the import/exports from zone to zone, since within the zone there are no trading restrictions.

The process of getting to the final DA prices, the so called market coupling, is broadly divided in the following steps:

TSOs calculate how much capacity the grid can transport safely for every hour of the day. This is called the capacity domain. There are multiple subregions within the SDAC which each have their own domain
The domains are published and traders can put in bids. Traders base these bids on all kinds of information, the capacity domains probably being one of them.
The prices per zone are calculated by the market algorithm which combines all information about capacity domain and trader bids together. The algorithm called EUPHEMIA matches supply and demand in each zone and determines import/export between the zones. It does this by making a global optimization of welfare. Specifically it finds the set of interactions (import/export) between zones, in which the so called global welfare is the maximum out of all options. Global welfare here means the sum of both producers and consumers benefit from cross border trades.

After the last step above is completed the market is called cleared and all traders receive information about whether their bids are accepted and the prices per zone are published.

Negative prices in the Netherlands and Belgium

On trading day 2023-04-18 the resulting prices from the DA market were quite volatile and for NL collapsed in the afternoon. See figure 1 for DA prices for NL and her neighbours.

Figure 1: DA prices for 2023-04-18 and 2023-04-19 for NL, BE and DE

Negative prices are pretty non intuitive. It is countra-intuitive to be paid to consume energy. However, it is quite logical if you keep in mind that electricity consumption and production always needs to be in balance. Some generators cannot switch off for one hour and then start up the next hour, so they keep running. Their electricity needs to be consumed so they rather pay for this to happen then to shut down. Other production sources have even less flexibility.
Negative prices do signal that there is too much production for the consumption present. Consumption will happen but the fact that they want to be paid shows that they normally would not want to do that. In other words, it shows that for specific hours there is not enough consumption for the amount of electricity produced.

The number of hours with negative prices is on the rise due to the fast adding of renewables, especially solar pv of which there is a lot in NL. On sunny days this high production outstrips the demand for consumption. Solving this in the future requires more flexibility in the system. For example more storage or more consumers (heavy industry for example) that are capable of shifting their demand more to the hours with the highest renewable infeed.

The first occurence of negative prices in NL was 2019-06-02 14:00. But below -50 EUR/MWh has only happend in 31 hours since start of 2015 (the start of the entsoe database on prices). Below -100 EUR/MWh only 10 times! The other times were in April and May of 2022. Of the 10 hours 3 were for 2023-04-19. So this is still a very rare occurence!
Thus, could we find other factors then just high solar and wind infeed, which contributed for this scenary on 2023-04-19 specifically?

Capacity Domain of CORE region

NL is part of the CORE region in which the flowbased capacity calculation methodology is active. In this methodology fixed capacity on borders are not defined fixed. Insead, a set of constraints are defined, from which the import/export of one zone can (and most of the times will) influence the limitations on the import/export of another zone. From these constraints it is possible to calculate the absolute limits of import/export for each zone by doing an optimization calculation, optimizing the import/exports of all zones to enable the maximum import or export for a specific zone. This info is usually a good starting point to see whether any special capacity restrictions were active on certain hours. This data is expressed in min max netto positions. The netto position (NP) of a zone is the sum of all exchanges on its borders, in which a positive NP denotes net export and a negative NP net import. In figure 2 below this info is plotted for days 2023-04-18 and 19 for NL. These numbers are defined within the subregion of CORE.

Figure 2: Theoretical borders of netto position of NL within the CORE region

Here we indeed see some clear trend breaches from 18th to the 19th. It seems that the import and export boundaries are smaller, and the import boundary is even a clear straight line. This trend breach is even more visible for our neighbour BE, as shown in figure 3.

Figure 3: Theoretical borders of netto position of BE within the CORE region

Here the restrictions seem even larger relatively speaking compared to the day before.

Thus, what could be the cause of these restrictions?

Individual validations

To answer this question a quick explanation is needed. As part of the capacity calculation process done together by all the Transmission System Operators (TSOs, for NL that is TenneT Netherlands), it is possible to execute a so called individual validation (IVA). This means that a TSO can do a net security calculation and if it determines that certain parts of the capacity domain result in unsafe situations, it will apply capacity restrictions on applicable constraints within the domain. This is only allowed when the TSO can show that even with all possible interventions it theoretically can still not solve the detected issue. This is done every day and all TSO’s apply so called IVA’s from time to time.
All validations applied including a justification explanation are available on the excellent publication tool here.

In table 1 below the application of IVA’s is shown for 2023-04-18, expressed as percentage of the number of constraints in the domain on which an IVA was applied as well as the average size in % of Fmax. Fmax means the maximum amount of MW that a constraint can handle physically. Only TSOs which applied IVAs are shown.

tso	% of num constraints	average size as % of Fmax
ELIA	0.29	10
PSE	6.10	0.90
RTE	63.64	24.29
TRANSELECTRICA	19.15	2.44

Table 1: percentage of presolved domain on which an IVA is applied as well as the average per TSO for 2023-04-18

Here we can see that a minority of TSO’s have applied IVA’s and those that did are mostly minor. RTE (French TSO) does jump out here, with quite a large part of the presolved domain being hit for French constraints. However the market seems to not have moved in that direction of those constraints that day, so no major price disturbance was visible.

In table 2 below the same data is shown for 2023-03-19, showing a much more extreme picture.

tso	% of num constraints	average size as % of Fmax
50HERTZ	30.56	13.55
AMPRION	19.05	15.88
ELIA	100.00	69.24
PSE	7.02	3.33
SEPS	2.13	13.17
TENNETBV	27.66	15.38
TENNETGMBH	9.00	7.11
TRANSELECTRICA	17.65	13.08

Table 2: percentage of presolved domain on which an IVA is applied as well as the average per TSO for 2023-04-19

Here, we can see that the majority of TSOs has applied IVAs and Elia even a very high size (last column). Why they did that is neatly published on the aforementioned publication tool page here. Here we can see something very interesting when looking at the “Justification” column, most TSO’s actually applied IVAs due to a failure in their local tooling, applying a so called “fallback”. This is the case for 50Hertz, Amprion, TenneT NL, TenneT DE and Elia. So what does this mean exactly?

Fallbacks

In complex systems of any kind it is always possible that something goes wrong. While the TSO systems are designed very robustly sometimes things go wrong. What exactly goes wrong can be a number of things, for example an IT crash (possibly due to bugs), missing/incomplete input data or highly implausible results (as judged by operators). To still ensure safe grid operation a fallback procedure is then used in which a reduction is applied that leads to a situation that is assumed safe regardless of the circumstances. This is by definition a quite conservative low value.
A fallback situation is not unique to the local validation step, there are fallbacks defined for all steps of the market coupling process. These vary from rather simple things like the low values mentioned or replacement input data from a reference timestamp (for example if up to date grid models are unavailable), to more complex completely different backup processes. For example, if in the worst case the market coupling completely fails and no prices could be computed a so called “shadow auction” is held. These are a much more simplified (and thus less efficient) way to still get some trades going.

It is important to stress that, while rare, applying fallbacks is an expected process step. TSO’s have the obligation to ensure safe grid operations and less capacity but garantueed to be safe is always better than more capacity but high risk for grid safety. Errors of any kind are always possible to happen and all TSO’s have responded in an agreed way. It is therefore wrong to blankly blame certain TSO’s for the mere fact that they reduced capacities. No TSO does this just for fun and all fallback incidents are extensively investigated and if needed measures for the future taken.

As for the size of the fallback, each TSO is allowed to define its own fallback rules. Why Elia decided to apply such a relatively large reduction as fallback is not known. Because of its size it probably had the most impact, in interplay with the other fallbacks. It is possible that TSOs or regulators will release calculations on this in the future.

The effect of IVAs of the 19th

The effect of these restrictions we already saw in figure 2 and 3. To clarify, the relative impact on the theoretical boundaries on the netto position is expressed in figure 4 and 5 below, as a percentage of change compared between 18th and 19th. Keep in mind that the days are not exactly the same, and these boundaries shift every day and every hour. However the fact that these days were normal workingdays directly after eachother makes it comparible as a larger picture.

Figure 4: Relative change in maximum export possible for BE and NL between 2023-03-18 and 2023-04-19

Figure 5: Relative change in maximum import possible for BE and NL between 2023-03-18 and 2023-04-19

Here we can see that the main zone impacted is BE, which makes sense given table 2, in which mainly BE constraints were hit by reductions. But also NL was quite impacted, especially on the main sunny hours.

For NL in the context of the highly negative prices, the most important metric here is figure 4, the export. As mentioned before, NL has a large amount of renewables (RES) infeed which outstrips demand. If not enough can be exported the prices are pushed far below zero. This is most probably what happend on the 19th. The export restrictions in the form of the various IVAs caused a large surplus within NL for the most sunny hours, pushing the prices to such extremes as we saw in figure 1.

Yet if we look at 1 the possible maximum export and 2 the realized export, we still see that this boundary was not hit. This brings up the last question I will deal with, why is that the case?

A dive in the flowbased domain

As mentioned the min max NP is a theoretical boundary. In the capacity calculation method used in the CORE region the constraints are not fixed per border. Instead they are are a set of constraints in which the export and imports of all zones are linked together. So, for example an export from Germany to Austria can create more possible export from Netherlands to Belgium. This theoretical boundary as shown in the figures is achieved by optimizing the netto positions from all zones to achieve a maximum (or minimum in case of imports) netto position of a certain zone.

If we run this optimization for the hour with the lowest price (13:00 on the 19th) to get the maximum export for NL, we get the configuration of netto positions for all zones in the CORE region as shown in table 3. Please note that these are netto positions within the CORE subregion, not the whole SDAC region.

zone	NP needed for max NL	historical NP
AT	-880	-3213
BE	-838	296
CZ	4260	-62
DE	-10499	3941
FR	3282	-1155
HR	-496	-410
HU	-2698	-431
NL	3519	1818
PL	1520	53
RO	-1746	-380
SI	5191	-460
SK	-615	2

Table 3: configuration of netto positions needed to enable max export for NL on 2023-04-19 13:00

Thus, in order for NL to reach its maximum export in the capacity domain, as available on 2023-04-19 13:00, 1 Germany would need to import more then 10GW, 2 SI would have flipped from slightly importing to massive exports and 3 AT would have a much higher export as well. These extremes seem unlikely to be the best solution for the optimization. It is thus not surprising that the market algorithm decided to not go into these extreme corners.

These calculations cannot determine if without the reductions the price would have been non-negative. Determining this requires a full market simulation. However, given the large solar infeed in NL it is quite possible that the prices would be less negative but still under the zero.

Conclusion

This post has become quite long but I hope that anybody reading it has learned a bit more about how the market works and why the 19th was such a special case. If you have any remaining questions feel free to shoot me a message on twitter or linkedin or send me an email (info on the contact page of this blog).

DISCLAIMER: All info in this post is based on public available information. Theories and assumptions described in this post reflects only my personal view and analysis of the energy market. They are NOT, in any way, related to my position as process specialist or reflect TenneT strategy or position. For company based opinions on this topic, please contact the communication office of TenneT.

Flowbased: an update about the the Nordic Flowbased parallel run results

Fri, 31 Mar 2023 00:00:00 +0000

The Nordic Flowbased external parallel run is still going strong and its two months since my first look so time for an update!

If you missed it, please see the first part for a general introduction and some data remarks.

Flowbased Domain Results

Lets start with looking at the capacity calcuation process and domain. The flowbased process output seems to be still quite stable. From business day 2022-09-01 until the last business day possible, at time of writing 2023-03-31, all 5088 hours have a domain. All FB domain data in this post is for this time range.
According to the FB domain backup page only for two business days, 2023-03-16 and 17 a fallback or backup was applied. Thats only a failure rate of 0.94%. Thats pretty good for a parallel run.

When looking at the number of average constraints per hour the numbers have barely moved since the last post. See table 1 below for current numbers per TSO, I have filtered on actual branches, so excluding virtual ones.

TSO	average number of constraints
Energinet	153.6
Fingrid	110.4
Stattnett	153.1
SvK	184.9

Table 1: Average number of constraints per business hour per TSO

In the below figure 1 the total number of constraints per month per TSO is visualized, which seems quite stable.

Figure 1: Total number of constriants per TSO per month (figure is interactive)

To zoom in a bit on the constraints themselves, we can look at the average Remaining Available Margin (RAM) as percentage of maximum flow Fmax for the presolved domain in figure 2. Here too the picture is quite stable accross time. One thing that has changed last 2 months is that Stattnet has seen a small drop in average RAM, but still achieves >70% (an indicator that compliance with the 70% MACZT rule is in the right direction).

Figure 2: Average RAM as percentage of Fmax per TSO per month, presolved domain (figure is interactive)

The last domain indicator to look at is the average reference flow Fref in the presolved domain. Here not much has changed, with Fingrid having a consistently high Fref. Why this is the case is unclear to me. If anybody from Fingrid reads this feel free to shoot me an email about it ;)

Figure 3: Average Fref as percentage of Fmax per TSO per month, presolved domain (figure is interactive)

This is a highover average, for a look at more detailed average RAM per TSO an automatically updated dashboard is (still) available here. In figure 4 below a live embed is shown.

Figure 4: Average RAM as percentage of Fmax for the last 30 business days (live dashboard embed, interactive)

Market Simulation Results

Just like last time lets also look at the simulated market results. As a recap, these are simulated with historical orders but with the new flowbased domain from the parallel run. From these simulations the prices and net positions are then published.

At the time of writing this, simulated market results are available for business day 2022-11-14 until 2023-02-26 onwards. All 2520 hours in this range have a result associated with them.

Lets begin with the simplest visualization: the average shift in prices per bidding zone. Negative means a lower price in the simulation compared to the historical result, positive the other way round. This is shown in a map form in figure 5 below.

Figure 5: Average difference between simulated prices and production prices per hour over all hours, negative number means a lower price on average in the simulations. All in EUR/MWh

Here we can see that the south of Norway and mainland Denmark see a decrease in prices as well as in the middle of Sweden, while other zones see increased prices. The same story arises in terms of import and export of energy. On average the zones with decreasing prices see more import and vice versa.
This is visualized in terms of average change in netto position (the sum of all imports and exports of a zone) in figure 6 below. A negative netto position means net import and a positive number net exports. Thus a negative average difference in netto position means the zone has shifted towards the import direction and vice versa.

Figure 6: Average difference between simulated Netto Position and production Netto Position over all hours, negative number means shift into import direction. All in MWh

This coupling between prices and netto position shifts makes sense in that flowbased has made it possible to shift more power from a historical cheap zone (thus increasing the prices and the export) to a historical more expensive zone (thus decreasing prices and increasing import). This largely corresponds to zones with high load, such as larger cities and industry, in Norway and Denmark.

However this high over indicator is also market situation sensitive, on how much the prices shift. A much more interesting indicator is the price spreads on borders. A more efficient working capacity calculation should result in more price convergence, ie lower price spreads on bidding zone borders. The average difference in price spreads per border is shown in table 2 below.

border	average difference	border	average difference
NO3-NO5	-28.63	DK2-SE4	0.97
NO1-NO3	-25.55	DK1-NL	1.15
FI-NO4	-19.47	NO1-SE3	1.21
NO4-SE2	-16.12	NO1-NO5	1.72
SE1-NO4	-12.94	NO1-NO2	3.23
DK1-SE3	-6.05	NO2-NO5	4.17
DK1-DK2	-5.22	FI-SE3	7.35
FI-SE1	-4.55	SE1-SE2	9.11
NO3-NO4	-2.74	NO3-SE2	9.41
SE3-SE4	-0.57	SE2-SE3	13.96

Table 2: Average difference in price spread per bidding zone border, averaged over all hours, all in EUR/MWh

Like last time we can see large decreases on most borders and small on some others. Except the SE2-SE3 is quite big. For this I dont have an explanation, if any nordic expert is reading this feel free to send me a message about it ;)

Finally the last interesting indicator that I want to look at is the overall price spread in the whole region. So the difference between the maximum and minimum price for the whole region per hour. In figure 7 this is shown, together with a comparison with production value. The blue bars signal the difference of simulation vs production, so a negative value is a decrease in the spread for that hour.
For an interactive dashboard see here

Figure 7: Price spread in Nordic region for both simulated and production values, for all data available. All in EUR/MWh (live dashboard embed, interactive)

On average the price spread has decreased with 13.71 EUR/MWh over the whole dataset. That is less then my last post but that is mainly due to the fact market price volatility has decreased overall since the winter. To counter this its better to express it as a percentage change per hour, which on average calculates to a 6.73% decrease in regional price spread over the whole data set.
When looking at the difference graph some extreme peaks can be seen at the end of November. I am not sure if that is due to market volatility in general or that something in the simulation went off. I would love to know why that is, but its not tracable from the published data. When filtering out 9 timestamps which have an increase of over 500% in price spread, the average overall comes at an even more substantial 11.15%!

Overall the parallel run shows good results. The process seems stable and the capacity domains are available for all hours. The price convergence is still a substantial improvement when compared to the current capacity calculation system. Good results that are being backed up with more and more data as time goes on!

Flowbased: a quick look at first results of the Nordic Flowbased parallel run

Mon, 23 Jan 2023 00:00:00 +0000

The Flowbased capacity calculation methodology that I have written about earlier is not meant only for continental Europe. In the Nordics (NO, SE, DK and FI) the TSO’s and market parties have been working on an implementation in their region as well. In the recent months they have started to publish results from their external parallel run. This is a process in which you run the new methodology in parallel to the old one still in production to be able to compare results.

In this post I take a quick look at these first results.

Some remarks on the data

I have pulled the data from the publication tool as well as the RCC page.
The Flowbased results are available for quite some months but during the run the process is continuously improved. I have therefor opted to view the last couple of months, arbitrary chosen starting from 2022-09-01 until the last business day available when writing this, 2023-01-23. For this timerange there is at least one constraint for every hour in the dataset.

For the simulated market results I have pulled the excel files from the RCC page. Only a couple weeks are available with gaps. Only two days in September 2022, another three in october 2022 and all days between 2022-11-11 and 2022-11-30 and again from 2022-12-11 and 2022-12-25. From 2022-12-11 they are published continuously but with a delay, so hopefully more results for after 2022-12-25 will come available soon.

UPDATE: I later found a bug in the date parsing. So the market results are actually continious starting at 2022-12-11. I have rectified this in the dashboards. For the statistics presented in this post the differences are small so I will leave it as it is. See the update for more recent data in which this bug is also rectified.

Flowbased Domain Results

So first lets look at the set of Flowbased constraints themselves, called the final domain. It is available from the beta version of the publication tool, where various parameters are published.

A couple of remarks can be made from the available final domain. Most basic information is there, the capacity given to the market (RAM) as well as the maximum physical flow (Fmax) and the reference flow (Fref) is given.
Designations of which constraints are presolved (a subset of constraints which are the ones that are really relevant with all others being less restrictive then this subset) are also available.
The theoretical ranges of maximum bilateral exchange and net positions published correspond with the domain when I calculated them myselves for a couple of test hours. So the basis is all there which signals a good milestone of a running process.
On average there are 703 constraints per business hour, however this includes some helpful virtual ones as a summary for borders as well as allocation constraints. Curiously no equality constraints are published for the internal HVDC lines. When doing calculations on the domain I had to add them myself.
When filtering for only actual CNECs 595 are left on average per business hour, or 14282 per business day. This is split out per TSO in table 1 below.

TSO	average number of constraints
Energinet	155
Fingrid	107
Stattnett	149
SvK	184

Table 1: Average number of constraints per business hour per TSO

These numbers seem quite low to me and suggest that only a subset of operational CNECs is being used. This is quite normal at the start of conducting parallel runs.

There also some things missing, again to be expected for a beginning phase of a parallel run. No EIC codes are given and many CNEC names look like placeholders. Information about substations of a constraint is only filled in sparsely.
Validation corrections for net security are also not present yet.
Virtual capacities (AMR) are mainly zero, that is quite rare in CORE Flowbased and I am not sure if that is due to not publishing all of them or if they are indeed not needed.

Now to visualize the domain for some quick checks.
In figure 1 below the average Remaining Available Margin (RAM) is shown per TSO as a percentage of the total capacity possible on a line (Fmax) over all presolved constraints.

Figure 1: Average RAM as percentage of Fmax per TSO per month, presolved domain (figure is interactive)

It seems all TSO’s have a quite high average RAM and all above the 70% minimum required from European regulation. I couldn’t find any reference to a minimum RAM mechanism as in CORE, but the fact that this quite high average RAM is achieved it seems plausible that one is used, but not published (yet).

In figure 2 below the average reference flow as percentage of Fmax is shown. This is the flow that is already there before the day-ahead market for various reasons.

Figure 2: Average Fref as percentage of Fmax per TSO per month, presolved domain (figure is interactive)

Here some interesting difference can be seen. Apparently Fingrid reports a consistently higher reference flow on average on its CNECs than the other TSOs. I am not expert on the Nordic grid and thus have no idea why. If any Nordic expert is reading this feel free to reach out ;)

This is a highover average, for a look at more detailed average RAM per TSO I have made a dashboard which is automatically updated here.
In figure 3 below a live embed is shown.

Figure 3: Average RAM as percentage of Fmax for the last 30 business days (live dashboard embed, interactive)

Market Simulation Results

The Flowbased domain results are not the only outputs of the parallel run. These capacity results are also used to simulate the market with real market data. From there we can see changes in price results. As outlined above these results are not a big set, only 960 hours are available. These are only the prices and prices spreads, no social welfare analysis is shown. This is available in the reports that are published next to it, but those are only available for two weeks so I have left them out of scope for this post for now.

A simple but interesting visualization is the average price shift between simulations and production. In figure 4 below this is visualized in a map per bidding zone present in the external parallel run. Here you can also see some non Nordic bidding zones. These have interconnections with the Nordic zones and are thus also impacted.

Figure 4: Average difference between simulated prices and production prices per hour over all hours, negative number means a lower price on average in the simulations. All in EUR/MWh

From here it can be seen that Denmark, Netherlands and Germany and the south of Norway have lower prices while the rest has slightly higher prices. This does not mean that the latter do not benefit. These are purely the Day Ahead prices, with the market coupling optimizing for global welfare (both producers and consumers) for all zones summed.

Price shifts are caused by shifts of import/exports. This can be visualized too by looking at the average change in netto position (the sum of all imports and exports of a zone). A negative netto position means net import and a positive number net exports. Thus a negative average difference in netto position means the zone has shifted towards the import direction and vice versa. This visualization is shown in figure 5 below.

Figure 5: Average differnce between simulated Netto Position and production Netto Position over all hours, negative number means shift into import direction. All in MWh

From here it can be seen that the higher load zones, the more densily populated areas of South of Norway and Sweden, shifts towards import direction while the more production heavy areas (the north of those counties) shift more to export. This corresponds to the price shifts in figure 4.

A much more interesting indicator however is the price spreads on borders. A more efficient working capacity calculation should result in more price convergence, ie lower price spreads on bidding zone borders. The average difference in price spreads per border is shown in table 2 below.

Border	Average Difference	Border	Average Difference
NO3-NO5	-51.92	SE4-PL	0.63
NO1-NO3	-48.33	SE4-LT	1.03
FI-NO4	-22.03	NO1-NO2	1.77
NO4-SE2	-17.99	FI-EE	2.23
NO4-SE1	-12.04	DK1-DK2	2.33
FI-SE1	-8.19	DK1-NL	2.57
DK2-SE4	-7.61	DK1-DE_LU	2.66
NO3-SE2	-6.88	SE4-SE3	2.69
DK1-SE3	-5.90	NO3-NO4	2.81
SE4-DE_LU	-4.35	NO1-SE3	3.22
NO3-SE1	-3.74	NO1-NO5	3.64
SE2-SE3	-3.24	DK2-DE_LU	3.75
		NO2-NO5	4.57
		NO2-DK1	5.54
		NO2-NL	6.87
		NO2-DE_LU	7.90
		FI-SE3	8.18
		SE1-SE2	9.21

Table 2: Average difference in price spread per bidding zone border, averaged over all hours, all in EUR/MWh

The table shows a large decrease in average price spread for some borders and a small increase on others, so overall the price convergence seems indeed to be improving when using Flowbased.

Another interesting indicator is the price spread in the whole region. So the difference between the maximum and minimum price for the whole region per hour. In figure 6 this is shown, together with a comparison with production value. The blue bars signal the difference of simulation vs production, so a negative value is a decrease in the spread for that hour. Only the last full block of results is shown here.

Figure 6: Price spread in Nordic region for both simulated and production values, for last continuous range available. All in EUR/MWh

The average difference is -34.84 EUR/MWh and in 71% of the shown timestamps the price spread in the simulation results is lower then the prices in production. That is quite a nice result. The disclaimer here however is that the timerange is small and more data is needed for a more confident conclusion. But for a first result its encouraging!

The full dashboard for the simulation results I created can be found here.

These are all very preliminary statements and more results and indicators are needed to say more about this. Especially the global social welfare change is the best indicator, but data on that is only available very sparsely.

Whats next?

The publication of these first results are a big milestone for the Nordic Flowbased implementation. It shows they are able to run a stable process and get plausible results from them. Big congratulations to all involved! The planned go live date is somewhere in 2024 Q1 so there is still a year to further flesh things out. In particular I am curious abouet the net security validations and more info about the branches such as eic codes. More social welfare analysis is also crucial to build confidence in that the system really is better and more efficient for society as a whole.
I will try to repeat this blog post later on this year when more results become available to see where it is going. In the meantime, as always, if you have any questions or remarks please don’t hesitate to contact me, especially if you are a Nordic Flowbased expert ;)

Aten: Market Cap Reached Incident Baltics 2022

Mon, 29 Aug 2022 00:00:00 +0000

This post is part of a series written to celebrate the launch of Amun Analytics. This company will serve as the business umbrella for all my data analytics projects. If you have some interesting data analysis and visualization projects in which you need some help, shoot me an email! Now lets dive into the topic of today.

High Prices in the Baltics

On Wednesday 17th August 2022, the Day Ahead (DA) market price in the Baltic region hit the price cap of €4000 per Mwh. Between 17:00 and 18:00, the price for Estonia, Latvia and Lithuania hit simultaneously the then active price cap (depicted in figure 1 below).

Figure 1: Day Ahead prices for the incident day for Baltic region (figure is interactive)

What is interesting about this event is the pattern of the hours before and after this sudden increase, as shown in table 1 below.

hour	Lithuania(LT)	Latvia (LV)	Estonia (EE)
16:00	750.08	750.08	750.08
17:00	4000	4000	4000
18:00	750.08	750.08	750.08

Table 1: Excerpt of hour before and after price increase

From one hour to the next (i.e. from 16:00 to 17:00 hrs) there is a sudden sharp increase in the price. This is followed by a sharp decrease (i.e. from 17:00 to 18:00hrs) to the exact same level. What factors may possibly explain this movement? What is so different at this specific hour in which the price cap was reached?

Intermetzo: the Price Cap

Although the Day Ahead (DA or spot) market prices have been continuously high across Europe when compared to the year before, hitting the price cap is still very rare. So what is this price cap exactly?
In order to have some algorithmic price limitation, market biddings in day ahead have a cap. This mechanism also acts as a circuit breaker to try to calm down extreme situations. It can be seen similar to mechanisms found in stock markets, but with a much longer timespan.
At the beginning of this year the price cap stood at €3000/MWh. However European regulations determine that if the DA price hits more then 60% of the current price cap anywhere in the market area SDAC for any hour, a price cap increase of €1000/MWh will be applied five weeks from that day.
This increase already happened at 4th of April 2022 in France. Hence the cap stood at €4000 euros per MWh at the 17th of August.
With the DA price hitting the price cap again on the 17th of August the price cap will be €5000 per MWh five weeks after. There is currently no theoretical limit to this process.

The Market situation

Back to the Baltics. Lets take a look at the market situation. The energy exchange Nordpool publishes the so called aggregated bid curves, one for supply and one for demand. These curves are calculated by adding all supply bids and all demand bids that were available, all expressed in a price for a certain amount of power that is being offered to sell or buy. The final clearing market price is then derived from where supply and demand intersects. This is all done on an hourly basis. Nordpool publishes these for the whole Baltic region.
For the price capped hour these are displayed in figure 2 below.

Figure 2: Aggregated bid curves for Baltic region for 2022-08-17 17:00

The hour after is displayed in figure 3, the hour before looks very similar and is not shown here.

Figure 3: Aggregated bid curves for Baltic region for 2022-08-17 18:00

As you can see the situation is very tight for both hours. But specifically if the supply curve(blue) was only shifted with approximately 50 MW, a non capped price could have been found (Nordpool does not publicly publish these figures as numbers so exact calculation is impossible). In fact when, comparing the hour 17:00 in figure 2 with the hour 18:00 in figure 3, it seems the situation is very similar except about 50MW in supply is missing. The hour before (not shown here) displays the same behavior.
What could be the cause of this missing supply for hour 17:00?

Capacities on the Borders of the Baltics

The Baltics are a corner of the SDAC region, but with often used connections to Sweden and Poland in the South (with Lithuania) and to Finland in the north (with Estonia). Power can also be exchanged with Russia and Belarus, but here capacity is offered only incidentally.
Let’s see if a possible cause of the mentioned missing MW can be found in how much capacity was available for import (import gives more supply, export more demand).
For the northern border, the capacities offered to the market to move power in (so called Net Transfer Capacity or NTC) are shown in figure 4 below for Estonia for the import direction (for example FI > EE).

Figure 4: Import NTC for Estonia for the incident day (figure is interactive)

Although a decrease was apparently applied for part of the day it is not specific to the hour 17:00. In other words, this cannot be a plausible cause for the specific shortage in the hour we are investigating.

Now let’s look at the southern borders, again import NTC’s are shown, this time for Lithuania. See figure 5 below.

Figure 5: Import NTC for Lithuania for the incident day (figure is interactive)

Here we do see an interesting situation around 17:00. It seems that the Polish border was restricted between 16:00 and 22:00. Simultaneously the cable from Sweden was also restricted between 10:00 and 18:00.
To make this more clear, let’s add up all the import capacities on all borders, to look at the total import capacity per zone for the Baltic region, excluding internal transfer capacity. This result can be seen in figure 6 below.

Figure 6: Total import NTC for the Baltics for the incident day, excluding internal transfer capacity (figure is interactive)

A clear small dip can be seen at the hour 17:00. To zoom in on this, the numbers are put in table 2 below.

hour	Lithuania(LT)	Estonia (EE)
16:00	1050	858
17:00	1010	858
18:00	1050	858

Table 2: Excerpt of import NTC for the hours before during and after the 4k EUR price

Thus, it seems that at the exact hour of the shortage (17:00) there was a 40MW lower import capacity for the Baltics on the southern borders with Poland and Sweden. In the hour before and after a bit more capacity was available on the Polish and Swedish connection respectively. Together with the bidding curves shown earlier (figure 3 and 4 above) this seems like a plausible cause for the sudden increase in price at hour 17:00 and normalization in the hour after.

This reasoning only holds if the full import capacity was actually used Otherwise it is not constraining the situation. To check this reasoning the utilization, defined as scheduled commercial exchange divided by the capacity offered, is plotted per external border in figure 7 below.

Figure 7: Import capacity utilization for the incident day for all external Baltic borders

From the figure above, it is possible to infer that the market did indeed use 100% of the import capacities available for 17:00. Thus the restricted capacity is a plausible cause of the hitting of the price cap, especially compared to the lower priced hours before and after.

In Conclusion

The hitting of the price cap in the Day Ahead market is a rare and extreme situation. The market in the Baltics was quite tight in terms of supply and demand. However, the market was able to clear far below the price cap for most hours. At 17th August 2022 17:00 it did not.
From the bid curves it followed that an extra supply of 40-50MW would probably have been enough to make the market clear and keep a similar price compared to rest of the day. A plausible cause for this difference is a restriction on the import capacities for the Baltic region at that hour.

So is this a definitive cause to blame for this sudden hitting of the price cap? No, the aggregated bidcurves are only available as simple images and the market coupling algorithm is much more advanced than simply shifting some curves around. More complex market interactions could have been contributing. However the above laid out analysis does point to it as a strong plausible contributing factor.

It can also be said that you cannot “blame” any such specific factors of the system. After all, it worked as its designed to. Capacity restrictions are there for many reasons. Grid operators do not restrict capacities because they think it is fun. They do from a net security perspective. Worst case scenario, it is possible that without the restriction no grid would have been possible to operate at all at that hour! To confirm this case there is not enough public information available. We remain thus waiting for a proper debrief by the regulating authorities.
In the end it comes down most simply to the fact that there was not enough supply for that hour willing to sell at a price that demand was willing to pay.

Finally, what this also mostly shows is the importance of interconnectivity for the grid. A relatively small change in interconnectivity can have major consequences. Its important to keep this in mind for any person or entity who have to take decisions concerning this.

Appendix

To show that 2022-08-17 was a non typical day in terms of capacities, we can repeat figure 6 for a reference day, for example the same weekday a week earler. This is shown in below figure.

Appendix A: Total import NTC for the baltics for reference day, excluding internal transfer capacity (figure is interactive)

Again there is some restriction on the north border but the total picture is much less severe then the incident day of 17th.

The prices for the same day are also more balanced and in line with the rest of the SDAC, as shown in below figure as a repeat of figure 1.

Appendix B: Total import NTC for the Baltics for reference day, excluding internal transfer capacity (figure is interactive)

Aten: negative prices and wind curtailment

Tue, 03 May 2022 00:00:00 +0000

In the weekend of 23th and 24th of April 2022 the day ahead prices in the Netherlands reached record lows. On Saturday it reached €-222.36 at 12:00 and on Sunday €-117.21 at 13:00. Now there are many discussion to be had about the place of negative prices in the current market design, which I will not get into now here. However the cause is quite clear: a huge ammount of renewable energy sources. With the growth of these sources negative prices will happen more and more in the future.
Yet many units keep running during times with very low prices. This can have many reasons, for example because shutting down and starting up again is too expensive. Or because units are active on the balancing market instead. Only when none of these are true units shutdown.

One of the typical examples of this is offshore wind units. Shutting down and starting up is quite low cost for wind units, but on top of that they also do not receive SDE subsidies if there are six or more hours of negative prices. Blocks of these happened on both Saturday and Sunday resulting in an almost perfect correlation between plunging day ahead prices and offshore wind output, shown here below.

Wind offshore output (blue, left axis) and day ahead price (green, left axis) for the Netherlands on 24-25 April 2022 (click to enlarge)

Data from co2monitor.nl and ENTSO-E transparency for wind output and day ahead prices respectively.

Aten: non-intuitive flows in flowbased capacity calculation: NL -> BE

Wed, 16 Mar 2022 00:00:00 +0000

Intuitive and non intuitive flows

The recent market developments put the spot light on all kinds of interesting behaviour of the way European electricity market works. One of those is non intuitive flows. At 4th November 2020 the electricity day ahead market value broke a fundamental rule: allocation of cross border flows was no longer restricted to go from a low price zone to a high price zone. Flows from a low to a high price zone are called intuitive, since it is intuitive that exporting from a low price to high price will average out the final price between the zones and maximize global welfare.
However this is not always the best optimization. Sometimes the optimal approach is to allocate a flow from a high to a low price zone (also known as non intuitive flows). For example if it makes capacity available for a flow from a low price zone to a high price zone on a different border.
Another example is when a flow goes from a high price zone to a low price zone to an even higher price zone. Non intuitive optimization was enabled for the first time to optimize the integration process within the European electricity market at the 4th of november 2020.

Side note: A more ellaborate explanation of flowbased flows can be found on energeia (subscription needed), written by my colleague Joost Greunsven.

So how many times do these flows occur in practise? And how large are they are relatively to the maximum possible exchange?

Border flow utilization

To more easily compare border flows on different timestamps, we define a border flow utilization percentage. This is a percentage indicating the share a commercial scheduled flows ($F_{scheduled}$) takes on the maximum possible exchange between two zones ($bex_{max}$) per hour.

Mathematically, the utilization of cross border flows is then defined as follows, for for example in the direction $NL \rightarrow BE$ : $$ F^{NL \rightarrow BE}_{util} = \dfrac{F^{NL \rightarrow BE}_{scheduled}}{bex^{NL \rightarrow BE}_{max}}*100 [\%] \tag{1} $$ With:

$F^{NL \rightarrow BE}_{util}$ as the utilization [%]
$F^{NL \rightarrow BE}_{scheduled}$ as the amount of day-ahead scheduled commercial flows [MW]
$bex^{NL \rightarrow BE}_{max}$ the maximum possible bilateral exchange as calculated by the CWE capacity calculation [MW]

To determine whether a flow is intuitive we look at the price delta in the direction of the flow, defined as: $$ \Delta P_{NL \rightarrow BE} = P_{BE} - P_{NL} \tag{2} $$ For a positive $\Delta{P}$ a flow goes from a low price to a high price zone and is thus considered an intuitive flow. Vice versa a negative $\Delta{P}$ signals a non intuitive flow.

Non intuitive flows in practice

So outside theory, how does this look in practise?
In the below figure we can observe the border utilization for the first half of March 2022 for the $NL \rightarrow BE$ border. The yellow line depicts the $\Delta P_{NL \rightarrow BE}$. Every bar corresponds to an hour on the day-ahead market. Intuitive flows, those which occur when $\Delta P \geq 0$, are colored green. Non intuitive flows are then flows which occur when $\Delta P < 0$ and are colored blue.
From this figure, we can observe that most of the time there are intuitive flows. However, there are periods of multiple hours in which non-intuitive flows occur. The border utilization on those hours is about the same as directly before and after these periods.

border trade utilization for border NL -> BE with price delta NL (click to enlarge)

Why are these non intuitive flows occuring? The answer seems to be the related to the second reason given at the start of this post. The flow is going to an even higher price zone: France. Although France is historically an energy exporter, due to various factors (out of the scope for this post) it has been importing electricity for the past two months. It has seen relatively high day ahead market prices because of these factors. This effect becomes clear when we plot the price delta on the Belgian border towards France, so $\Delta P_{BE\rightarrow FR}$, as depicted in the below figure.

border trade utilization for border NL -> BE with price delta BE (click to enlarge)

Here the non intuitive flows (blue bars) clearly line up with high peaks of the $\Delta P$. Thus it seems the market algorithm has allocated flows going from $NL\rightarrow BE\rightarrow FR$. That explains the non intuitive flow pattern on the Dutch Belgium border. A beautiful example of the advantages of optimizing on a larger area specifically and the advantages of the European market integration project in general!

If you want to discuss, share or like this you can do that at the corresponding LinkedIn post:
Go to LinkedIn Post

Utilization methodology technical remarks

There are some limitations to our method of defining an utilization.
Comparing maxbex with scheduled exchanges is a bit comparing apples with oranges:

The maxbex is calculated within CWE flow-based capacity calculation and represents the maximum possible flow between two CWE bidding zones, if all other exchanges between CWE bidding zones are 0 (CWE being the Central Western European capacity calculation region).
The scheduled exchanges result from all day-ahead exchanges, not just those in CWE.

As a result, utilisation values may reach >100% on a border. For example if a relative high amount of non-CWE exchanges are scheduled on a zone border, they may surpass the maxbex value. However, maxbex cannot be calculated for non-CWE. In addition, scheduled commercial flows are only available on EU level. Therefor, this is the closest we can get to defining a semi accurate percentage measure.

Acknowledgments

This post is based on the research done in close collaboration with my wonderful colleague and walking encyclopedia of the electricity markets Joost Greunsven.

Data sources

All data comes from the public sources of ENTSO-E transparancy platform and JAO utility tool.

Aten: electricity generation sources in NL 2021

Tue, 11 Jan 2022 00:00:00 +0000

When I talk to friends and family about the electricitymarket, they often do not realise the large reliance of the electricity market on gas. This inspired me to look at 2021 electricity production sources.
The plot below shows largest electricity source per hour in the Netherlands (sources are color coded over time; days at the x-axis and hours at the y-axis).
This plot makes evident the market’s reliance on gas (i.e. yellow = gas). It also highlights the market’s shift towards renewables sources (i.e dark green = wind; dark blue = solar).
Time of the day and seasons also seem to influence the usage of energy sources. It is evident that day time electricity generation in summer time is largely powered by Solar energy. That can be seen by the nice block of blue for Solar energy at the middle of the day.
The effect of increased coal usage (light blue) due to high market prices in the second half of the year during is also visible, with it being the dominating power sources at night.
What do you think, how long will we keep being reliant on gas?

Largets electricity generation source in NL per hour (click to enlarge)

Color legend:
Light green: biomass
Yellow: gas
Light blue: coal
Orange: nuclear
Red: other
Dark blue: solar
Pink: WKK (greenhouses)
Dark purple: wastepower
Dark green: wind (onshore + offshore)

source: data from co2monitor.nl, “Wind” here is onshore and offshore combined.

If you want to discuss, share or like this you can do that at the corresponding LinkedIn post:
Go to LinkedIn Post

Aten: Price Convergence 2021 in CWE region

Tue, 21 Dec 2021 00:00:00 +0000

Bar chart of price convergence per day in 2021 for CWE region (click to enlarge)

What is price convergence?

The electricity market in Europe is split into so-called bidding zones. The rule is that within these zones you can trade without transport restrictions, as if the entire grid there is one big conducting plate. This also means that for the day ahead marketi, there is one price per hour per bidding zone. If neighbouring bidding zones have enough interconnection capacity available to satisfy the market direction, then the price in those zones will converge: it will be the same. So, price convergence can be seen as a measure on how much transport restriction the market has run into between bidding zones.

The figure explained

The above plot shows the price convergence per day in the Central Western European (CWE) region, which consists of NL BE DE FR and AT.
It is calculated by subtracting the minimum price from the maximum price to get a price delta for each hour. Then this delta is categorized in bins such as between 0 and 1 euro, 1 and 2 euro difference etc. The bins are colour coded, going from dark green at full price convergence (d=0) to dark red for high non convergence (d>50 euro). The bins with corresponding colours are shown at the bottom of the figure. This is then shown in a bar chart in which the 24 hours in a day form 100% of the bar.

The current market developments of high prices and high volatility also has an effect on the price convergence. As can be seen the last two months there is a high percentage of high non convergence. Which raises an interesting question, is this a temporary unique situation due to high fuel prices, or does it show shortcomings in the current market design?

If you want to discuss, share or like this you can do that at the corresponding LinkedIn post:
Go to LinkedIn Post

Aten: introduction and the first plot

Mon, 13 Dec 2021 00:00:00 +0000

As those that I have spoken with personally recently are well aware by now, I am working at TenneT at the System Operation Transport division, mainly on flowbased capacity calculation and bidding zone review topics. I have chosen TenneT because I wanted to play a part in something that both fascinates my and in which I hope to make a small difference for good: the energytransition and the energy market.
The more I work on it the more I become fascinated by all the intricate details and endless possibilities to analyse it, encouraged by my helpful co-workers. From this fascination a new hobby was born: project Aten. I like to name my projects after greek-roman, old norse or ancient egyptian mythology and this one is named after the egyptian sungod Aten.

Project Aten: What is it?

Project Aten is a data lake and visualization platform that I am building with all kinds of data from the energy market and energy transition that I find interesting. I am trying to look at both the market part of it (for example day ahead prices) as well as the capacity calculation part (for example the flowbased capacity domain). It is purely a personal project and will only consist of publicly available data, but ideas can (and probably will) come from from my work at TenneT.
The pipeline of the project will be roughly: data ingestion with python -> PostgreSQL database -> plotting with grafana dashboard or a python plotting library -> live dashboards and occasional LinkedIn posts.
On top of that I will try to make all underlying aggregated data available in the form of an API.

Enough talk, show me!

There will be three places of output from this project. First, the public dashboards can be viewed at data.boerman.dev. Secondly, the API is available at aten.boerman.dev. Lastly, and probably the most visible, I will write occasional LinkedIn posts with attached blog post like these with some more information, you can find (and connect/follow) my profile here.

I have a visualization suggestion!

Great! I always love getting feedback. Be sure to let me know either on LinkedIn (in a post comment or private message) or send me an email, my address can be found here.

First output: Price volatility in DA market for 2021

Heat map of day-ahead electricity prices for NL in EUR/MWh (click to enlarge)

So as a first output I created the above plot. It shows a heatmap of the day ahead electricity prices for the NL zone in EUR per MWh. Day ahead means it is the price agreed today for delivery of energy tomorrow, this is done per hour, so there is a so called DA hourly price throughout the day. At the y-axis the time is shown, starting at first January of 2021. At the x-axis the price in EUR.
This price can actually be negative! Then there are moments that there is so much energy produced that suppliers need to get rid of, that they will pay consumers for taking it. This is a funny and quite unique aspect of the electricity market, owing to the fact that any energy produced needs to be immediately consumed every timeframe to keep the grid in balance and that its not so easy to ramp up or down on many sources. So it can be profitable to pay a consumer to take your produced energy in hour A so you can keep your generation running for hour B when you can turn a profit.
The last couple of months the electricity market has gone crazy with very high prices, mainly due to very high fuel prices (gas) for big generators. The reasons for high gas prices warrants a whole seperate story so it is not going to be explained here. The heatmap shows not only a much higher average price but also a very high volatility of a market trying to find the sweet spot between high fuel prices for static generators and volatile renewables. Maybe the time is finally there for better flexibility solutions like battery or other forms of storage?

If you want to discuss, share or like this you can do that at the corresponding LinkedIn post:
Go to LinkedIn Post

Graduation project: My presentation

Tue, 30 Jun 2020 00:00:00 +0000

Graduated!

After long months of work, today I finished my graduation project with a 7.5! That means I am now a graduated msc Electrical Engineering. Many thanks to Kees van Berkel and Marc Geilen for their supervising work!

Due to the corona crisis the presentation was done fully online through microsoft teams. I have made a recording of the 30 min presentation, links to all material can be found below. I will finish my blog parts about my graduation project somewhere in the future, but first I am going to enjoy a holiday over the summer.

Material

Adventures in data presenting: Covid-19

Wed, 18 Mar 2020 00:00:00 +0000

While I’m finishing the last touches on my graduation thesis (posts about that will follow soon) I wanted to take my mind off a little bit with some hobby. I have always been fascinated by presenting data in a clear and automated way and wanted to experiment a bit with that. As a subject for this I chose the big crisis of this moment: the corona virus or Covid-19. This posts serves as quick write up documenting various scripts and tools that I used to experiment a bit with data visualization. You cant visualize anything without actual data so that was the first hurdle to solve!

Collecting Covid-19 data

I wanted to visualize the spread and growth of the virus. With some quick googling there seems to be many, sometimes conflicting, data sources on this. I wanted to stick as much as possible to official government or scientific sources. The World Health Organization (WHO) publishes daily situation reports here. However these are pdf’s generated from word which is generally difficult to parse.
Many countries (including my own the Netherlands, more on that later) publish individual statistics in various formats. Gathering all of these would be an enormous task. Luckily for me the Johns Hopkins university in Baltimore, Maryland in the USA quite quickly published an excellent dashboard, which draws from a manually curated database that is composed of information from sources of many individual countries as well as the WHO. Even better this data is published in simple csv format on github. For the worldwide part of my own visualization I decided to use this as source.
For specifically the Netherlands the government agency working on health data is the RIVM, which publishes a daily updated chart. Sadly they dont allow going back in time although you can download the data in csv format. I reversed engineered their csv download link and managed to download a couple of days back as well. After that I wrote a script that checks every hour if there is a new updated csv from the site and then downloads it to my server. It turned out that the RIVM changes their way of visualization during the crisis and I had to change the script quite a couple of times to keep pulling in data. At first it was a specific url with csv, later they seem to be settling on a plugin that reads csv data that is neatly embedded into the html. A quick python script solved the extraction for this. An archive of all data that I managed to retrieve can be found here on my github.
With the data collected onwards to the visualization!

Visualizing in Grafana

My goal for visualizing was that it should be easy to change, open source and look nice. After trying multiple options I was most impressed with grafana. Although there are many configuration options and I have only explored the basics, creating graphs of the growth of the virus was easily done. For now I stick to simple line and bar charts but they already are quite informing. The end result after some tinkering can be found at my self hosted grafana instance located at https://data.boerman.dev/. There are dashboards for all cities in the Netherlands as well as countries per continent or specific individual countries, all detailing the amount of corona virus cases. They mostly work on mobile too. Im still trying to optimize the use of user input variables for a better user experience. For now simple selections of countries or municipalities is possible.

The biggest struggle was to get my collected data into grafana. Grafana supports a vast ammount of them, so it is easy to get overwhelmed in the beginning. I tried one of the most used ones, InnoDB, but found it quite difficult to get my data in the right format to fit into InnoDB. Especially frustrating was the fact that their bindings to my favourite language python was tricky to get working properly. So in the end I switched to a more standard general purpose database in the form of postgresql. For reading the actual csv files the excellent read_csv function from the pandas python library was used. It supports a lot of very usefull tweaks, allowing for parsing almost any form of csv’s. For example it is possible to tell it to skip certain rows and columns or for example empty lines when parsing. The resulting dataframe can then be processed for extra insights. In my case I wanted to create a table that also held the derivative of the data in order to plot it. With pandas this can be easily done with the diff function on a dataframe.

For the worldwide data I also wanted a split between continents, after some searching I found the pycountry_convert library which can parse the name of a country to a country code and then assign a continent. The second part of the import script takes care of this. Due to John Hopkins not always using the official standarized names this required some manual renaming of input data in the script.
When all parsing is done, the resulting dataframe tables can easily be written to postgresql with the to_sql routine, which through the sqlalchemy library supports postgresql natively.

Cobbling this together resulted in two scripts, one for the Netherlands data and one for the world data. By making heavily use of so many great python libraries (once again confirming my great love for the language and its ecosphere) the final scripts are relatively small in terms of lines of code. I have made an overarching bash script which runs on a timer (through systemd) on my server. It checks if there is any new data to download, and if so calls the parsing scripts after downloading. Grafana will then load this when a dashboard needs displaying, resulting in up to date data visualization!

Staying informed: a telegram bot

I was quite happy with the results so far, but I also wanted to stay informed of the spread of the virus (and if my scripts keep running). To do this I made a relatively simple telegram bot, its source can be found here. Telegram is an alternative to WhatsApp with very good bot development support. For now the bot is focused on the Netherlands. It allows you to subscribe to one or more dutch cities and get messages with new RIVM data when they become available, as well as query the current top 20 infected cities in number of active cases. You can connect to it via telegram at the link: t.me/Covid19NLNewsBot.

Conclusion

In the end I’m quite happy with my shiny grafana dashboards, as well as my interactive bot. Although in the grand scheme of things fairly trivial, the setup of data gathering, parsing and displaying was very educational and a lot of fun. In the end this is obviously the goal of hobby projects such as these. To summarize the dashboards can be found at https://data.boerman.dev/, all the scripts at https://github.com/fboerman/covid-19-data.
Next I’m planning to figure out the map plugin of grafana, as well as make a dashboard with live counters of the status of covid-19 worldwide.
If you have any questions or possible feature requests hit me up via email frank@fboerman.nl or telegram (@AwesomeFrank).

Graduation project part 3: FFT theory

Thu, 13 Feb 2020 00:00:00 +0000

Introduction

Welcome back to a new part in my blog series about my graduation project. As mentioned in the previous parts, the Fourier Transform will be dominant in the implementation of the Coherent De-Dispersion. This part will dive into the relevant theory behind the Fourier Transform. This includes another algorithm that is need to accomplish the implementation of the Fourier Transform: the CORDIC.

The Fourier Transform

DFT

The Fourier Transform, named after its inventor Joseph Fourier, converts a signal from the time to the frequency domain [wiki]. The Inverse Fourier Transform does the opposite, converting a signal from the frequency domain to the time domain.
In other words the Fourier Transform decomposes a signal in time into its frequency components, showing which frequencies make up the original signal. Both the time signal and the frequency signal can be complex, which in that case constitutes the phase information of the signal. In its continious form the transform it is defined as show in equation 1 below.

$$ F(x) = \int^{\infty}_{\infty}f(x)e^{-2\pi ix \xi}dx \tag{1} $$

The system that is being designed is however not continious but discrete, therefor the Discrete Fourier Transform [wiki] is needed. The DFT is defined as shown in equation 2,

$$X[k]=\sum^{N-1}_{n=0}x[n]\cdot W^{nk}_N, k=0,1,…,N-1 \tag{2}$$

in which

$$ W^{nk}_N=e^{\dfrac{-j\cdot2\pi\cdot nk}{N}}=W^{\phi}_N=e^{\dfrac{-j\cdot2\pi\cdot \phi}{N}} \tag{3} $$

This $W^{nk}_{N}$ is called the twiddle factor. More on this below.

Radix-$2$

For implementing the DFT of size $N$, in which $N$ is a power of 2, the Fast Fourier Transform (FFT) is used[1]. This divide and conquer type algorithm recursively breaks up the DFT into 2 smaller DFT’s per iteration of size $N=N_1N_2$, in which $N_1$ and $N_2$ are also a power of two. This reduces the complexity from $\mathcal{O}(N^2)$ to $\mathcal{O}(N\cdot log_2\cdot N)$. This pattern can then be written out in so called stages of butterflies.
The FFT is calculated in a series of these stages, with total number stages $S=log_r(N)$, as can be seen for example in the figure below (taken from [2]). The $r$ here is called the base of the radix and is also the number of inputs to each knot in the graph, so the figure corresponds to radix-$2$. The pattern of crossing wires in the flow diagram are named butterflies. The most simple butterfly can be observed near the output at the right of the flow diagram, everything left of it are interleaved butterflies. The number of inputs of each knot is equal to the radix base $r$, in this example 2. Then each butterfly (consisting of two of these knots) with input pair $A,B$ calculates $A+B,A-B$ as two outputs. So each bottow line of the butterfly is multiplied with $-1$, which is not shown in the diagram.
Each number in the flow diagram represents a complex rotation $\phi$ as defined in equation 2. These rotation factors are called twiddle factors or twiddles. It follows that there is no rotation needed for $\phi=0$ and for $\phi=[N/4,N/2,3N/4]$ the rotations are trivial: $W^\phi_N=[1,-j,-1,j]$. These trivial rotations can be executed by swapping the real and complex components.

radix-$2$ flow diagram for $N=16$ , taken from [2]

DIF and DIT

There are actually two ways to use the optimization of the FFT. These are called the division in time (DIT) and division in frequency (DIF). In terms of optimization they are identical. In later parts it will be shown that they are identical in implementation cost as well. Their main difference is the order of in and output samples.
The way the butterflies are defined means that any FFT will reshuffle the order of samples in a way that corresponds to the bit reversing of the indices. Lets look at a small example. For an FFT of size $N=16$ the indices are bitwise expressed in $\log_2{(N)}=4$ bits. A sample at index $2$ is “0010” in binary, reversed this is “0100” which is $4$ in decimal. So in other words a sample with index 2 ends up at index 4. The difference between DIT and DIF is that with DIF the input samples are in natural order, but the output samples are in bit reversed order. With DIT this is the other way around.

The above figure is an example of DIF, which can be recognized in the order of output indices that are shown. In terms of flow diagram DIF starts splitting the DFT according to the FFT algorithm into two parts from the end of the sample stream. This is illustrated in the figure below, for a small FFT of size $N=8$:

flow diagram DIF FFT $N=8$, taken from [2]

In contrast, the DIT starts splitting the DFT from the beginning of the sample stream, as illustrated in below figure.

flow diagram DIT FFT $N=8$, taken from [2]

The consequence of this is the aforementoined sample ordering, natural order input -> bit reversed output for DIF and bit reversed order input -> natural order output for DIT. The above figures also mean that that the flow diagrams of the DIF and the DIT are the transpose of eachother. This is important for the actual implementation, described in later parts.

An example of a higher radix is shown in the figure below, which depicts a radix-$4$ flow [3]. One butterfly in radix-$4$ can be seen as a rewritten double radix-$2$ butterfly. This results in a reduction of multiplications by 25% and an increase in the number of additions with 50%, as proven in [4] and measured in [3]. From dataflow programming mapping to dsp slices in the FPGA, it follows that for every adder there is also a multiplier present. So a trade-off of less multiplications but more additions does not have any actual gain in our programming model. Therefor radix-$2$ is the chosen FFT type.

radix-$4$ flow diagram for $N=8$, taken from [3]

Radix-$2^2$

The series of stages of radix-$2$ can be further optimized. Rewriting the twiddle factors by splitting all non trivial rotations $\phi$ into a trivial and non trivial part $\phi’$ results in the following expression:

$$Ae^{-j\dfrac{2\pi}{N}\phi’}\pm Be^{-j\dfrac{2\pi}{N}(\phi’+N/4)}=[A\pm(-j)B]\cdot e^{-j\dfrac{2\pi}{N}\phi’} \tag{4}$$ in which A and B are the inputs of the butterfly and $\phi’=\phi$ mod $N/4$. In the resulting expression the bracketed part has only a trivial rotation. This is then rotated with a non trivial angle. In practice this is used to move the non trivial part to the next stage for all odd stages, as depicted in the figure below. This method reduces the total number of non trivial rotations in the flow graph at the expense of a slighter more complex control circuit.

Radix-$2^2$ DIF flowdiagram, taken from [5], only the rotation angles have changed with respect to previous radix-$2$ flowdiagram

FFT Implementation Architecture

For the actual implementation architecture of the FFT there are two main choices to be made. The sample flow direction: feed forward only or with feedback, and the amount of inputs: serial or parallel. The four possibilities out of these two choices are called Single-path Delay Feedback (SDF), Multi-path Delay Feedback (MDF), Single-path Delay Commutator (SDM) and Multi-path Delay Commutator (MDC) architectures class for serial feedback, parallel feedback, serial feed forward and parallel feed forward respectively.

In the feedback options SDF can be used for very high $N$ as seen in [6], but it has a very high latency due to the extensive use of large buffers in a serial fashion. MDF generally requires more hardware due to the feedback loops, as shown in [3], and most designs focus on smaller FFT’s both in terms of $N$ as well as in the amount of parallel samples $P$ such as in [7][8]. Due to this focus they are very hard to scale up.
The feedback loops in both SDF and MDF would also mean encountering complex problems with fixed point scaling with higher $N$, more on this in later parts.

SDC has this problematic focus on small size as well, with literature focusing on achieving high speed but lower $N$. One example for a more scalable SDC combines the SDC with SDF [9], but this is only scaling in FFT size $N$ and not in parallel input $P$. It also removes the bit reverse ordering after one forward FFT pass, which is not optimal in the general flow of the de-dispersion design, as will be shown later.
Taking mentioned aspects in consideration, MDC seems most optimal to use for $DM$-trails, for high throughput, low latency and avoidance of complex scaling methods. But the selected MDC architecture needs to be able to scale to $N=2^{16}=65k$ and a high $P=8$ or $P=16$. There are many ways to do this, for example using memory banks in a rotating way [9]. Another way is using cascading building blocks both in time (in terms of $S$) as in parallel samples (in terms $P$). Examples are [3], [11], [5] and [12]. The first two are not focused on larger scale in terms of $P$, but the latter promise high throughput, low latency and high scalability. Therefor this architecture will be the one to form the basis of the implementation of this project. Please note that [12] is an expanded and updated form of the architecture in [5] with added complexity. For our use case this is not strictly necessary and overcomplicates the project. Therefor [5] will be used as the basis for the implementation for our $DM$-trails. The details and implementation steps will be discussed in the next part.

Twiddle factor generation

The mentioned architecture mainly sets out the way to implement the flow diagram, however it is missing in a way to generate the twiddle factors. Twiddle factors are complex numbers that denote a rotation in the complex plane. There are many ways to generate them, but due to time constraints for this project a relative simple method is used. It makes use of the specific pattern of twiddle factors that are needed for the choosen architecture. More details will be given in the next part about the implementation, however in terms of mathematical theory there are two steps: generating a base angle, rotating this angle with itself. The CORDIC and self rotations are important since the quantization error of the FFT is largely depending on the error in its twiddles factors. These will be discussed below to complete the relevant theory part of this project.

Rotation matrix

Both of these steps make use of the standard rotation matrix $R$. $R$ defines a rotation in 2D space for a given angle $\varphi$ and is defined as show in equation 5 below:

$$R(\varphi)= \begin{bmatrix} \cos(\varphi) & -\sin(\varphi) \\ \sin(\varphi) & \cos(\varphi) \end{bmatrix} \tag{5}$$

By taking a complex number $C=x+yj$ as a vector on the complex 2D plane, written as $V_C=[x;y]$, this standard rotation matrix can then be used to generate a desired twiddle factor by rotating the starting vector $[1;0]$ with the desired angle.

CORDIC

The standard way of rotations in hardware is making use of the CORDIC algorithm. The COordinate Rotation DIigital Computer algorithm, sometimes also referred to as Volder’s algorithm after its inventor, is a multifunctional algorithm which in its most simple form (the rotation mode) rotates a vector with a given angle [wiki].

The rotation mode CORDIC (from now on refered to as CORDIC) iteratively decomposes the desired angle of rotation $\alpha$ into smaller angles which are easily translatable into hardware. Some notable variations on it are the repeating of the CORDIC principle in multiple stages with special versions of angles in [15] and a simplification of the hardware scheme to prevent scaling in [14]. More versions as well as extensive background can be found in the historical overview in [13].

The CORDIC rewrites the standard rotation matrix in a form with only two angle dependent parts and a scaling and then decomposes the rotation angle into steps of an angle that is easily computed in hardware. The algorithm then runs for a number of iterations, approximating the desired angle further each extra iteration. Each iteration adds or subtracts the remaining angle with the iteration angle. This iteration angle is chosen as the inverse $\tan$ of a power of two. In the rewritten rotation matrix this results in a simple power of two multiplication. Afterwards it scales the end result, which follows from the rotation matrix rewriting.

First the rotation matrix is rewritten using trigonometric identities, shown in equation 6 below.

$$ R(\alpha)= \begin{bmatrix} \cos(\alpha) & -\sin(\alpha) \\ \sin(\alpha) & \cos(\alpha) \end{bmatrix} = \dfrac{1}{\sqrt{1+\tan^2(\alpha)}} \begin{bmatrix} 1 & -\tan(\alpha) \\ \tan(\alpha) & 1 \end{bmatrix} \tag{6} $$

Here angle $\alpha$ is then chosen as follows: $\alpha=\sum^{n-1}_{i=0} d_i\cdot \tanh(2^{-i}) \tag{7}$ with $d_i\in\{+1,-1\}$ the decision variable per iteration and $n$ the number of total iterations. When this choice of $\alpha$ is filled in the expression becomes equation 8 below.

$$ \begin{bmatrix} x_{i+1} \\ y_{i+1} \end{bmatrix} = K_i \begin{bmatrix} 1 & -d_i2^{-i} \\ d_i2^{-i} & 1 \end{bmatrix} \cdot \begin{bmatrix} x_i \\ y_i \end{bmatrix} \tag{8} $$

From this last matrix the iterations can be written as a series of state equations per iteration shown in equation 8 below (taken from [13]). These are then finally directly implemented as a cascade of dataflow nodes that execute these state equations, with tokens holding a vector of all states. The states are described as: $x_i$ and $y_i$ for the real and imaginary part of a vector on the complex plane representing the rotated vector, $\omega_i$ as the remaining angle and $d_i$ as the decision variable. Here $i$ is the iteration number with $i \in {0,1, …, n-1}$

$$\begin{aligned} &x_{i+1}=x_i-d_i\cdot 2^{-i} \cdot y_i \\ &y_{i+1}=y_i+d_i\cdot 2^{-i} \cdot x_i \\ &\omega_{i+1}=\omega_i-d_i\cdot \tanh(2^{-i})\\ &d_i=-sign(\omega_i) \end{aligned} \tag{8}$$

The higher the total iteration number $n$, the more accurate the approximation. As a rule of thumb the accuracy improves approximately 1 bit per iteration. However if the desired angle can be approximated exactly by a number of decomposed angles, then increasing iterations after that will decrease accuracy. The most common scenario for this to happen is with the first iteration $i=0$ which corresponds exactly to $\tanh(2^{-0})=\dfrac{\pi}{4}$. So if the desired angle is 45 degrees the first iteration of the CORDIC already approximates that angle within 1 iteration and any further iterations will decrease the accuracy. Special care should be taken in the implementation to prevent this from happening, since there are indeed twiddle factors which correspond to this angle.

Self Rotation

For the twiddle generator it is necesarry to rotate a complex number with its own angle once or twice, i.e. $\alpha’=2\alpha$ or $\alpha’=3\alpha$. How to do this in a simple and hardware implementable way?
First write the complex number $x+yj$ as a rotation of the base vector $[1;0]$ in the complex plane, this is the standard rotation matrix for $\alpha$, $R(\alpha)$, shown in equation 9 below. This results in a form that can also be deduced from the euler formula [wiki]: $e^{jx}=\cos{x}+j\sin{x}$.

$$\begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} \cos(\alpha) & -\sin(\alpha) \\ \sin(\alpha) & \cos(\alpha) \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} \cos(\alpha) \\ \sin(\alpha) \end{bmatrix} \tag{9} $$

Then the trigonometric identities [wiki] can be used to rewrite the expression into the parts of the input complex number $x+yj$. This is shown for $\alpha’=2\alpha$ in equation 10 and for $\alpha’=3\alpha$ in equation 11.

$$\begin{bmatrix} x'\\ y' \end{bmatrix} = \begin{bmatrix} \cos(2\alpha) \\ \sin(2\alpha) \end{bmatrix} = \begin{bmatrix} \cos^2(\alpha)-\sin^2(\alpha) \\ 2\cdot\sin(\alpha)\cdot\cos(\alpha) \end{bmatrix} = \begin{bmatrix} x^2-y^2\\ 2\cdot x\cdot y \end{bmatrix} \tag{10}$$

$$\begin{bmatrix} x'\\ y' \end{bmatrix} = \begin{bmatrix} \cos(3\alpha) \\ \sin(3\alpha) \end{bmatrix} = \begin{bmatrix} 4\cdot\cos^3(\alpha)-3\cdot\cos(\alpha)\\ 3\cdot\sin(\alpha)-4\cdot\sin^3(\alpha) \end{bmatrix} = \begin{bmatrix} 4\cdot x^3-3\cdot x\\ 3\cdot y-4y^3 \cdot y \end{bmatrix} \tag{11}$$

Conclusion

This concludes the overview of relevant math theory of the FFT part. In the next part of this series the Garrido architecture is discussed, extended and implemented in the floating point domain. In the part after that the conversion to fixed point domain is discussed and a full simulation of the coherent de-dispersion is presented and discussed.

References

[1] J. W. Cooley and J. W. Tukey, “An algorithm for the machine calculation of complex fourier series,” Mathematics of Computation, vol. 19, no. 90, pp. 297–301, 1965.

[2] A. V. Oppenheim, R. W. Schafer, and J. R. Buck, Discrete-time signal processing, Second. Prentice-hall Englewood Cliffs, 1999.

[3] S. Mookherjee, L. DeBrunner, and V. DeBrunner, “A high throughput and low power radix-4 fft architecture,” in 2014 48th asilomar conference on signals, systems and computers, 2014, pp. 1266–1270.

[4] S. Dirlik, “A comparison of fft processor designs.” 2013.

[5] M. Garrido, J. Grajal, M. A. Sanchez, and O. Gustafsson, “Pipelined radix-2^k feedforward fft architectures,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 21, no. 1, pp. 23–32, Jan. 2013.

[6] H. Kanders, T. Mellqvist, M. Garrido, K. Palmkvist, and O. Gustafsson, “A 1 million-point fft on a single fpga,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 66, no. 10, pp. 3863–3873, Oct. 2019.

[7] Hang Liu and Hanho Lee, “A high performance four-parallel 128/64-point radix-24 fft/ifft processor for mimo-ofdm systems,” in APCCAS 2008 - 2008 ieee asia pacific conference on circuits and systems, 2008, pp. 834–837.

[8] J. Wang, C. Xiong, K. Zhang, and J. Wei, “A mixed-decimation mdf architecture for radix- 2^k parallel fft,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 24, no. 1, pp. 67–78, Jan. 2016.

[9] Z. Wang, X. Liu, B. He, and F. Yu, “A combined sdc-sdf architecture for normal i/o pipelined radix-2 fft,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 23, no. 5, pp. 973–977, May 2015.

[10] J. Yang, D. Zhang, Y. Gong, and B. Liu, “A novel design of pipeline mdc-fft processor based on various memory access mechanism,” in 2016 international conference on cyber-enabled distributed computing and knowledge discovery (cyberc), 2016, pp. 62–65.

[11] S. Mookherjee, L. DeBrunner, and V. DeBrunner, “A low power radix-2 fft accelerator for fpga,” in 2015 49th asilomar conference on signals, systems and computers, 2015, pp. 447–451.

[12] M. Garrido, K. Möller, and M. Kumm, “World’s fastest fft architectures: Breaking the barrier of 100 gs/s,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 66, no. 4, pp. 1507–1516, 2019.

[13] P. K. Meher, J. Valls, T. Juang, K. Sridharan, and K. Maharatna, “50 years of cordic: Algorithms, architectures, and applications,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 56, no. 9, pp. 1893–1907, 2009.

[14] Y. Xue and Z. Ma, “Design and implementation of an efficient modified cordic algorithm,” in 2019 ieee 4th international conference on signal and image processing (icsip), 2019, pp. 480–484.

[15] M. Garrido, P. Källström, M. Kumm, and O. Gustafsson, “CORDIC ii: A new improved cordic algorithm,” IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 63, no. 2, pp. 186–190, Feb. 2016.

Graduation project part 2: De-Dispersion

Sun, 22 Dec 2019 00:00:00 +0000

Introduction

This second post of the series about my graduation project focusses on the De-Dispersion methods that are considered and their algorithms. These build on the models presented in part 1. Equation numbers are numbered through in all parts, so references can be towards part 1.

Coherent versus In-Coherent De-Dispersion

De-dispersion is the process of reversing the effects of frequency dispersion of a signal, in this case the signal of a radio pulsar transmitted through the ISM.
The two models presented in part 1 with equation (2) and (4) can each be used as the basis for a de-dispersion method.

In the first method (so called in-coherent de-dispersion) the time shift per frequency channel is calculated with (2). Then this time shift is applied to the whole channel time series. This successfully removes the dispersion between the channels. The advantage of this method is that it has a relatively low computational intensity, which is an important factor due to the scale of pulsar observations. The disadvantage of this is that only the dispersion between the channels is negated, but there is still dispersion left within the channel. Especially for low frequencies and/or high $DM$’s this dispersion can still be very significant [1].

In the second method (so called coherent de-dispersion) the transfer function of (4) is used. Here the input signal is convolved in the time domain with the inverse of the transfer function (4). Since this is a phase only filter this convolution needs to be done on the complex input signal, so before the traditional squaring of the signal done by telescope systems, due to needing the phase information. Using the Convolution theorem [wiki] convolution in the time domain is element wise multiplication in the frequency domain. So the high level flow of coherent de-dispersion then becomes:

Coherent de-dispersion flow diagram

In general coherent de-dispersion is more computational and hardware intensive. Both in higher data rate due to needing the full complex signal, as well as computational intensity due to using convolution instead of simple multiplication in time domain.

Pulsar search

Now lets take a step back and look at the ultimate goal: searching for pulsars. Summarizing from part 1, pulsars are fascinating objects and very valuable for physics research. Observations of them are hampered by the effect of frequency dispersion. This dispersion can be modelled and removed with help of these models. However these require knowing the correct parameter $DM$ for a certain observation. This is not known beforehand and its possible range is vast. Therefor a big part of pulsar search in terms of de-dispersion converges on brute forcing many trails of $DM$’s to find the correct one, if there is a radio pulsar present in the data.
These $DM$ trails are completely independent of each other and over large ranges. There are some proven bounds on the possible range of $DM$ depending on the direction in the galaxy[3], but the search space is very large, depending on the step size between trails. In the below figure the $DM$’s of known pulsars are plotted versus their rotation period $P(s)$. It’s colors correspond to different projects that discovered these radio pulsars. Both axis are a logarithmic scale, showing the vast range of possibilities.

Spread of $DM$ of known radio pulsars with their rotation period $P(s)$

Doing one $DM$ trail reveals nothing about the direction of the correct $DM$. If it is close to the correct one it is possible to see some pulse emerging, it can not be seen if the $DM$ should be higher or lower. To see if the trail has succeeded can be done by visual tools or by automated systems. This pulsar candidate selection is in itself a huge field of research, which this project will not touch. Recent advancements include for example the use of neural networks [2].

The two main methods of de-dispersion each have their own trade-off. Coherent is more accurate but more computational intensive, incoherent vice versa. Combining these can help negate their draw backs. By using coarse grained trails of incoherent de-dispersion and then switching to fine grained de-dispersion when a pulse seems emerging the drawbacks of the two models can be negated. This is called semi-coherent approach.
Below figure shows the smearing due to the combined effects of dispersion within a channel, finite time resolution, and finite DM steps over the full band-width, as a function of DM. Taken from figure 3 in [4]. This illustrates the combined coarse and fine grained de-dispersion, which results in the seawaves like line of semi-coherent. This shows that by combining the methods, de-dispersion can be performed, with less computational intensity.

smearing in semi-coherent dedispersion (figure 3 from [4])

This project will further focus on the computational intensitive coherent de-dispersion and its performance.

Coherent Dispersion Measure Trials (cdmt)

The Coherent Dispersion Measure Trails is an algorithm implementing the coherent de-dispersion method as a convolving synthetic filterbank, that de-disperses with as high as possible throughput of $DM$’s [4]. By splitting the input into independent parts and then utilizing a heavily parallelized computation platform (in the original paper a GPU, or graphics card in popular culture) a high throughput is obtained.
As introduced before, the observations are split into multiple frequency bands. This is done at the telescope input and has nothing to do with the algorithm itself. This first split is thus a split in the frequency domain. The algorithm itself splits the timeseries of a single channel into blocks, this second split is a split in the time domain. This is an important distinction to keep in mind.

To describe the algorithm some parameters need to be introduced, there are $n_s$ input frequency channels, $n_c$ blocks in the timeseries of size of $N_{bin}$ samples with $n_d$ samples overlap per block. This means that each block has $NN=N_{bin}-n_d$ unique samples. These settings are used in $n_{DM}$ trails.
$N_{bin}$ is also the FFT size, thus this influences the frequency resolution of the Fourier transform. Each split block in a timeseries of a channel has $n_d$ samples overlap according to the overlap-save method [wiki] to prevent pollution by the wrap around region of the Fourier transform. The amount of overlap $n_d$ for fully preventing pollution is based on the sample rate of the input and the dispersion sweep within the channel. However to vastly reduce complexity and make the operations more efficient a fixed $n_d$ will be used. This will reduce the efficiency of the algorithm, but makes the time series blocks fully independent from eachother and thus better parallelizable.

In the original description of cdmt [4] the parameters are chosen for optimal use with the LOFAR [5] telescope system. This means $N_{bin}=2^{16}=65536$ (or $64k$) and $n_d=2^{11}=2048$ (or $2k$). To compare the results of the final implementation fairly, these parameters are also used throughout this project.
The $n_c$ depends on the total length of the timeseries $N$ that the algorithm runs on. Then it becomes $n_c=\dfrac{N}{NN}$, $n_c$ needs to be an integer so $N\%NN==0$.
For the Fourier transform the FFT algorithm [6] is used in its radix-$2^2$ form. More on this in later parts.
For the actual dedispersion the transfer function from (4) is inverted by substituting $i$ with $-i$ and then combined with a so-called taper function. This taper function has some anti-aliasing properties but has no real effect on the transform itself. This combination is called the chirp function.

For any given $DM$ the algorithm thus goes through the follow steps for a single frequency channel:

Read the timeseries of the target frequency channel and chop it into $n_c$ blocks of length $N_{bin}$ with overlap $n_d$ according to the overlap-save method.
Compute the chirp function for the given $DM$ with length $N_{bin}$
For each block $n^i_c$ do:
1. convert $n^i_c$ to the frequency domain by computing the FFT
2. multiply the chirp function with the result of the FFT (so in the frequency domain)
3. convert back to the time domain by computing the IFFT of the result of the previous step
4. cut off the overlap padding from the block
Append all resulting blocks together to get the final output signal

Synchronous Dataflow

As mentioned in part 1, the design will be done using Synchronous (or static) Dataflow (SDF) in the library StaccatoLab (embedded in python) created by Kees van Berkel of the TU/e who is my graduation supervisor. This means that this project is a form of Dataflow programming.
Broadly speaking there are two trends to use dataflow programming in computer science.
The first and most used one is to use it to model the flow and dependencies of asynchronous models. This can for example be used to identify bottle necks in platforms with multiple calculation units that need to exchange data. But also algorithms and processes with more general parallel behaviour can be modeled.
The second trend extends this and uses dataflow to do actual programming that can be executed with actual data, instead of only abstract flow. In other words when the dataflow graph is executed with real input data, real output data (as opposed to abstract observations) is the output. So the full model is actually executed with data instead of only as an abstraction to analyse its flow. This will be the method used in this project.
Further implementation details can be found in the StaccatoLab docs [9], but for a basic understanding of the to be presented dataflow graphs the following rules summarize the specific model used in StaccatoLab:

A dataflow graph consists of nodes, that execute actual actions on the data, edges that connect the nodes, in the form of queues, and tokens that flow through the graph according to its rules, consisting of certain data. Execution of a node is also called firing.
Each node is self-timed. This means that a node can fire immediately when there are sufficient tokens on each of its input edges
A graph is clocked, so the firing of firable nodes is synced globally. In practise this means that time is an integer and that a firing can only occur on a multiple of an integer.
Only one output token per firing: at most one token is produced per output by a firing node
The queues in edge have a certain capacity (called slack), a node will only fire if there is either enough slack on its output edges, or it is guaranteed that by simultaneous firing of connected nodes enough slack will be freed up

More information on the fundamentals of SDF see [8] and [wiki].

High Level Simulation

To see how the algorithm works a high level simulation was performed. For such a simulation test data is needed.
The author of [4] has supplied a program to generate a pulsar signal in the same format as real observations. This was used to generate test data by taking this input signal and convolving the entire input signal with a generated inverse chirp function. Inverse chirp function means that the frequency dispersion effect is applied instead of removed, in other words the result is a dispersed signal for a de-dispersion method. This data is used as input to the high level simulation.

In the figure below the dataflow graph of the high-level simulation is plotted.

Dataflow graph of high level simulation of coherent dedispersion algorithm

In this graph each token holds a vector representing a block of data of complex values. A short description of all the nodes:
The src outputs the $n_c$ blocks of size $NN$
The $ovlp$ node then adds overlap of size $n_d$ samples and outputs tokens holding vectors of size $N_{bin}$
The $fft$ node calculates the complex FFT of a block
The $dm$ node generates possible $DM$ values. For this simulation that is only a single one to show the working
The $chirp$ node uses this $DM$ value to calculate the chirp with size $N_{bin}$
The $mlpl$ node does a element wise complex multiplication of the chirp with block $n^i_c$ in the frequency domain
The $ifft$ node converts the block $n^i_c$ back to the time domain by inverse FFT
Lastly the $unpd$ node removes the padding and the $snk$ appends the resulting blocks of size $NN$ together to form the final output

The result from a simulation run is plotted in the figure below. This is an example with aforementioned test data and parameters. Thus the $DM$ is already the correct one and this is done for one channel.

Output of high level simulation

The blue peaks are the simulation results. It can be seen that although these are at the same position as the green original signal, they sometimes vary in magnitude with respect to the original green signal. This is due to the effects of splitting the signal into blocks for de-dispersion, in combination with the generation method of the test data, which is not a fully realistic model. This method was explained earlier.

Implementation Details in parts: step by step

Now that the high level overview of the algorithm is clear, the details can be fleshed out step by step. The major parts for this are:

setting up the channelized input data
setting up the $DM$ trails
implementing the FFT such that it has a high throughput
generating the chirp function
saving the output data

Due to the fact that coherent de-disperion moves the problem largely to the frequency domain, the FFT becomes dominant in the complexity of the implementation work. Therefor the next parts of this series will focus on what turned out to be the largest part of the implementation: the FFT.
In the next part the relevant mathimatical theory for the FFT is discussed and a pipelined radix-$2^2$ feedforward architecture is selected, presented in [7]. An analyses on why this is a good candidate is also explained in the next part.

References

[1] D. Lorimer and M. Kramer, “Handbook of pulsar astronomy.” 2004.

[2] R. P. Eatough et al., “Selection of radio pulsar candidates using artificial neural networks,” Monthly Notices of the Royal Astronomical Society, vol. 407, no. 4, pp. 2443–2450, Jul. 2010.

[3] J. M. Cordes and T. J. W. Lazio, “NE2001.I. A new model for the galactic distribution of free electrons and its fluctuations.” 2002.

[4] C. G. Bassa, Z. Pleunis, and J. W. T. Hessels, “Enabling pulsar and fast transient searches using coherent dedispersion.” 2016.

[5] M. P. van Haarlem et al., “LOFAR: The LOw-Frequency ARray,” vol. 556, p. A2, Aug. 2013.

[6] J. W. Cooley and J. W. Tukey, “An algorithm for the machine calculation of complex fourier series,” Mathematics of Computation, vol. 19, no. 90, pp. 297–301, 1965.

[7] M. Garrido, J. Grajal, M. A. Sanchez, and O. Gustafsson, “Pipelined radix-2^k feedforward fft architectures,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 21, no. 1, pp. 23–32, Jan. 2013.

[8] E. A. Lee and D. G. Messerschmitt, “Synchronous data flow,” Proceedings of the IEEE, vol. 75, no. 9, pp. 1235–1245, 1987.

[9] C. H. van Berkel, “StaccatoLab docs,” to be published..

[10] R. Amirfattahi, “Calculation of Computational Complexity for Radix-2 (p) Fast Fourier Transform Algorithms for Medical Signals,” J Med Signals Sens, vol. 3, no. 4, pp. 217–224, Oct. 2013.

Graduation project part 1: Pulsars Stars

Tue, 17 Dec 2019 00:00:00 +0000

Introduction

This post will be the first in a series about my graduation project. As a final deliverable I will have to write a scientific paper as well, which will be made available for download when its done. The idea behind these blog posts is that they serve as a more expanded version of this final paper for general documentation. The goal is to make it accessible for persons not directly involved with my field of study. It is highly recommended to take a look at the various links and references posted for a better understanding of the core concepts. All posts about my graduation will be in english.
The main concept that is used for designing my work is dataflow architecture [wiki], more specifically Static (or Synchronous) Data Flow [1]. More on this in later parts.
The implementation of this work is mainly done in StaccatoLab, a python library for programming dataflow graphs created by Kees van Berkel of the TU/e who is my graduation supervisor. Since the main goal is to explain my research and results I will not explain the StaccatoLab implementation part in detail. Mainly the plots of the dataflow graphs and its simulation results will be used. More on this in later parts.
These posts are subject to change while I work towards my final graduation and the end of February / beginning of march of 2020.
This first part will go into the details behind the application that this project tackles: processing the signals of pulsar stars.

Pulsar Stars

Radio Pulsars (or Pulsar Stars in popular culture) [wiki] are rapidly rotating heavily magnetised neutron stars. They are invisible to the naked eye, but can be discovered in telescope observations of the universe. In its most basic form they are a dense heavy neutron star, the exact composition being dependent on the relation between the density and pressure of matter composing the star, which is called the equation of state.
Pulsars slowly lose rotation speed over time, this energy loss $\dot{E}$ is called the spin down luminosity. The bulk of this energy is lost due to magnetic dipole radiation and pulsar winds. Only a tiny fraction of $\dot{E}$ is actual radio emission, the type of energy that reaches our earth and can be observed by us [2].
Pulsar stars emit a beam of this radio transmission, which rotates around its axis in a circular fashion. These emissions are observed on our earth as (relative) short radio pulses. These radio pulses can be compared to the effect of light of a lighthouse near a harbour. A lighthouse emits a beam of light that is rotating. From a (relatively) fixed observation point, for example a ship, this is observed as a short light pulse that has a fixed period. For the human eye this is a blinking light.
Below is an artist impression of a pulsar star with its radiation beam.

artist impression of a pulsar star

From this artist impression an important property of the pulse signal can be seen. The angle of the radiation beam is not necessarily a multiple of 90 degrees, as depicted. Relatively with the observation point, in most cases our earth, this means that in the vast majority of cases you only observe one half of the beam. In other words in the pulse signal there is only one pulse per period of the pulsar star. This is important when interpreting the results.

The energy loss $\dot{E}$ causes a slow down of the spinning speed and thus an increase in the period of the pulse signal. However these phenomena are predictable and pulsars therefor have very precise timings. These timings can be used to study relativity theory as well as gravitational waves, relativity theory and many more other fundamental physics phenomena [2].

Pulsar Star Observation

The radio signal of an observed pulse travels light years through space to reach our earth. On its way it encounters the InterStellar Medium (ISM) [wiki]. The ISM consist of cold ionised plasma and can be approximated with known transfer functions from physics theory. The ISM acts on the signal with four distinct effects:

Frequency Dispersion
Faraday Rotation
Scintillation
Scattering

Of these four aspects frequency dispersion is dominant and the main focus of this project. For completeness the other three will be discussed shortly below.

Faraday Rotation [wiki] is caused by differential phase rotation between left and right circular polarisation of the signal. In the ISM this is dependent on a scaled quadratic relation to the wavelength of the light: $ \beta = RM \cdot \lambda^2 $, in which $\beta$ is the angle of rotation in radians, $RM$ is the Rotation Measure constant and $\lambda$ the wavelength.

Scintillation [wiki] is the effect of random changes in magnitude of the signal, caused by electrons and protons in the ISM plasma. It can also be seen by the naked eye in the form of the twinkling light of stars. This effect can be modelled by putting a thin screen half way between the source and observer with a refraction index of $\Delta \mu$ [2] [3].

Scattering [wiki] is the effect in which radiation (such as the emission from pulsar stars) is changed direction due to non uniform effects of the medium it is passing through. In the ISM this is thus due to the changing density of the plasma. [4]

Frequency Dispersion

Frequency dispersion [wiki] is the effect in which signals with different wavelength have different propagation speeds through a non-vacuum medium. When a signal composed of multiple frequencies (and thus wavelengths) is transmitted through this medium, the different frequencies \textit{within} the signal will slowly shift in time from each other. This effect can be observed with the naked eye with light, due to the fact that this effect results in a different refraction angle for specific wavelengths in certain materials. The best example of this is a dispersive prism, as show in the wiki article.
In pulsar signal observations this effect can be seen in the below figure, which is a real observation of a pulsar:
[source]
Here repetitions of the observed pulse can be seen smeared out over the frequency spectrum in time. The arrival of different frequencies (the vertical axis) at different points in time (the horizontal axis, with as unit the pulsar period) is clearly visible.
The result of this is that when looking at the magnitude of the received signal at a telescope the pulse has been smeared out so much that it disappears into the noise floor. This can be seen in the below figure, which shows an example pulsar signal in green and the same signal after frequency dispersion applied in red. This is from simulated data.

magnitude of original and dispersed signal over time

Frequency Dispersion model

Before presenting the mathematical model of the frequency dispersion effect from theory, we need to revisit the ISM and define some parameters for it. The ISM is not uniform, different paths from a pulsar to an observation point do not have the same effects. This is due to the different column density of free electrons along the travelled path by the signal. To model this effect there is the Dispersion Measure($DM$), defined as follows:
$$ DM=\int_0^d\eta_e(l)dl \tag{1}$$
Here $\eta_e(l)$ is the amount of electrons at a point along the path, $d$ the distance of the path and $l$ the path itself. This $DM$ is the parameter that will be used in the model of Frequency Dispersion in the ISM. It is not known a priori to an observation of a radio pulsar.

There are multiple ways to model the Frequency Dispersion in the ISM, of which two will be presented. These are tied to the two main algorithms of removing the dispersion, so called de-dispersion, whom will be presented in the next part in this series. These models both work with the concept of so called frequency channels. These are created by splitting the frequency spectrum into smaller bands of equal size. For example one of the largest projects in this field, the SKA [wiki], splits a bandwidth of 300 MHz into 4096 frequency channels for the SKA1-Mid system for pulsar search [7].

In the first model for a channel between $f_1 < f < f_2$ the time delay of the arriving frequency channel can be described by: $$ \Delta t = 4.15\times{10}^{-6} \times DM \times (f_1^{-2} - f_2^{-2}) \tag{2}$$
The second model describes the dispersion as a phase only filter in the following form in the frequency domain:
$$V(f_0+f)=V_{int}(f_0+f)\times H(f_0+f) \tag{3}$$
here $V(f)$ and $V_{int}(f_0+f)$ are the observed and emitted signals around a center $f_0$ and the filter transfer function $H(f)$.
This transfer function is then of the following form:
$$H(f+f_0) = exp\bigg[\dfrac{2 \pi \cdot i \cdot f^2 \cdot k_{DM} \cdot DM}{f_0^2(f+f_0)}\bigg] \tag{4}$$
here $f_0$ is the center frequency of a frequency channel and $f$ is the frequency offset within the channel. $DM$ is the dispersion measure as discussed earlier and $k_{DM}$ is the measured constant of proportionality of the ISM model. Please note that these models follow the physics convention in terms of writing.

For more background information behind these models see [2].

The parameter $DM$ is not known a priori when looking at an observation, this means that many trails over a broad range have to be done in the de-dispersion method to search for the correct one. To select the correct range for these, models of the Galactic electron density can be used to estimate the maximum $DM$ for a certain range and direction in the galaxy [6].
Selecting the right $DM$ through many trails of de-dispersion in an efficient and fast way, so that new pulsars can be found in astronomy observations, is the main application that this graduation project will tackle. In the next part of this series the main methods of de-dispersion algorithms will be discussed. Furthermore the implementation direction will be introduced with its trade-offs.

References

[1] E. A. Lee and D. G. Messerschmitt, “Synchronous data flow,” Proceedings of the IEEE, vol. 75, no. 9, pp. 1235–1245, 1987.

[2] D. Lorimer and M. Kramer, “Handbook of pulsar astronomy.” 2004.

[3] K. R. Lang, “Interstellar Scintillation of Pulsar Radiation,” vol. 164, p. 249, Mar. 1971.

[4] T. J. W. Lazio, “Interstellar Scattering,” in Future directions in high resolution astronomy: The 10th anniversary of the vlba, asp conference proceedings, vol. 340. Edited by j. Romney and m. Reid. San francisco: Astronomical society of the pacific, 2005., p.419, vol. 340, J. Romney and M. Reid, Eds. 2005, p. 419.

[5] T. U. of Manchester Jodrell Bank Observatory, “Observations of pulsars.”.

[6] J. M. Cordes and T. J. W. Lazio, “NE2001.I. A new model for the galactic distribution of free electrons and its fluctuations.” 2002.

[7] L. Levin et al., “Pulsar searches with the ska,” Proceedings of the International Astronomical Union, vol. 13, no. S337, pp. 171–174, Sep. 2017.

De Energie Transitie en Nederland

Mon, 29 Apr 2019 00:00:00 +0000

Klimaat en de politiek

Global warming: het is een hot topic de laatste tijd. Het speelde een grote rol in de afgelopen Tweede Kamer verkiezingen (2017), met Jesse Klaver van Groenlinks als voornaamste aanjager. En met success, GroenLinks won flink. Mede hierdoor werd er vanaf 2018 een groot polder project gestart: het Nederlandse klimaat akkoord.
Ook bij de afgelopen Provinciale Staten (en dus indirect de eerste kamer) verkiezingen was het wederom een hot topic. Dit keer was het resultaat anders, de grote winnaar van deze verkiezingen (FvD) is groot tegenstander van het klimaat akkoord.

Het doel van het klimaat akkoord (die nog niet definitief gesloten is) is een reductie van CO2 uitstoot van 49% in 2010 t.o.v. 1990. Dit is onderdeel van de internationaal gesloten akkoord van Parijs. Een van de onderdelen om dit te bereiken is Energie, of om precies te zijn de Energie Transitie. Het doel hiervan is simpel samen te vatten, maar lastig uit te voeren:
“In 2030 komt 70 procent van alle elektriciteit uit hernieuwbare bronnen”

Een veel gehoorde reactie op deze plannen is dat mensen denken dat Nederland het al heel goed doet met hernieuwbare energie, en willen er dus geen nieuwe grote investeringen erin doen. Ik had al eens gehoord dat dit verre van het geval is en daarom ben ik eens even in de cijfers gedoken.

Even een kleine technische note om mee te beginnen. Alle cijfers in deze post komen van het CSB statline programma. De meeste energie statestieken m.b.t. hernieuwbare energie wordt pas bijgehouden sinds begin jaren 90, dit verklaart de xas domein van veel van de gebruikte grafieken. Alle data is verwerkt in python, de source hiervan kan je vinden mijn github.

Energie Productie

Wat mij meteen opviel aan deze plot, is de relatieve bescheiden deelname van windenergie. Er wordt al jaren veel aandacht in de media en politiek besteed aan de voors en tegens van wind turbines en veel provincies pronken er mee als ze weer een nieuw park neergezet hebben. Voor extra duidelijkheid staan in onderstaande tabel de productie windenergie t.o.v. totaal energie productie uitgedrukt in procenten van de laatste 5 data punten.

Jaar	%
2013	5.58
2014	5.61
2015	6.86
2016	7.09
2017	9.01

De deelname van wind is dus vrij bescheiden, een kleine 9% en vrij traag groeiend. Voor wat extra duidelijkheid hieronder een donut chart van het laatste jaar waar CBS data van heeft, 2017.

De meeste categorie spreken voor zich, alleen wat is precies een steg-eenheid? Een snelle blik op wikipedia leert je dat dit een gecombineerde Stoom- en gasturbine is. Het maakt gebruik van een slimme combinatie om het rendement op het gas stoken te verhogen.

Uit bovenstaande plots rijst al een wat somber beeld op als je het klimaat akkoord in je achterhoofd houdt. Het overgrote deel van de productie is verre van hernieuwbaar en komt uit stoom en gas.

Energie Balans

Voordat we doorgaan door in te zoomen op de hernieuwbare energie bronnen is het belangrijk om ook de energie balans kort te beschouwen. Over de afgelopen 30 jaar is Nederland altijd een netto importeur geweest van energie, de balans per jaar is weergegeven in onderstaande grafiek.

Waarschijnlijk neemt het klimaat akkoord als definitie dat hernieuwbare energie als percentage van verbruik i.p.v. eigen productie telt. Dit betekend dus dat er ook nadrukkelijk gekeken moet worden naar het buitenland en waar onze geimporteerde energie vandaan komt.

Energie Productie uit Hernieuwbare Bronnen

Dan nu een blik op de cijfers voor hernieuwbare bronnen. In onderstaande figuur is het aandeel van hernieuwbare energie in productie binnen Nederland geplot. Links onderverdeelt in de verschillende categorie van herkomst, rechts als percentage van het totale verbruik in Nederland. Met andere woorden dit gaat over de energie productie die ook binnenlands geconsumeerd wordt. Dit is een specifieke keuze die volgens het CBS “voortvloeit uit Europese afspraken”.

Hieruit blijkt dus dat er nog wel wat te winnen valt. In het laatste jaar waar cijfers bekend van zijn, 2017, was het percentage slechts 14.33%. Dit is a far cry van de targets die het kabinet gezet heeft. Echter deze targets gelden vooral voor het percentage hernieuwbare energie van het daadwerkelijk verbruik van binnenlandse energie. Oftewel als je veel “vieze” stroom exporteert en veel groene stroom importeert, kan je dit nog wat verbeteren. Maar laat dit cijfer inderdaad wat gunstigers zien?

Energie Verbruik uit Hernieuwbare Bronnen

Het antwoord hierop is te zien in onderstaande grafiek. Dit laat het aandeel van hernieuwbare energie in de binnenlandse energie consumptie zien, uitgesplitst over enkele categorieen.

Dit laat een triest plaatje zien. Met nog 11 jaar te gaan tot de gestelde target van 70% hernieuwbare energie van energie consumptie, laat Nederland een magere 6.6% zien over 2017. Dit is wel een verbetering t.o.v. de 1.22% waar we mee begonnen in 1990, maar het is bar weinig. Het laat zien dat er nog veel werk aan de winkel is en dat het klimaat akkoord grote en aggresieve maatregelen nodig heeft om deze targets te halen. Het antwoord op mijn gestelde vraag is dus nee, dit laat een nog triester beeld zien dan de energie productie.
Wat wel leuk is om te zien is de opkomst van de electrische vervoersmiddelen. Beginnend in 2006 is deze ieder jaar vertegenwoordigt in de verbruiks statestieken.

Conclusie

Al met al staat Nederland er dus niet best voor om haar doelen te halen. Een magere 6.6% hernieuwbare energie in het binnenlands energie verbruik laat zien dat er nog veel werk aan de winkel is. Nederland ligt hiermee ook achter op het Europese gemiddelde van 17.4% zoals geraporteert door het EEA, European Environment Agency. Er is dus werk aan de winkel en ik hoop dat de huidige politiek dit ter harte neemt! Dit zal echter alleen gebeuren met betere bewustwording van de stemmende Nederlanders. Alleen zij kunnen echte verandering pushen.

Deploying Django Channels Application

Wed, 24 Apr 2019 00:00:00 +0000

Django

For quite some time python has been one of my favourite programming language. Although its not the best choice for all use cases, the right tool for the right job after all, it has a very nice flow to it when programming. Things are easily done, the documentation quite good and there is a lot of help available through google.

One of the best examples for the excellent ecosphere of libraries that can be used with python is Django, a framework to create web application. With my company I use this exclusively. We also use the Django extension Channels, which extens django with support for things like websockets, as well as preparing it for a better codebase in the future. Channels will eventually be merged into the core Django.

I will not get into why Django and Channels is so awesome in this post. But I do get questions about the best way to deploy it, and have to do this often myself. Therefor I have made this short post to explain this.

Deployement Structure

The standard way to set it up is by running the python behind a reverse proxy and use redis for the channels backend routing. This way each component can do what it does best. The reverse proxy in the form of a webserver can handle things like http2, ssl and other standard web things. It can also directly server the static files. The python process will then handle all application things. Lastly a fast key value store like redis can synchronize the backend as well as provide caching support. Redis needs almost zero configuration, especially if yo urun seperate systems on seperate machines, like most of my own use cases.

For the reverse proxy I use Nginx and for the python process daphne. Instead of Nginx, you can also use apache. For the python process django channels provides daphne with a lot of documentation, but you can use any ASGI compatible service. I use dahpne because of the support on it and wrap it into a systemd service.

Obviously you also need a database. I usually use postgresql. Database setup is pretty standard so I’m gonna leave it outside of this post.

In the example config below <projectname> is assumed to be the name of the folder in which the routing, settings etc files are stored. This is automatically generated by django-admin upon creation of a django project.

Django Application Setup

Make sure your django application follows the direction of the django channels deployement page found here. Most importantly you need a myproject.asgi file as well as setup CHANNELS_LAYERS correctly.

If you setup redis on default settings (on localhost) this will suffice:

# channels, a new and better way to run Django including websockets.
ASGI_APPLICATION = '<projectname>.routing.application'
CHANNEL_LAYERS = {
    'default': {
        'BACKEND': 'channels_redis.core.RedisChannelLayer',
        'CONFIG': {
            "hosts": [('127.0.0.1', 6379)],
        },
    },
}

For the asgi setup the follow minimal example is enough for most applications:

import os

import django
from channels.routing import get_default_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<projectname>.settings")
django.setup()
application = get_default_application()

For setting up routing etc consult the django channels docs, that is out of the scope of this short deployement guide.

Daphne Setup

To serve the above django application I use daphne. This is quite easy, simply go to the folder of the application and execute the following:

daphne -u /tmp/daphne.sock <projectname>.asgi:application

This will start daphne in non daemon mode and open a local sock opbject for requests. The reason I do it in non daemon mode is because I can then easily wrap it into a service file. All messages from daphne are printed to stdout and can be easily read then from the systemd journal.

A minimal systemd service will be something like this:

[Unit]
Description=daphne service to run django application
After=network.target
After=postgresql.service
After=nginx.service

[Service]
Type=simple
RuntimeDirectory=daphne
PIDFile=/run/daphne/pid
User=django
Group=django
WorkingDirectory=<application root>
ExecStart=/usr/bin/daphne -u /tmp/daphne.sock <projectname>.asgi:application
ExecStop=/bin/kill -s TERM $MAINPID
[Install]
WantedBy=multi-user.target

If you installed everything into a python virtual environment (recommendable!) then simply replace /usr/bin/daphne by the location of daphne inside of the virtual environment. It will automatically pick this up and use the virtual environment.

Reverse Proxy setup

For the reverse proxy setup I use nginx. For nginx it is recomendable to use a setup in which every site has a sub config in sites-enabled folder, this is however up to you. I will share here an example config for the server block for the django application below.

upstream app_server {
    server unix:/tmp/daphne.sock fail_timeout=0;
}

server {
    listen 443 ssl http2;
    server_name <hostname>;
    
    client_max_body_size 20M; # this is optionally, I usually put it very big in nginx and do proper size checks in the application
    
    location /static/ {
        sendfile on;
        location ~* \.(?:ico|css|js|gif|jpe?g|png|svg|woff|bmp)$ {
            expires 7d;
        }
        alias <application static folder>;
    }
    
    
    location / {
        proxy_pass http://app_server;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header X-Forwarded-Proto "https";
        proxy_set_header Connection "upgrade";
        add_header Referrer-Policy "no-referrer-when-downgrade";
    }
}

Starting Setup

Simply start redis nginx and the database in the default way, for example with systemd. Daphne can be started with the default command for a custom service:

systemctl start daphne

Dont forget to enable all services that are necesarry on your server to auto start on boot, for example with

systemctl enable daphne

And thats it! Have fun with you live production ready Django Channels Application!

Gesprekken met mijn oma: Pensioenkosten

Sun, 24 Feb 2019 00:00:00 +0000

Mijn opa en oma wonen vrij dichtbij mij in de buurt. Met mijn familie gaan we daar dan ook geregeld heen. Mijn oma is een intelligent mens en al mijn hele leven herinner ik me haar als iemand die nadenkt over de wereld om haar heen en daarover discusieert. Enige tijd geleden kwamen we op een zondag op een gevoelig onderwerp: de pensioenen. Deze zijn veel in het nieuws omdat deze alleen maar lijken te versomberen terwijl de economie wel goed draait op het moment. Ook de verhoging van de pensioen leeftijd is altijd een hot topic.

Een van mijn statements van die zondag was dat het pensioenstelsel zoals die in de jaren 50 is ontworpen (de AOW wet is in 1956 ingegaan [wiki]) niet meer houdbaar is voor mijn generatie vanwege de bevolkings ontwikkelingen. Hier was niet iedereen om de tafel het mee eens. Dit prikkelde mijn interesse om dit te proberen onderbouwen met harde cijfers.

Even een kleine technische note om mee te beginnen. Alle cijfers in deze post komen van het CSB statline programma. Aangezien het CBS data over arbeidsdeelname pas sinds 1969 beschikbaar heeft starten alle grafieken vanaf daar en niet vanaf het begin van de AOW. Alle data is verwerkt in python, de source hiervan kan je vinden mijn github.

Pensioenbasis: de AOW

Het pensioen stelsel in Nederland is vrij uitgebreid, maar om de trend te laten zien als argument beperk ik mij hier even tot de basis daarvan: de AOW. De Algemene Ouderdomswet (AOW) is in Nederland sinds 1956 de basis van de pensioenen. Het zorgt voor een basis pensioen voor iedereen boven de pensioens gerechtigde leeftijd, die op dat moment werd vastgesteld op 65. In 2012 werd dit door het toenmalig kabinet gewijzigd naar een oplopende leeftijd afhankelijk van je geboorte jaar, in reactie op het probleem wat ik ga beschrijven.

De AOW wordt grotendeels gefinancierd door de AOW premies, deze worden betaald door iedereen in Nederland als onderdeel van de belastingen zolang je werkt. Mensen die AOW ontvangen dragen hier niet meer aan bij, wat dus heel kort door de bocht betekend dat mensen die op dit moment werken de AOW financieren van de mensen die op dit moment AOW ontvangen. Met andere woorden de huidige generatie betaald voor de generatie boven hen. Dit met het idee dat de generatie na hen hetzelfde zal doen voor hen. En hier zit dus de kern van het probleem.

Bevolkingsgroei

Laten we even beginnen met de basis, de algemene bevolking in Nederland stijgt al jaren, van 12.8 miljoen in 1969 naar iets meer dan 17 miljoen in 2017. Dit is in het onderstaande figuur weergegeven.

Bevolkingsontwikkelingen

Als we even inzoomen op deze groei zien we twee voor de AOW relevante ontwikkelingen.

Allereerst groeit ook de groep mensen die werken en dus AOW premies betalen. Niet alleen in absolute zin vanwege de groeiende bevolking, maar ook in relatieve zin door factoren zoals de vergrote arbeidsparticipatie onder vrouwen. In de onderstaande figuur kan je links de absolute stijging zien en rechts is het percentage van beroeps bevolking op de gehele bevolking uitgedrukt.

Hier kan je zien dan de beroepsbevolking is gestegen van 5.2 miljoen in 1969 naar iets meer dan 9 miljoen in 2017. Relatief uitgedrukt is het percentage beroepsbevolking op de volledige bevolking van 40.91% in 1969 naar 52.79% in 2017 gestegen.

Hier staat een andere zorgwekkende trend tegenover. Als gevolg van het feit dat de baby boom generatie nu de “ouderen” fase in gaat vergrijst de Nederlandse bevolking. Dit is te zien in de onderstaande figuur van de absolute(links) en relatieve(rechts) aandeel van 65+ in de Nederlandse bevolking.

Dit laat een absolute stijging zien van 1.2 miljoen in 1969 naar 3.2 miljoen in 2017, maar veelzeggender een relatieve stijging (ten opzichte van de totale bevolking) van 10.01% in 1969 naar 18.49% in 2017. [//]: # (TODO: grafiekje prognose groei van 65+)

Wat betekend dit? Dat zowel het deel van de bevolking dat betaald voor de AOW als de AOW ontvangers gestegen is, maar de ontvangende groep relatief veel harder. Laten we dan nu kijken naar het effect hiervan op de AOW cijfers zelf.

De AOW cijfers

Laten we dan nu de cijfers van de AOW zelf erbij pakken. Allereerst wederom de basis, hieronder staat de totaal aantal AOW uitkeringen in Nederland. Deze is gestegen, weinig verassend als je kijkt naar de stijgende hoeveelheid 65+ mensen. Er moet echter wel een heel belangrijke kanttekeing geplaatst worden bij deze data, zoals het CBS zelf al aangeeft. Per 1 april 1985 is de AOW voor mannen en vrouwen gelijkgetrokken, conform de EG (de voorloper van de EU) richtlijnen. Hierdoor wordt geen onderscheid meer gemaakt tussen gehuwde mannen en vrouwen, zij hebben beide recht op AOW, dat was voor 1985 niet het geval. Deze gelijkheid is er dus pas sinds een kleine 34 jaar, vrij recent als je erover nadenkt. Deze verandering kan je duidelijk zien in de sprong die de grafiek maakt rond 1985.

Dit brengt mij naar de laatst en belangrijkste grafiek. Om de bevolkings ontwikkelingen echt goed uit te drukken in de gevolgen voor de AOW kan je kijken naar de graadmeter hoeveel werkende er per AOW uitkering is. Hiervoor delen we de beroepsbevolking door het aantal AOW ontvangers. Hierbij negeren we voor het gemak even veranderende inkomens ongelijkheid omdat we ervan uit gaan dat de maatregelen van de overheid om op een eerlijke manier AOW te heffen ook echt 100% eerlijk zijn (een ander discutabel onderwerp). Dit ziet er als volgt uit:

Dit is dus hard gedaald, van naar 5.06 in 1968 naar 2.64 in 2017. Echter is er de aanpassing van de AOW in 1985 zoals genoemd, dit is dus niet een helemaal eerlijke vergelijking. Daarom kijken we naar de relatieve trend voor en na de wijziging voor een goeie vergelijking.

tijdsperiode	voor	na	procentueel
1969-1985	5.06	4.47	-13.20 %
1986-2017	3.39	2.64	-28.41 %

In beide perioden zien we dus een sterke daling, met na 1985 een zeer sterke daling. Dit betekend dus dat relatief gezien de hoeveelheid werkende mensen ten opzichte van de hoeveelheid mensen die een AOW ontvangen krimpt.

Conclusie

De jongere generaties (zoals die van mij) moeten dus relatief meer betalen voor onze ouderen dan de generaties voor ons, door de manier waarop onze AOW opgezet is in combinatie met de toenemende vergrijzing. Met de verwachte voortzettende vergrijzing van de Nederlandse bevolking zal dit alleen maar erger worden. Het is niet dat ik niet wil betalen voor mijn opa en oma en hun generatie genoten. Graag zelfs, ze hebben ons land opgebouwd en samen met de generatie van mijn ouders (die ook langzaam richting de AOW groep opschuiven) de mogelijkheden geschept voor mijn bestaan. Echter moet er wel serieus gekeken worden naar hoe dit systeem betaalbaar gehouden kan worden, zodat mijn en latere generaties niet bezwijken onder de toenemende lasten. Het verhogen van de AOW leeftijd waar nu naar gegrepen wordt is een optie, of dat de beste is zullen we de komende tijd gaan zien.

Welkom

Sun, 24 Feb 2019 00:00:00 +0000

Welkom op mijn persoonlijke blog! Hier deel ik een verzameling van gedachtes en wat data onderzoeken die ik uit hobby interesse uitgevoerd heb.

Contact

Mon, 01 Jan 0001 00:00:00 +0000

For questions, compliments or other forms of messages you can reach me at my personal email frank@fboerman.nl.

For business inquiries you can reach me at my personal business umbrella frank@amunanalytics.eu. For registration information on my business see https://amunanalytics.eu/.

You can find me on linkedin here and my twitter (or X) here. I am also active on github here.

Over Mij

Mon, 01 Jan 0001 00:00:00 +0000

Hallo ik ben Frank en welkom op mijn persoonlijke website! Een beetje over mezelf, ik ben afgestuurd Msc student Electical Engineering (specializatie Embedded Systems) aan de TU/e en woon tegenwoordig in Nijmegen.
Ik werk bij TenneT als process specialist in de System Operations Transport (SOP-TRS) afdeling.

Mijn hobbies zijn allerlei software projectjes, tegenwoordig vooral het maken van handige visualizaties en het lezen van geschiedenis boeken. Verder wandel ik graag in de natuur en breng ik graag tijd door met mijn vriendin en onze katten.

In mijn studententijd heb ik veel gedanst en was ik actief lid bij ESDV Footloose. Verder run ik sinds mijn studententijd met een vriend een software bedrijfje, Kolibri Solutions. Daarnaast heb ik ook een eigen ZZP bedrijfje Amun Analytics.

Op deze website deel ik een verzameling van gedachtes en wat data onderzoeken die ik uit hobby interesse uitgevoerd heb.

Mijn linkedin kan je hier vinden, mijn CV is hier te downloaden.

Ik ben ook actief op meerdere plekken op github: [1] [2] [3]

Ik ben te bereiken op frank@fboerman.nl.

Disclaimer: alles op deze website zijn mijn persoonlijke meningen en uitingen.