Fake Facts: the Justice Scoreboard of the Directorate General for Justice and Consumer protection

The European Commission has a justice policy which involves all national judiciaries, whose functioning is monitored. However I argue that this monitoring is done unsatisfactory. Justice is an important subject in the European Union. For example, trade and crime do cross borders and economic competition contracts and regulations need the guarantee of enforceability. It is obviously unavoidable that a lot of European law is involves regulation and law enforcement. National judiciaries play an important role in the enforcement of EU-law. And it seems perfectly justified that the European Commission monitors the functioning of those national judiciaries. The monitoring of that function is done through the so-called Justice Scoreboard. This Justice Scoreboard, however, is a methodologically inadequate device to evaluate the functioning of national justice systems. The main problem of the Justice Scoreboard is that the data it is based on are overall not reliable and based on nationally – differently! – defined statistics. Consequently, national numbers cannot adequately be compared, which leads to a misleading presentation of the data in bar charts.

The European Commission has developed a policy framework under the header ‘Building a European Area of Justice’. This policy framework is based on the Charter of Fundamental Rights of the European Union, in particular article 47 which guarantees an effective remedy to infringements of rights guaranteed in EU law. Therefore, the Commission has installed a framework to safeguard the rule of law in the European Union, which involves an infringement procedure based on article 7 TEU, for “situations where there is a systemic breakdown which adversely affects the integrity, stability and proper functioning of the institutions and mechanisms established at national level to secure the rule of law”.

At the time of writing of this blog, the European Commission considers activating the infringement mechanism against Poland and Hungary. Apart from this supervisory competence, the European Commission supports judicial training, and it also supports the effectiveness of national justice systems via the European Semester, explicitly stating the economic relevance of effective justice systems. The European Semester is the EU’s annual cycle of economic policy coordination. Last, but not least, the European Commission deploys the Justice Scoreboard, in order to, as they say on page 1 of the 2017 justice scoreboard:
“assist the EU and Member States to achieve more effective justice by providing objective, reliable and comparable data on the quality, independence and efficiency of justice systems in all Member States.”

A critical review of the Justice Scoreboard leads to a number of observations. First, it is important to note that this is an evaluation tool, based on performance measurement. Second, what is measured in the Justice Scoreboard is, for example, timeliness of justice per case category, which is based on the judicial systems evaluations of the Commission for the Enhancement of Efficiency of Justice (CEPEJ) of the Council of Europe. However, the data presented here have been acquired in a non-reliable way, because they are delivered by the ministry of justice contact points, and the gathering of those national data is an immense job in itself. This leads to subjectivity in answering the CEPEJ questionnaire and unexplainable differences in data from the same country over time.

Third, the case categories are based on the aggregation of cases that are defined as, for example civil litigious or non-litigious. Apart from the fact that this divide may differ between member states, civil litigious cases can be of many different kinds, and also rules of procedure do differ between member states. In addition, the registration of when a case has started and when a case has ended differs per member state. Any comparison across borders as seems to be done in the Justice scoreboard appears precarious. Finally, where the CEPEJ Judicial Systems evaluations explain in great detail that the quality of data gathered should make the reader cautious with interpretation of those data, the Justice Scoreboard most recent version only refers to the CEPEJ Judicial Systems evaluation reports. These methodological problems have been explained extensively in by Adriani Dori, , and by Alina Ontanu, Marco Velicogna and Francesco Contini. Given the fact that the data in the Justice Scoreboard show scores per country for 2-4 years, the development of scores can at best be compared per country. An international cross border comparison is highly questionable.

It is the fate of a lot of performance measurement tools that aim at informing political decision makers, that most of the data are effectively never used. Important changes originate from elsewhere, as can be shown by the example of the letters the European Commission has sent to Poland and Hungary to stop thwarting the independence of their judiciaries. It can also be demonstrated by the most important changes in the Netherlands Judiciary, such as the changes in the judicial map and the development of e-filing for civil and administrative cases. They are not based on the performance measurement of the quality management system RechtspraaQ as one would expect, but on changes in the environment of the Judiciary, such as the recent economic crisis, or changes in the organization of the public prosecutions office and the police.

My advice to policymakers who develop monitoring instruments is to keep things as simple as possible. For judiciaries, measuring incoming cases, cases decided in a time span, and number of pending cases at the beginning of the timespan and at the end of the timespan allows for a calculation of the ratio between incoming cases and cases decided, the disposition ratio. The results may be compared with a pre-defined standard for timeliness of judicial decision-making.

When cases are not similar across national borders, which often isn’t the case, it may still be possible to show national or local disposition ratios. However, comparing disposition ratios across borders still would not be right, not even if the registrations and the data gathering would be very reliable. A serious EU political supervision of judiciaries’ effectiveness in the member states cannot be based on a comparison of such numbers. The only escape here would be to legislate uniform rules of procedure in civil and administrative cases EU-wide, as this would allow similar cases and case categories in different countries. And that is not going to happen within the foreseeable future.

In so far, policy and effectiveness evaluation studies for monitoring judicial systems at the EU level could better follow a mixed method approach, using qualitative assessments for transnational comparisons. Unfortunately, when recently discussing methodological difficulties of performance measurement for EU policymaking in the Justice field with EU civil servants, a leading civil servant declared loudly (and in an intimidating fashion) that she did not want to hear all this, because they just had the relations with their national contact points (page 2. Justice Scoreboard 2017) in order. Even though we received an apology later on from one of her colleagues, it still seems as though also some EU policy makers are driven more by fake performance measurement facts than by what really matters for citizens.