Annex A. Methodology applied for this study in Ireland
The methodology used for the Ireland judicial workload study builds on the well-tested methodologies applied in other countries, while being adapted to the Irish courts. The weighted workload model is grounded in the understanding that different types of court cases vary in complexity, and therefore differ in the time required by the judge. For example, a typical assault case requires more time for a judge to prepare, hear and decide than the average theft case. The model generally calculates judicial time and position needs based on each court’s total annual workload, and requires three core data elements:
1. Case filings, or the number of new cases of each type opened each year.
2. Case weights, which represent the average amount of judge time required to handle cases of each type over the life of the case.1
3. The “judge year value”, or the average amount of time a judge has available for case-related work in one year.
First, to calculate case weights (i.e. average hours needed per case per judge), the average time needed for all case action types per case must be collected and calculated from the time study data (i.e. sum of time needed for all case action types per case category divided by number of cases processed by case category per judge). By developing separate case weights for different case types, the study can reflect the variable case complexity and the different amounts of judge time needed for handling different types of cases.
Second, the total annual case-related workload is calculated by multiplying the annual filings for each case type by the corresponding case weight, then summing the workload across all case types. The time judges need to handle all non-case related work (e.g. other administrative tasks, co-ordination meetings, community outreach, work-related travel time) is then added.
Annual workload (per case type) = annual filings x case weight
Total annual workload = sum of annual workloads + non-case related work
Third, each court’s workload is then divided by the judge year value to determine the total number of full-time equivalent judicial positions needed to handle the entire workload.
FTE judges required = total annual workload / judge year value2
This study represents the first application of the weighted workload methodology in Irish courts. To adjust this model to the Irish court system, significant preparation steps had to be completed before detailed data collection instruments could be designed.
In order to define the full scope of the study, it was important to clarify the levels of courts that would participate in the workload analysis. It was decided that all court levels would participate except the Supreme Court, as its jurisdiction and role differ from other courts. With respect to the specialised courts, the judges decided to exclude the Drug Treatment Court as it has a unique focus, limited caseload and is adequately staffed.
To ensure that the study design, implementation, analysis and resulting recommendations were fully reflective of court operations and their environment, the OECD team requested that a working group be formed. The Liaison Group consisted of one judge from each of the participating court levels, and initially one, then two representatives from the Courts Service.
The responsibilities of the judicial members of the liaison group included:
Choosing the court levels and study methodology to be applied in consultation with the Court Presidents and Chief Justice.
Choosing the exact timing for the data collection implementation at each court level.
Advising the OECD team on the definitions of case types and case-related and non-case related events to be used during the time study.
Supporting the development and testing of the different data collection forms and data collection guides needed for each court level.
Co-ordinating with their Court Presidents on the number and selection of a representative group of judges to include in the time study data collection and Delphi vetting process.
Reviewing and agreeing to the definitions used to develop the core data elements needed to define the judicial year value, e.g. FTE calculation.
Distributing and collecting electronic time study data sheets and sending them to the OECD study team.
Reviewing the initial results of the time study and the Delphi vetting process.
Advising and commenting on the core set of recommendations resulting from the study.
Due to heightened considerations about the anonymity and privacy of the study participants, the liaison judges took on additional responsibilities to train participating judges from their relevant courts, collect completed timesheets and assist the study team in following up on data entry issues.
The responsibilities of the Courts Service liaisons were to:
Advising the judges and OECD team on the availability of required case data and options for compiling data as needed.
Providing the administrative staffing and processing information required for the study.
Collaborating with the study team and liaison judges on identifying which case data can be used for the time study data collection to ensure the full caseload of the judges was reflected.
Providing the OECD team with the needed case data in electronic version as determined during the study design process from Courts Service and other sources.
Providing feedback on initial study recommendations related to court case administration and related Courts Service operations and efficiency issues.
The timing of the study was driven by the government’s creation of a Judicial Planning Working Group to inform requests for judicial positions. This required some adjustments, particularly due to the unprecedented virtual approach to the study3 in view of the continuing COVID-19 health restrictions. While electronic data collection tools had been used previously for such studies, as well as a variety of information exchange and virtual meeting options, these tools were used to complement rather than replace site visits, observations and in-person meetings. Given that the pandemic had increased judges’ access to and familiarity with virtual communications, the courts decided that it could be feasible to conduct a fully virtual workload study in Ireland.
The significant use of virtual hearings combined with solid social distancing concepts had allowed the Irish courts to start operations again after a few months of lockdown in 2020. By the time the study started, many court operations were conducted closer to pre-pandemic levels. This was essential for ensuring that the data collection could capture case data, court operations and related judicial time spent that was sufficiently reflective of court operations in general.
While the initial time period for the study was envisaged for five months – not without precedent internationally (National Center for State Courts, 2016[1]) – the duration of the study had to be extended given that it had to start at the end of the judicial year, which is a very busy month for all courts, and therefore certain case types and work actions would not be scheduled during this time. The study schedule also had to accommodate the fact that July and August are official court vacation time in Ireland. As a result, operations in July do not reflect the typical judicial workload, and no data could be collected during July and August. While judges are partially working throughout this time, no hearings, except for emergency matters, are conducted. A compromise was found for the Circuit Court, High Court and Court of Appeal by starting data collection in July 2021 for one week, and then resuming data collection during the first two weeks of October 2021, the start of the new Court Year. The District Courts, which usually start their hearings in September, agreed to begin their three weeks of data collection during the last week of September 2021.
This timing, combined with a very tight timeline for completing the study, meant that the time study data collection was limited to the shortest time period possible, while still reflecting a representative workload, as confirmed by the judges in the Liaison Group. While typically, workload studies are conducted over four to six weeks; this study was conducted over a three-week timeframe. While this timeframe was not ideal and shorter than usual, it nonetheless can be considered reliable as the study period was representative of the annual workload. In addition, the Delphi study ensured that any time collection data gaps were appropriately addressed in the time estimates.
The timeline for conducting the Delphi study was dependent on the timing of the time data collection, as its results indicate the average time spent per judge on individual case processes by case category. This information was needed for the Delphi vetting process. As a result, the Delphi vetting for the Court of Appeal, High Court and District Court was conducted in late October/early November 2021. Given that Circuit Court judges tend to be especially busy during this time (the start of their court year), it was decided that a preliminary Delphi study would be conducted in September 2021 using the initial time study results from the first week of data collection in July 2021. Adjusting the Delphi results when all time study data were collected was therefore expected to take significantly less time.
In preparation for the study, and to inform the study design, relevant background material was initially compiled in June from online sources. This informed the development of an initial and follow-up questionnaire for the judiciary and Courts Service to identify the availability of core data and compile qualitative information. Additional qualitative information needed was compiled from published documents. Other information, including from unpublished documents, was provided throughout the period by judges, the Courts Service and other national stakeholders, such as the Director of Public Prosecutions (DPP) and Tusla.4 Peers from Canada, the Netherlands, the United Kingdom and the United States also provided valuable feedback and information on the practices in their countries.
At the end of June 2021, interviews were conducted as part of a virtual mission with stakeholders working at and with the courts. This included other members of the judiciary, registrars, prosecutors, legal aid providers, and members of the bar association and the judges association. Further interviews with relevant stakeholders, including all Court Presidents, judges in special positions or committees, and other Courts Service staff, were conducted throughout the duration of the project as additional information needs arose.
At the end of June, bilateral meetings began with the liaison judges to collect the position and staffing data needed to define judicial FTE (see Section 4.4. for details), and to develop an electronic time study data collection form and electronic user guide specific to each court level.
To assist in the development process, the OECD team has shared a template document used in other jurisdictions, with a view to adapting it to the Irish context and capturing the full range of work related to different case types.
The first task was to determine:
The tasks and activities (case-related events and non-case-related functions) that judges perform in and out of court.
Based on experiences in other jurisdictions and to ensure accurate data collection, one way to make the data collection form more user-friendly and not overly complicated to create “composite case categories”, or categories of similar cases that require relatively equivalent judicial effort. As case volume and processing pace varies by court level, the detail of data collected tends to vary by court level accordingly. This was the same for the Irish courts.
In addition, it was important to ensure that all case types are captured and can be matched to regularly collected case data. The number of case categories that the Courts Service collects varies significantly across court levels and is not easy to establish. More importantly, the categories that the Courts Service uses do not fully match the court lists, i.e. the way judges organise themselves and think about case categories. In this context and in coordination with Courts Service staff, liaison judges worked to establish meaningful composite categories for criminal and civil cases, and family cases as necessary.
6 criminal and 15 civil case categories for the Court of Appeal;
7 criminal and 17 civil case categories, plus several sub-categories for the High Court;
2 criminal, 8 civil and 2 family case categories for the Circuit Court;
3 criminal, 4 civil, 2 family and one “other cases” category for the District Court.
Under each case category, judges recorded which specific case actions they were involved in and how many cases were handled during each action (see Table A.1. for an example of one case category). In addition, District Court and Circuit Court judges indicated the locations they were sitting to determine potential local processing variations.5 The final time data collection sheets used by the judges at each court level have been made available to the Irish authorities. The data collection instrument for the Delphi study built upon these case and action categories.
Judges perform a variety of functions, both in and out of court, related to the handling of cases (case-related activities), as well as non-case-related activities. The study team collaborated with liaison judges to develop a full set of non-case-related tasks and activities judges perform. These lists were similar for all courts, with only slight variations. The example from the High Court is shown in Table A.2 .
The time study collection instruments were developed to allow participating judges to enter the number of cases per individual case category and action handled. As mentioned, these data are needed to calculate case weights by case action. In practice, this part proved particularly difficult for judges at the higher volume Circuit and District Courts. At the Circuit Court level, judges could draw on the help of their Judicial Assistants to track the number of cases dealt with each day. District Court judges, however, do not have judicial assistants, and the registrars were unable to provide such support. As a result, the number of cases handled during a day was not always fully captured and related entries could not be included in the overall analysis. Further details related to specific data collection issues faced at the different court levels are provided in Chapters 4 and 6.
The aim was to include a representative sample of judges that covered the entire range of case types and specialty courts. For District and Circuit Courts, a representative coverage of regional differences (i.e. Dublin Metropolitan District (DMD), provincial locations of different population levels and regional representation) was also discussed. The selection was made by liaison judges in co-ordination with their Court Presidents. Specifically, this included:
A series of bilateral meetings also took place in July with the Liaison Group members from the Courts Service, with a view to developing a case data collection instrument that matched the time study data case categories.
Usually, case data collection for a workload study relies on collecting data from a case management system. Typically, this takes four to eight weeks to complete, depending on the level of automation and number of data elements needed. This timeline does not include follow-up data requests as findings evolve or as new data needs emerge. In the case of this study, however, the lack of an integrated case management system for each court level presented significant challenges.
An additional challenge was related to the current approach to data collection by the Courts Service, and particularly that much of the collected data do not reflect or match the actual workload of the courts and judges. While the OECD detected some of these challenges at the outset of the study and inquired if other data collection options, such as sample case file reviews, could be possible, it was not deemed possible at the time by the Courts Service in view of their lack of resources.
Due to these challenges with data, it was difficult to ensure the case categories used in the time study matched the categories used by the Courts Service, which introduced further significant delays. There were two primary data issues that needed to be considered:
The meaning of a “case”, as used by Courts Service, varies for different case categories.
The timing of when “case” data are currently counted varies across case categories.
Usually, the data requested for a workload study are the number of cases filed annually in each established case category per court level (and specialty court, location, if that is part of the study focus). Ideally this information is available for a three to five year period to capture case trends and calculate a meaningful average caseload as it develops over time. Considering the impact of the pandemic on the courts’ operations and the mentioned challenges to provide needed data, the initial request to the Courts Service was for three years of data prior to the start of the pandemic. Available 2020 data could also be considered to better understand case trends during the data collection period.
Regarding “cases filed”, this is generally understood as cases that will eventually come to a judge. A workload study must be based on the full number of cases coming to judges each year to ensure the full workload is considered, not only the cases the judges had time to handle. If the study only considered “cases disposed by judges” it would be limited to the number of cases they were able to handle, which would not lead to a full understanding of the current annual workload.
In Ireland, numbers reported by the Courts Service as “incoming cases” refer to all matters filed at the court. For civil and family cases, these can frequently represent an action taken by parties to notify the opposing party of their intention to take the matter to a hearing to encourage an out-of-court settlement. To manage the courts efficiently, it may be beneficial to distinguish within incoming cases those that eventually settle from those that reach a judge. In the United States and United Kingdom, information on which cases settle after being filed with the court is collected to ensure that the business of the court is presented correctly.
For some case categories, other data are collected in Ireland. For example, personal injury cases are shown in the annual report as “incoming” and “resolved”.6 Resolved is reported as “resolved by courts” and “resolved out of court”. The number of cases resolved by courts is reported to be about half, or less, of all incoming cases, for all court levels and for both years shown. Furthermore, in some case categories the total sum of cases resolved by court and out of court does not match the total number of incoming cases. Which leads to questions regarding whether half of “incoming” cases settle, and if so, is this before or after a case is listed for a hearing, i.e. after it has incurred some judicial time. Of the 4 596 personal injury cases reported as resolved in 2019 at the High Court, only 374 resulted in an award being made, which leads to uncertainty regarding what happened with the remaining cases, as no cases resolved out of court were reported. It is explained that only a few personal injury cases involve a substantive court hearing, which means that it could be assumed that those cases where an award was made resulted in a substantive hearing. However, there is limited information on the status of the over 4 000 cases that may not have resulted in a full hearing and award, and whether they required judicial time (See Table A.3). This example demonstrates some of the questions presented by the current data collected and reported by the Courts Service. It also demonstrates the challenges involved in establishing how much time judges are spending on handling their incoming cases.
To address some of these challenges, a combined effort of the Courts Service, the judges and the OECD team was necessary to develop a more detailed database of case categories from a range of sources that would satisfy the requirements of this study. Where this was not possible, groups of experienced registrars and judges were asked to provide estimates, mirroring the Delphi study approach used to confirm case weights.
The time and Delphi collection gathered data both for the overall time judges need to handle different cases and separately for the major case events per category. Therefore, it was possible to apply these individual time requirements by steps to calculate judicial time needed when cases settled or plead out before a substantive hearing was held, assuming that this could be established.
This issue did not apply to the same extent to criminal cases, as the DPP tends to bring cases to court that will be prosecuted and are ready to be heard. However, even in these events a plea may be entered before a matter is listed. This is likely a rarer event, but would still need to be counted in the future by the Courts Service. In addition, many pleas reportedly tend to be entered immediately before or right at the start of a substantive hearing, meaning that less judicial time is needed. Related case data would need to be collected by the Courts Service to reflect these differences.
In parallel, criminal cases (as well as a range of civil and family cases) presented other types of challenges. The definition of a “case” used by the Courts Service appears to differ to what a “case” means to a judge or what it generally means in various other court systems. Judges, and most others, refer to what comes to them in a file as a “case”, for example a person commits a crime, is apprehended by police, charged with multiple counts that are reviewed by the prosecutor and all or some charges may be dropped if there is no supporting evidence. The prosecutor submits the “case”, i.e. a file outlining which charges are brought against the offender to the court. This is one “case”, or one case that may include several charges. In other cases, multiple offenders may be involved in a crime, but this typically tends to remain one case for the court until a plea or jury verdict is issued, at which point separate sentencing hearings may be held for each defendant. However, the Courts Service reports on incoming offences and defendants, not cases. The number of defendants can be a good enough proxy for actual case numbers, as few cases have multiple offenders. However, cases resolved, while reported separately by guilty pleas, trials and other decision points (several of which may apply to one case), are currently reported only by offences resolved. These data do not allow the incoming numbers of offenders to be linked to potential outcomes, which determines how much time judges generally spend on them.
As outlined for each court level in Chapters 4 and 6, at the two lower-level courts, data could eventually be linked to case record numbers to provide a true count of cases. For the Court of Appeal, appeals are regularly filed by one offender and counted as one matter. For the High Court, however, developing a similar dataset required significant additional time. To provide more detail, the High Court President, other judges and the Courts Service conducted additional data reviews for select case categories and sample months in 2018 and 2019. These additional data substantiated the Delphi estimates and assisted in clarifying case data calculations. More detail is provided in the High Court section in Chapter 4.
In addition, for many family and civil case matters, similar issues arose. Cases incoming was reported with a range of meanings, such as “orders made” (e.g. childcare District Court, domestic violence District Court) and “applications” (e.g. chancery High Court), all of which can relate to multiple orders or applications made in one case. Or, as mentioned with personal insolvency cases, it is unclear what is counted at any of the trial court levels.
When possible, the OECD aimed to apply data from other sources, such as the DPP and Tusla, to confirm incoming and resolved data. As different agencies report data differently, a direct comparison was not possible, but some approximations confirmed the additional detailed analysis.
One remaining challenge was related to the limitations of having clear data definitions or data collection standards. Ireland would benefit from developing more relevant and reliable data collection mechanisms to support the ongoing efforts to develop new case management applications.
Access to solid data that holistically reflects the work of the judges and other efforts is essential to predict court resources needed for the future and ensure the proper functioning of court management. The related recommendations in Chapter 6 aim to support Ireland to ensure that the future state of case data collection is better focused on enabling good case and court management.
References
[2] Courts Service (2020), Annual Report 2019, https://www.courts.ie/acc/alfresco/9bd89c8a-3187-44c3-a2e9-ff0855e69cb5/Courts%20Service%20Annual%20Report%202019.pdf/pdf#view=fitH.
[1] National Center for State Courts (2016), Vermont Trial Court System Judicial Officer and Court Staff Weighted Caseload Study, 2015, https://www.vermontjudiciary.org/sites/default/files/documents/Weighted_Caseload_Report.pdf.
Notes
← 1. While case weights represent averages of different types of cases and could be subject to outliers, the study methodology aimed to mitigate this risk through the Delphi study, which asked the expert participants to estimate an average minimum and maximum time needed, not considering significant outliers. After the initial results, the Delphi study participants were then requested to consider again if outliers may have influenced their estimates and if adjustment might be needed.
← 2. See the calculations for Ireland in Chapter 3, pp 43-45.
← 3. In July, an inquiry by the OECD team to the National Center for State Courts, the predominate organisation conducting court workload studies in the United States, indicated that all workload studies had been halted for the duration of the pandemic.
← 4. Tusla is the Child and Family Agency of Ireland, regulated by the Child and Family Agency Act 2013 (available here).
← 5. Considering the limited data collection period, this did not provide sufficient data to draw solid conclusions for different locations
← 6. See , (Courts Service, 2020[2]) from the Courts Service Annual Report 2019, p. 48.