Appraisal of Guidelines for Research and Evaluation

The AGREE ( A ppraisal of G uidelines for Re search and e valuation ) is an instrument for the development, improvement and evaluation of treatment guidelines for clinical use. It is a questionnaire for checking and naming quality criteria for treatment regulations. The process was developed in international cooperation between universities and institutes and is supervised by a council made up of members of these educational institutions, the AGREE NEXT STEPS Consortium. The Canadian Institutes of Health Research is funding the project.

history

The foundations that led to the development of the AGREE Instrument were laid in 1997 with the "Apprasial instrument for clinical practice giudlines" or "Cluzeau Instrument". However, these did not adequately meet the requirements for the evaluation of guidelines for action, which is why the AGREE instrument was published in 2003.   

The procedures underlying this method inspired the “AGREE Collaboration” to create the “AGREE I” method, which was finally published in 2003.

The AGREE Collaboration is an international group that, led by European scientists, worked on a method to further improve the evaluation of instructions for treatment and to generalize it, especially for the medical sector.

After a few years of further work, a revised version of the original AGREE instrument was finally published in 2009 by the “AGREE Collaboration”. Since it was only a revision and not a completely new version of the instrument, it was published under the name "AGREE II".   

Finally, the group put the “AGREE Enterprise Website” online, which among other things has integrated “My AGREE” and other application helpers for AGREE and is intended to make AGREE easier to use. In addition, various application descriptions for AGREE and information about the various research groups are summarized here.

New features included a simplified option for multiple evaluations of the same instruction.

Since 2013, a method for evaluating the clinical credibility of the assessment of instructions has been developed. As soon as it is ready, this should complement the AGREE II method.

The AGREE Collaboration worked with Minervation, a web design company from Oxford , to develop the individual websites .

The AGREE instrument

The AGREE instrument is structured like a questionnaire. A total of 23 categories are used, each of which is intended to examine different aspects of a clinical treatment recommendation. The criteria are divided into six subject areas, called domains. Each category can then be given a score based on how satisfied the evaluator was. The range of points to be awarded ranges from 1 (does not apply at all) to 7 (applies completely).

	Name of the category after the AGREE designation:	Use of the criterion:	Implementation example in a directive:
Domain 1: "Scope and purpose":
Category 1:	"The overall objective (s) of the guideline is / are clearly described."	The aim is to check whether and how appropriate the description of the guideline is to the actual content of the guideline. The following points should be included in a treatment guideline: - Care aspect (e.g. prevention , screening , diagnostics , treatment ) - Expected benefits or expected results - Target population (e.g. patients, population)	Identification of a strategy for the prevention of contrast-induced renal failure in clinical practice
Category 2:	"The health-related question (s) dealt with in the guideline is (are) clearly described."	It should be checked whether the health issues are clearly described.	How often do patients diagnosed with imaging tests that use contrast media develop breast cancer?
Category 3:	"The target population (e.g. patients, population) of the guideline is clearly described"	It should be checked whether it is clearly described which population is affected by the treatment recommendation. It is important to ensure that the following questions are answered: Age group gender clinical description of the disease possible comorbidities in this group of patients	A treatment guideline for the identification of suitable methods for the prevention of contrast medium-induced kidney failure in the identification of liver metastases in men between 50 and 70, taking into account correlating liver failure.
Domain 2: "Stakeholder Involvement":
Category 4:	"The guideline development group includes members of all relevant professional groups"	It should be checked whether the professional groups involved in the creation of the guideline are mentioned in the guideline. External experts and similar professional groups are excluded.	The following persons have worked on this guideline: Mustermann, Max (doctor) Musterfrau, Eva (Controlling)
Category 5:	"The views and preferences of the target population (e.g. patients, population) were determined"	The aim is to check whether the experience and expectations of the target population have been taken into account. The surveys and interviews carried out for this purpose should be documented.	To create this guideline, breast cancer patients were asked what a gain in quality of life would be for them.
Category 6:	"The target user group (s) of the guideline is (are) clearly described."	The aim is to determine whether the user group of this guideline is clearly documented.	This guideline is addressed to the following users: Oncologists Nephrologist Family doctors
Domain 3: "Accuracy of guideline development"
Category 7:	"A systematic search was made for evidence."	The aim is to investigate where the data used to create the guideline come from. This information should be clearly documented in a separate chapter / document.	The tumor metastases were compared with existing CT images using the “Tumors of the liver” database
Category 8:	"The criteria for selecting the evidence are clearly described"	It should check whether a clear explanation is given as to how the user target group should come to their decisions.	A study carried out in 2011 suggests that there is a tumor metastasis in the liver with a carcinoma in the area of ​​the fundus ventriculi. The images from the CT also contain evidence-generating information. From this it can be deduced that chemotherapy should be initiated.
Category 9:	"The strengths and weaknesses of the evidence are clearly described."	The methodological quality of the criteria described in category 8 is to be assessed.	The quality of the CT images shows significant deficiencies. These should therefore be used conservatively.
Category 10:	"The methodological procedure for formulating the recommendations is clearly described."	The procedure for creating the recommendations is to be evaluated.	Various clinical studies were carried out to create this treatment specification. All had in common the result that an increased addition of contrast medium leads to an increased likelihood of developing kidney problems.
Category 11:	"The health benefits, side effects and risks were taken into account when formulating the recommendations"	The wording of the guideline is to be evaluated with regard to benefits, side effects and risks.	The therapy is intended to slow the growth of breast tumors. However, therapy with the drug has been shown to cause brain tumors, and mood swings are often reported.
Category 12:	"The underlying evidence can be clearly assigned to the recommendations."	It is to be checked whether the treatment recommendations are clearly linked to the available disease data.	Treatment with chemotherapy causes the faster dividing cells to die off due to the cytostatics in the drug.
Category 13:	"The guideline was assessed by external experts before it was published."	The aim is to check whether and to what extent the guideline was evaluated before it was published.	The guideline was tested for 6 months in the teaching hospital of Heidelberg University and is optimally applicable for the selected target group
Category 14:	"There is a procedure for updating the guideline."	The possible procedure for updating the guideline is to be evaluated	After the introduction, the use of this treatment instruction will be monitored. In addition, after six months, an adjustment to any changes in the clinical process is planned.
Domain 4: "Clarity of design"
Category 15:	"The recommendations of the guideline are specific and clear."	It should be checked to what extent the recommendations provide sufficiently specific information about the treatments required.	If a breast tumor occurs, chemotherapy with drug XY must be initiated immediately, while the antibody XZ is administered intravenously. A radiation treatment or an operation can be carried out as a support
Category 16:	"The different alternatives for treating the disease or the health problem are clearly shown."	The extent to which a sufficient spectrum of alternative treatments is available should be assessed.	As an alternative to treating breast cancer with the antibody XZ, the drug YZ can also be used to reduce the mood swings that occur
Category 17:	"The key recommendations of the guideline are easy to find."	It should be assessed to what extent the most relevant treatment recommendation is clearly evident.	In the clinical study, therapy with the antibody XY proved to be most effective, which despite the side effects caused the most effective tumor reduction
Domain 5: Applicability
Category 18:	"Possible conducive and hindering factors for the application of the guideline are described."	It should be checked whether the treatment guideline addresses the possible obstacles or positively promoting factors. There should be a direct connection between the treatment recommendation and factors.	Possible obstacles in the realignment of the treatment of pain patients can arise if there are still delivery difficulties in the procurement of cannabis from abroad.
Category 19:	"The guideline makes suggestions and / or names instruments that support the application of the guideline recommendations."	The aim is to assess the extent to which the guideline improves the applicability of the guideline through recommendations of material or actions.	The doctors each receive a short version of the treatment guidelines as a laminated sheet to take with them to the ward.
Category 20:	"The possible financial effects of the guideline recommendations were taken into account."	It will be assessed whether the guideline adequately addresses the financial impact of its application	One possible effect of purchasing a positron accelerator can be an increased electricity bill, which must continue to be covered by public funds.
Category 21:	"The guideline names measurement criteria for the evaluation of the process and / or result quality of the guideline."	It should be assessed to what extent the guideline provides criteria for its own evaluation.	An evaluation sheet can be found in the appendix to these treatment guidelines. The points: effectiveness of the treatment, application of the treatment, patient condition, ... are discussed
Domain 6: Editorial independence
Category 22:	"The financing organization had no influence on the content of the guideline."	It is assessed to what extent the non-interference of the funding organization is guaranteed in the creation of the guideline	Before the guideline was drawn up, the hospital's funding body provided a financial ramen that the doctors adhered to when drawing up their guidelines. Furthermore, there was no influence from the financing body.
Category 23:	"Conflicts of interest of the members of the guideline development group were documented and taken into account when creating the guideline."	It should be assessed whether the conflict of interest within the guideline-producing organization has been documented and taken into account in the guideline	When applying this guideline for the treatment of breast cancer, there may be a conflict between the treating physician and the controlling department, as the physician wants to carry out the expensive radiation with the positron accelerator more frequently than is paid for by the health insurance.

Evaluation of the method

Basically, the recommendation of the compiler, in order to ensure representativeness, is that at least two people should implement the guideline with the AGREE instrument. A number of evaluators of four people is assumed to be optimal. The AGREE INSTRUMENT is evaluated according to the following scheme:  

example

	Criterion 1	Criterion 2	Criterion 3	Buzz domain
Evaluator 1	4th	6th	7th	17th
Evaluator 2	5	2	7th	14th
Evaluator 3	6th	4th	6th	16
Evaluator 4	4th	5	3	12
Sum criterion	19th	17th	23	59

The overall rating of the domain is calculated as follows:

(Calculated number of points - minimum possible number of points) / (maximum possible number of points - minimum possible number of points) = overall evaluation domain

where the following applies to the minimum possible number of points:

Minimum possible number of points = 1 (does not apply at all) x (categories) x (evaluator)

for the example:

Minimum possible number of points = 1 (does not apply at all) x (3) x (4) = 12

and for the maximum possible number of points applies:

Maximum possible number of points = 7 (fully applies) x (criteria) x (evaluator)

for the example:

Maximum possible number of points = 7 (fully applies) x (3) x (4) = 84

This results in an overall rating for the domain for the example:

(Calculated number of points - minimum possible number of points) / (Maximum possible number of points - minimum possible number of points) = (59-12) / (84-12) = 0.652

Usually it is given in percent: 0.652 * 100 = 65.2%

The value that results can be used to compare two different guidelines. The AGREE INSTRUMENT does not assess which value should be perceived as good or which value is bad. The user should consider for himself what percentage should be achieved in the domain.

Comparison of the versions

The AGREE II tool is a method that has been further developed from the original AGREE I, although it has some significant changes compared to the previous tool, but essentially has the same objectives as AGREE I. The changes made relate to the formulations of the individual categoricals.

The new concept was developed after the original developers realized that their concept still had some opportunities for improvement and optimization.

The main aim was to improve the measurement properties, practicability and user-friendliness.

As a result, members of the AGREE Collaboration founded the so-called AGREE next Steps Consortium, which was responsible for the further development of AGREE I to AGREE II.

In 2009 AGREE II was finally published and replaced the obsolete AGREE I.

The core of the new tool is a user manual for better use of AGREE II and 23 areas, which are organized in 6 categories. The manual provides detailed information on each of the 23 questions and also gives a very detailed description of how these are to be used in practice. This makes the use of AGREE II much easier, even for inexperienced users.

AGREE I already had 23 areas, but the individual formulations and orientations of these 23 areas in AGREE II have been slightly changed.

The following table gives an overview of the changes from AGREE I to AGREE II in the 23 areas. Only the changes are mentioned.

Areas from AGREE I	Areas from AGREE II
Category 1: scope and purpose
1. The overall objective (s) of the guideline will be specifically described	1. The overall objective (s) of the guideline is / are clearly described. a)
2. The medical question (s) dealt with in the guideline is (are) specifically described.	2. The (s) dealt with in the guideline health-related question (s) is (are) clearly described.
3. The patients referred to by the guideline are specifically described.	3. The target population (e.g. patients, population) the guideline is clearly described.
Category 2: stakeholder involvement
5. The patient's views and wishes were determined.	5. The views and preferences of the Target population (e.g. patients, population) were determined. a)
6. The guideline is the target user group (s) (are) clearly defined.	6. The target user group (s) of the guideline is (are) clearly described. a)
7. The guideline was tested in a pilot test by members of the user group.	Area was deleted and included in area 19.
Category 3: Accuracy of guideline development
8. When looking for evidence were made systematic methods applied.	8. A systematic search for evidence was made . Renaming in area 7. a)
9. The criteria for selecting the evidence are clearly described.	9. No changes. Renamed to area 8. a)
10. The ones used to formulate the recommendations methods used are clearly described.	10. The methodological procedure for formulating the recommendations is clearly described. No changes a)
12. The connection between the Recommendations and the underlying Evidence is explicitly presented.	12. The underlying evidence can deny the Recommendations are clearly assigned. No change. a)
Category 4: clarity of design
16. Those for the treatment of the disease available alternatives are clearly shown.	16. The different alternatives for that Treatment of the disease or the Health problems are clearly shown.
17. Key recommendations of the guideline are easy to identify.	17. No change in content. Reformulated in the context of the German translation. The Key recommendations of the guideline are easy to find.
Category 5: Applicability
18. The guideline identifies instruments that make their Application can support.	18. The guideline makes suggestions and / or names instruments that make the application of the Support guideline recommendations. AND new assignment of the domain (from clarity of design to applicability) AND renaming to number 19.
19. The possible organizational barriers to the application of the recommendations were discussed.	19. Possible conducive and hindering factors for the application of the guideline are described AND the order changed. Renamed to Number 18.
20. The through application of the guideline possible costs considered.	20. By applying the guideline possible resource requirements was taken into account.
21. The guideline names key test criteria for monitoring and / or quality assessment.	21. The guideline names parameters for evaluating the quality of the process and / or results.
Category 6: Editorial independence
22. The guideline is editorially approved by the financing organization (s) independently.	22. The funding organization does not have one Influenced the content of the guideline.
23. Conflicts of interest of members of the Guideline development group were documented.	23. Conflicts of interest of members of the Development group of the guideline documented and when creating guidelines considered.
a) The wording in the original AGREE II is unchanged from that in the original AGREE I, but the German translation has been adapted.

Web links

• https://www.agreetrust.org/
• http://www.cihr-irsc.gc.ca/e/36331.html

Individual evidence

1. ↑ Introduction to agree. (No longer available online.) In: AGREE Enterprise. AGREE RESEARCH TRUST, archived from the original on July 4, 2018 ; accessed on July 2, 2018 . Info: The archive link was inserted automatically and has not yet been checked. Please check the original and archive link according to the instructions and then remove this notice.  
2. ↑ Brouwers, MC, K. Kerkvliet, K. Spithoff: The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines . In: Bmj . 352nd edition. 2016, doi : 10.1136 / bmj.i1152 .
3. ↑ Terrace, L: Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project . In: BMJ Quality & Safety . tape 12 , 2003, p. 18-23 .
4. ^ Makarski, Julie Brouwers, Melissa C .: The AGREE Enterprise: a decade of advancing clinical practice guidelines . In: Implementation Science . 1st edition. tape 9 , August 2015, ISSN  1748-5908 , p. 103 , doi : 10.1186 / s13012-014-0103-2 .
5. ↑ Cluzeau, FA, et al .: Development and application of a generic methodology to assess the quality of clinical guidelines . In: Int J Qual Health Care . No. 11 (1) , 1999, pp. 21-8 .
6. ↑ Vanommeslaeghe, F. De Mulder, E. Van de Bruaene, C. Van de Bruaene, L. Lameire, N. Van Biesen, W .: Selecting a strategy for prevention of contrast-induced nephropathy in clinical practice: an evaluation of different clinical practice guidelines using the AGREE tool . In: Nephrol Dial Transplant . 8th edition. No. 30 , 2015, ISSN  0931-0509 , doi : 10.1093 / ndt / gfv220 .
7. ↑ ^{a b c} Brouwers, MC Kho, ME Browman, GP Burgers, JS Cluzeau, F. Feder, G. Fervers, B. Graham, ID Grimshaw, J. Hanna, SE Littlejohns, P. Makarski, J. Zitzelsberger, L. : AGREE II: advancing guideline development, reporting and evaluation in health care . In: Cmaj . 18th edition. tape 182 , doi : 10.1503 / cmaj.090449 .
8. ↑ Jamaloodien, K .: Application of the AGREE II instrument in the evaluation of a selection of South African Clinical Guidelines . Ed .: University of Pretoria Faculty of Health Sciences School of Medicine. Pretoria 2014.
9. ^ Zeng, X. Zhang, Y. Kwong, JS Zhang, C. Li, S. Sun, F. Niu, Y. Du, L .: The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis , and clinical practice guideline: a systematic review . In: J Evid Based Med . 1st edition. tape 8 , 2015, ISSN  1756-5391 , p. 2-10 , doi : 10.1111 / jebm.12141 .
10. ↑ Graham ID1, Calder LA Hébert PC, Carter AO, Tetroe JM .: A comparison of clinical practice guideline appraisal instruments . In: Int J Technol Assess Health Care . tape 16 (4) , 2000, pp. 1024-1038 .