Published date: 1 July 2016

General

Civil Aviation Authority Advisory Circulars contain information about standards, practices, and procedures that the Director has found to be acceptable for compliance with the associated rule.

Consideration will be given to other methods of compliance that may be presented to the Director. When new standards, practices, or procedures are found to be acceptable they will be added to the appropriate Advisory Circular.

Purpose

This material is intended for applicants for air traffic service personnel licences and ratings, holders of air traffic service personnel licences and ratings, air traffic services instructors, air traffic service examiners, training organisations, and air traffic service organisations.

Related Rules

This Advisory Circular relates specifically to Part 65 Air Traffic Service Personnel Licences and Ratings.

Change Notice

Revision 2 alters a number of references to other Advisory Circulars in this document. This is to reflect the new numbering of those ACs in line with a project to standardise the numbering of all ACs.


Introduction

Civil Aviation Rules, Part 65 Air Traffic Service Personnel Licences and Ratings was issued on 1 April 1997. The new Rule Part introduced changes that included the new Area Control Automatic Dependent Surveillance Ratings, the Instructor Ratings, the Examiner Ratings, and the Flight Service Operator Licences. This Advisory Circular and the associated series of Advisory Circulars have been produced to support Part 65.

Following comments received, the AC65 series has been split into separate Advisory Circulars – one for each Part 65 Subpart, and one for each rating where more than one rating is contained within a Subpart.


Advisory circular intent and process

Guidance on how to comply with CAR 65 Subpart J is contained within this Advisory Circular (AC) Air Traffic Service Personnel Licences and Ratings – Air Traffic Service Examiner Ratings.

The intention of this advisory material is to provide advice with regard to the issue of air traffic service licences and ratings, the conditions under which those licences and ratings are necessary, and the privileges and limitations of those licences and ratings.

As part of its development, a CIRAG (Civil Aviation Industry Rules Advisory Group) TSG (Technical Specialist Group) was established to provide advice to industry. This group consisted of representatives from the air traffic service provider and representatives of the ATC personnel. Each element of the rule was considered and interpreted by the TSG, and advice was given on possible means to meet the criteria outlined in the rule.

The CAA is actively managing the development of syllabuses into specific objective format. This format will specify exactly what has to be covered, and to what standard, so that no matter who studies, who instructs, and who assesses, all are working to exactly the same standards. This development will be done using the resources of both the rules consultative process, and of the ATTTO NZQA framework process.

Appropriate references to the ICAO courses and NZQA unit standards are included for information and guidance.


SUBPART J — Air Traffic Service Examiner Ratings

65.451  Applicability

Subpart J prescribes rules governing the issue of air traffic service examiner ratings and the privileges and limitations of those ratings.

65.453  Eligibility requirements

65.453.1 Rule 65.453 (a) (3) requires an applicant for an examiner rating to have demonstrated to a Civil Aviation Authority testing officer the ability to perform the duties of an air traffic service examiner. The following would be acceptable in meeting this demonstration:

(a) Successful passing of an examination covering the syllabus given in Appendix A of this subpart advisory circular and Appendices A & B of advisory circular 65-9 (Air Traffic Service Instructor Ratings) [ It is acceptable that the examination is set and marked by the 172/141 organisation, and moderated by the Civil Aviation Authority testing officer.In these circumstances CAA would be provided with the marked examination papers and the model answers], and

(b) A practical examination using a similar structure given in Appendix B of this subpart advisory circular or Appendix C of advisory circular 65-9 (Air Traffic Service Instructor Ratings). [ The test environment is required to be acceptable to the CAA testing officer. Examples of an acceptable environment are provided in Appendix B of this subpart advisory circular].

65.453.2 Rule 65.453 (a) (3) requires an applicant for an examiner rating to have demonstrated to a Civil Aviation Authority testing officer the ability to perform the duties of an air traffic service examiner. The procedures followed by the testing officer are given in Appendix B of this advisory circular.

65.455  Privileges and limitations

65.455.1 Rule 65.455 (b) (1) requires an air traffic service examiner to hold a current air traffic service licence with a rating for the relevant service. The following is acceptable in meeting this requirement:

(a) Individual examiner ratings will be limited to certain privileges depending on the ratings held, and the level of operational awareness as determined by the CAA Testing Officer. The CAA ‘application for ATS examiner test’ form details all possible privileges.

(b) Testing for the issue or continued currency of Instructor Ratings – privileges will be limited to those examiners who have completed and passed all training as detailed in the 172/141 organisations exposition.

(c) Testing for the issue of or continued currency of licences, as a licence and at least one rating are assessed together, privileges will be limited to those examiners who hold a rating for the rating being assessed. In this case the examiner would be moderating the assessment by an Instructor (CHK).

(d) Testing for the continued currency of Instructor (CHK) Ratings – privileges will be limited to those examiners who hold the rating being assessed by the Instructor (CHK), and have demonstrated an adequate level of operational awareness to a CAA Testing Officer.

(e) It is acceptable that an ATS Examiner may exercise their privileges in one or more categories as detailed in subparagraphs (ii), (iii), and (iv) depending on the areas that the examiner has been trained and tested.

65.455.2 Rule 65.455 (c) allows the holder of an air traffic service examiner who is not exercising the privilege of providing an air traffic service, not to hold a current medical certificate. Factors that will be considered at the time of the renewal check are employment in a related field or relevant familiarisation that allows the examiner to maintain an adequate level of operational awareness.

65.455.3 An examiner may exercise the privileges of an instructor (OJT) or instructor (CHK) rating provided they have met the applicable currency requirements.


APPENDIX A

Syllabus

This syllabus is based upon the requirements of the Rule, ICAO courses, submissions from industry, and the NZQA unit standards. Appropriate references to the ICAO courses and NZQA unit standards are included for information and guidance. Where a syllabus topic merits greater emphasis, this is indicated by an asterisk.


APPENDIX B

Procedures –ATS Examiner Testing

The overall objective, and the degree, of testing is to provide adequate confidence, to the Director of the Civil Aviation Authority, that ATS Examiners will act in the overall interest of safety when conducting their duties.

Principles of testing

Testing is the process of defining, observing and measuring a candidate’s performance during a test.

The basic principles of testing should test competence not test ability.

a) Testing must assess knowledge and skill level and not the ability to pass a test.

b) The measurement of achievement must be relevant to the defined standards and not an end in itself.

c) Standards set should be reasonable when set against the kind of skill or information necessary to the performance of a job or part-job.

d) Tests must be pertinent to the kind of skill and knowledge being assessed so that a candidate’s competence is what is being determined and not their ability to memorise lists over the short-term or write clever exam answers or survive the artificial stress of a test environment.

The test has been designed to minimise the degree of subjectivity although the testing officer will still have to exercise judgement where factors may affect the test environment eg. live versus simulated.

Testing officer responsibility and scope

The testing officer who conducts the test is responsible for determining that the candidate meets the standards outlined in each performance criteria.

For each performance criteria that requires a good working or mastery knowledge/skill, the testing officer will:

a) Orally question the candidate on those elements; and/or

b) ask the candidate to perform the skill elements in a simulated or live environment; and/or

c) moderate the written/oral examination conducted by the service provider; and/or

d) observe the candidates in a seminar/workshop environment; and/or

e) use other methods as determined by the Manager Personnel Licensing.

Oral questions may be used at any time during the test. Discretion will be used in the live environment, in order that the candidate is not distracted from the supervision of an air traffic service.

The scope of authority of the CAA testing officer includes issue of ATS Examiner Rating and renewal of ATS Examiner Rating

Testing cycle

The testing process is considered to be a five-stage cycle:

Objective, standards, performance, measurement, conclusion.

Objective

The first stage is to determine the objectives. Since it would be meaningless to evaluate the candidate’s behaviour without considering what that behaviour should be, the process of testing should begin with clearly defined objectives.

Standards

To be proficient in evaluating a candidate’s performance during a test, the testing officer must be completely familiar with the standards for each exercise assessed.

Performance

During the test, the testing officer observes the candidate’s performance in response to situations presented.

Measurement

The testing officer measures the performance, compares it to the applicable standard and determines the assessment.

Conclusion

Based on the conclusions of the testing officer, a mark is awarded to the candidate. However, to be useful, the conclusions must be presented in such a way that the information is clearly understood and easily accessible to supervisors, and candidates. When a candidate commits major errors during the performance of an exercise or fails to meet the required standard, the testing officer must state the nature of the problem(s), in writing, in the Remarks page on the Test Report.

Characteristics of testing

A test may become useless if certain criteria are not respected. The following five characteristics, if used carefully when conducting a test, will result in an accurate and effective form of evaluation.

Reliability

Reliability ensures consistent results. As applied to the test, this would mean that two identical performances should result in the same test result.

Human factors can have a significant affect on test reliability. Some of these factors are:

a) fatigue ....................... (sufficient sleep or rest prior to the test)

b) emotions .................... (work or home personal problems)

c) state of health ............ (cold or flu etc.)

d) time of day ................. (very early in the morning etc)

e) distractions ............... (noise, interruptions etc.)

Testing officers should be conscious of these factors and attempt to reduce as many variables as possible. The testing officer may accept some of these factors as a reason for some lack of smoothness in the candidate’s performance, but never as a compromise for attaining the minimum standards. The testing officer should also be aware that his or her ability to accurately assess the candidate’s performance could be affected by these same factors.

Validity

Tests are valid if they measure what they are supposed to measure and nothing else. In terms of the test, assessment for ATS exercises must remain within the bounds of the appropriate test standards. The scope of the test must be such that when candidates pass, they have met the required standards for the issue of the examiner rating.

Comprehensive

A test is comprehensive if it contains a sample of all course material and measures each area of skill and knowledge required to ensure the standard is met.

Objectivity

Objectivity ensures the testing officer’s personal opinions will not affect the outcome or assessment of the test. In order to achieve the highest possible degree of objectivity, the testing officer should record observations to eliminate errors and assist recall in determining that a candidate has met the performance criteria.

Tests are marked to some degree on a subjective basis. Subjective assessments will be more valid if the testing officer has a sound and adequate background knowledge of the testing process, and the expertise to accurately assess test applicants.

Testing errors

In order to test effectively, the testing officer requires not only a sound knowledge of the characteristics of testing but also a firm understanding of the possible errors that can occur throughout the testing process. Errors in testing fall into several categories. The most frequent errors are as follows:

Personal bias error

Errors of personal bias are indicated by a tendency of a testing officer to rate candidates or a particular group of candidates—

All the same; all as average or, all as great at the high end of the scale or, all as poor at the low end of the scale.

Central tendency errors are indicated by a tendency to rate all or most candidates as average. The testing officer really “feels” that the performance of most candidates is not as good as it should be and therefore underscores a candidate’s good performance. On the other hand, the testing officer is reluctant to cope with the possible emotional response of a candidate. This results in padded or inflated assessments of poor performance thus both candidates are awarded an average assessment. This error may also occur because a testing officer does not want to think (put effort into making a decision).

Generosity errors are indicated by a tendency to rate all individuals at the high end of the scale and is probably the most common type of personal bias. This could be caused by a testing officer’s desire to be known always as a nice person.

Severity errors . In this case, all or most candidates are graded at the low end of the scale. Testing officers may feel that the published test standards are too low and score the test against their own set of standards. This type of testing officer feels that few people perform as well as they can.

Logical error

This error occurs when a testing officer assumes that a high degree of ability in one area means a similar degree of competence in another. This is especially true if the two items being assessed are similar or related. A pass on one or two exercises does not mean the candidate is so qualified on all exercises. The full test must be completed and marked.

Halo effect

This error occurs when a testing officer’s impression of a candidate is allowed to influence the assessment of performance. Halo error can result in rating an applicant too high or too low.

One form of halo error is the Error of leniency – leniency has its source in a testing officer’s likes, dislikes, opinions, prejudices, moods and political or community influence of people. For example, when testing a friend, acquaintance, or high profile individual, a testing officer may give an undeserved ‘pass’ or, conversely the Er ror of Stereotype . As with the error of leniency, the error of stereotype has its source in likes, dislikes, opinions, prejudices, etc. In this case however, a testing officer may allow personal opinion to influence the assessment of the candidate and award an undeserved ‘fail’.

Error of narrow criterion

This error may occur when a testing officer has a group of candidates to test. The testing officer may, under this condition, rate each applicant against the others within the group instead of against the standards. If the group to be tested is above average, a candidate who is of average ability may be awarded an undeserved ‘fail’. If the group of candidates to be tested is below average, then a candidate who performs the best within this group may be awarded an undeserved ‘pass’.

Error of delayed grading

Should a delay occur in awarding the assessment for an exercise, there might be a tendency to award an average result due to the lack of information and/or poor recall. By not making an assessment immediately after the event, testing officers may award assessments based upon an overall impression of the test. This results in an erroneous assessment and a test report, which is of little value.

Standards error

All the errors we have discussed result in a standards error. However, if a testing officer is not thoroughly familiar with established standards, as outlined in the applicable test standards, it is virtually impossible to conduct a test to that standard.

While these errors are presented here on paper in a clear and obvious way, under testing conditions this is not always so. Normally it is a combination of two or more of the errors and clear and obvious is not an apparent trait. Therefore testing officers must be aware of these potential errors and consciously prevent such errors from entering, in any degree, into the tests they conduct to ensure the validity of the test and the result they award.

Conducting the test

Test standard

Civil Aviation Rule (CAR) Part 65 and the associated advisory circulars specify the areas in which knowledge and skill must be demonstrated by the candidate before an examiner rating is issued/renewed.

Acceptable/unacceptable knowledge/skill of performance criteria is described in the worksheet.

If in the judgement of the testing officer, the candidate does not meet the minimum standard of any performance criteria, then the test is failed. It is not CAA policy to issue a partial pass. If a candidate fails to demonstrate proficiency in an area, then the test is failed and a complete retest is conducted. (The testing process is a sampling process therefore it is not appropriate to issue a partial pass, but rather to take another complete sample to provide the appropriate level of confidence.)

The testing officer or candidate may discontinue the test at any time after the failure of critical performance criteria makes the candidate ineligible to pass the test.

Any action, or lack of action, by the candidate, which requires corrective intervention by the testing officer to maintain safe operations, may be disqualifying.

It is vitally important that the candidate uses proper scanning techniques in a practical assessment. Ineffective performance will be disqualifying.

Unsatisfactory performance in any test item will result in the candidate being advised of the failure aspects and the further training believed necessary before a further test may be undertaken.

Recording unsatisfactory performance. When performance criteria are unsatisfactory the testing officer must record it on the test report against the specific performance criteria.

The test will ensure that skill and knowledge are demonstrated in the relevant performance areas. The examiner test process is designed to show a link between the theory of assessing and the practical of assessing candidates for licences/ratings where skills are to be demonstrated.

The assessment will consist of the briefing of the candidate, observing the assessment, the debrief and the administration process (form filling) including a review of the candidates training documentation. There may be supplementary oral questions to ascertain that the examiner is competent.

An examiner rating will not normally be issued/renewed if the examiner training theory course contained in the 141/172 exposition has not been completed. In exceptional cases this may be allowed provided the individual already holds an instructor or an examiner rating, and can demonstrate proficiency to the CAA testing officer. In these circumstances limitations may be imposed in terms of completing the required training within a defined period.

Open and closed book testing is acceptable for examiner tests. The candidate for an examiner rating will normally be allowed access to all documentation that would normally be used as reference material during an assessment.

Performance

Grade/Percentage equivalent

Demonstrate a Mastery Knowledge

equates  to a thorough understanding as determined by the assessor with, if applicable, an agreed minimum pass mark of between 95% and 100%                                      

Demonstrate a Good Working Knowledge

equates to a minimum pass mark of 80%              

Practical assessment process

To ensure that the assessment is effective the testing officer must:

a) Ensure the environment is as close as possible to the real life work situation.

b) Focus on the assessment task.

c) Establish a non-threatening and supportive atmosphere.

d) Communicate clearly to the candidate.

e) Listen effectively.

f) Question effectively.

g) Provide clear and constructive feedback.

h) Assess consistently.

Skills are assessed by:

a) Direct observation of practical task

b) Direct observation of simulated task

c) Indirect evidence of competence

d) Examination of Course reports, Periodic Training reports, Training Assessment reports, Annual Proficiency reports (in the case of renewals)

Communication with the candidate

Before the assessment commences the testing officer will brief the candidate on:

a) What is to be assessed

b) How it will be done

c) Who will do the assessment

d) How long it will take

e) When it will take place

f) When the results will be known

g) What will happen if any part of the assessment is assessed to be not competent

h) How any special needs will be met

i) What will happen to the results

j) Who will see them

k) What happens next

Dealing with the candidate on the day

The following should be taken into consideration:

a) Treat the candidate with respect

b) Reduce stress

c) Interaction with the candidate

d) The spacing between the assessor and the candidate

e) The position of the assessor

f) Cultural considerations

Test environment

The test should be conducted at a mutually agreed environment that allows a CAA Testing Officer to meet the requirements of the examination, and meets the CAA business policy. The following are considered to be a suitable environment:

a) Classroom

b) Simulated

c) Live

Access to headsets may be required.

The preferred test environment would be a ‘simulated’ environment where traffic and errors could be introduced to provide suitable opportunities for the examiner to demonstrate skills. As this is not always possible, a ‘live’ environment would be acceptable, but the limitation could mean reduced opportunities for the examiner to demonstrate skills and increased test duration.

The assessment may be either an actual assessment or simulated. That is, the candidate does not have to be undergoing a formal assessment although preference is for during an actual formal assessment (in a simulated environment). The actual assessment could be during an Annual Proficiency Assessment (APA).

Test documentation

Where appropriate the following documentation will be completed after an examiner test:

· Test form – ATS Examiner Rating (CAA 24065/04) available from the CAA website.

· Worksheet ATS Examiner Rating (This worksheet is incorporated into these procedures.).

· ATS logbook sticker (The following words are printed on a sticker, completed and entered into the logbook after successful completion of the test.)

“This is to certify that

________________________________________

has demonstrated the ability to perform the duties of an air traffic service examiner.

_________________________ Date __________

Testing Officer, Civil Aviation Authority

                                                                Expiry Date ___________”


Worksheet ATS Examiner Rating

                                                                                                                             Candidate: ___________________

EC - Elements of Competence, PC - Performance Criteria

NT - indicates not tested, NA - indicates not applicable.

EC. Assess staff for the issue of ATS licences, and ratings.

PC             Good working knowledge and skills in preparing and conducting written and oral examinations, in accordance with the CA Rules and Advisory Circulars, and/or the 172/141 exposition documents is demonstrated.

Yes/No

NT/NA

Demonstrated skills in designing a knowledge assessment. (Knowledge of relevant rules and documents.)

Demonstrated skills in conducting a knowledge assessment. (Questioning.)

Demonstrated skills in managing assessment stress

Demonstrated skills in logbook assessment

Demonstrated skills in briefing

Demonstrated skills in debriefing

Remarks:

PC.            Good working knowledge and skills in preparing and conducting practical examinations, in accordance with the CA Rules and Advisory Circulars, and/or the 172/141 exposition documents is demonstrated.

Yes/No

NT/NA

Demonstrated skills in designing a practical assessment (practical knowledge of relevant rules and documents)

Demonstrated skills in conducting a practical assessment

Demonstrated skills in managing assessment stress

Demonstrated skills in logbook assessment

Demonstrated skills in briefing

Demonstrated skills in debriefing

Remarks:

EC.            Assess ATS staff for the issue ATS instructor (OJT) Ratings

PC.            Good working knowledge and skills of how to assess that the instructor knows how adults learn is demonstrated

Yes/No

NT/NA

Demonstrated skills in assessing assessment of the 7 intelligences or learning styles and how it is applied in learning.

Demonstrated skills in assessing assessment of what characterises an adult learner and how they are best motivated.

Remarks:

PC.            A good working knowledge of how instructors are required to prepare written, visual and practical training aids and can prepare an individual training plan is demonstrated.

Yes/No

NT/NA

Demonstrated how to prepare an individual training plan.

Demonstrated knowledge of how to prepare ……………… training aid.

Remarks:

PC.            A good working knowledge of how instructors are required to conduct one-on-one theoretical and practical training sessions, taking into account the trainee’s particular learning style, is demonstrated.

Yes/No

NT/NA

Demonstrated a working knowledge of how instructors are required to conduct one-on-one theoretical training sessions, taking into account the trainee’s particular learning style

Demonstrated a working knowledge of how instructors are required to conduct one-on-one practical training sessions, taking into account the trainee’s particular learning style

Remarks:


PC.            A good working knowledge of how instructors are required to conduct one-on-one briefing and debriefing sessions is demonstrated.

Yes/No

NT/NA

Demonstrated knowledge of how instructors are required to conduct one-on-one briefing and debriefing sessions

Remarks:

PC.            A good working knowledge of how to reduce stress in trainees during training sessions is demonstrated.

Yes/No

NT/NA

Demonstrated knowledge of how to reduce stress in trainees during training sessions

Remarks:

PC.            A good working knowledge of how instructors are to write and complete training reports and records is demonstrated.

Yes/No

NT/NA

Demonstrated knowledge of how instructors are to write and complete training reports and records

Remarks:

PC.            A good working knowledge of when and how to initiate Training Interventions is demonstrated.

Yes/No

NT/NA

Demonstrated a knowledge of when and how to initiate Training Interventions

Remarks:


EC.            Assess the ability of ATS instructors (CHK) to issue ratings and validations

PC.            A good working knowledge and skill of how to assess that instructors know how to prepare written, oral and practical assessments is demonstrated.

Yes/No

NT/NA

Demonstrated skills in assessing assessment preparation

Remarks:

PC.            A good working knowledge and skill of how to assess that instructors know how to conduct written, oral and practical assessments in accordance with the CA Rules and Advisory Circulars, and/or the 172/141 exposition documents, is demonstrated.

Yes/No

NT/NA

Demonstrated skills in assessing the assessment process for knowledge and practical assessments

Remarks:

PC.            A good working knowledge and skill of how to assess that instructors know how to complete assessment documentation as required by the CA Rules and Advisory Circulars, and/or the 172/141 exposition documents is demonstrated.

Yes/No

NT/NA

Demonstrated knowledge/skill in completing the required assessment documentation

Remarks:


EC.            Complete the administration requirements for the issue of ATS licences, ratings, certificates of competency and validations

PC. .          A good working knowledge and skill of how to complete assessment and licensing documentation as required by the CA Rules and Advisory Circulars, and/or the 172/141 exposition documents is demonstrated.

Yes/No

NT/NA

Demonstrated knowledge and skill in completing the required assessment documentation

Remarks:

EC.            Describe the privileges and responsibilities of an ATS Examiner

PC.            A mastery knowledge of the privileges and responsibilities of an ATS examiner as prescribed in the CA Rules and Advisory Circulars, and/or the 172/141 exposition documents is demonstrated.

Yes/No

NT/NA

Demonstrated a mastery knowledge of the privileges and responsibilities of an ATS examiner as prescribed in CA Rule Part 65, Advisory Circulars and/or the 172/141 exposition documents

Remarks:

EC.            Have a mastery knowledge and skill of how to assess staff for the issue of ATS licences against performance criteria

PC.            Demonstrate a mastery knowledge and skill in comparing performance against performance limits.

Yes/No

NT/NA

Critical Area:

Fail

Pass

Demonstrated ability to detect critical or major errors eg. knowledge of mandatory fail aspects

Demonstrated ability to detect minor errors

Demonstrated ability to assess affective factors

Remarks:


EC.            Have a mastery knowledge and skills of how to act in the overall interest of safety

PC.            Demonstrate a mastery knowledge and skills of how to act in the overall interest of safety when conducting examinations

Yes/No

NT/NA

Demonstrated knowledge and skills in proactively reporting and dealing with safety incidents, defects etc.

Demonstrated knowledge and skills in cooperation, teamwork and a safety attitude

Demonstrated knowledge and skills in anticipation of safety issues prior to them becoming a problem eg. judgement

Demonstrated knowledge and skills in scanning

Demonstrated knowledge and skills in organisation and efficiency

Remarks:

Document Compliance Check

Documents Checked

Yes/No

NT/NA

Licence of candidate

Logbook of candidate

Last operational assessment of the candidate eg annual proficiency check (APA)

Previous assessments conducted by the candidate eg examination papers, completed assessments, training checks

Of those assessments conducted percentage of ‘passes’                   % in previous 13 months

CA Rule Part 172 Procedures as outlined in the exposition eg. Completed Instructor (OJT) Course, Instructor (CHK) Course, Examiner Course

Remarks:

Additional comment

Yes/No

NT/NA

Remarks: