I know by posting here I am reviving an older thread, but I have something to say! My background is in designing tests and measures and I think I can shed some light on a few things for some people.
For better or worse, I gave my test to Tim Just to grade at the US Open in 2016. I passed my Senior TD test this year in the same manner I passed my driving test: barely.
When taking the test, i was definitely aware the test is graded via an answer key. So, I do feel the grading is definitely CONSISTENT but it doesnât necessarily mean the grades are OBJECTIVELY best. Now, before you draw a line in the sand or agree with me, hear me out.
What is the goal of the test? This is always the first question before creating a test and if you cannot outline what youâre trying to measure then your test will be junk. I believe the goal of the TD tests is to ensure TD competency before awarding higher TD certifications (and the benefits that come with them). After all, you donât want a bunch of brand new people becoming NTDs because then you cannot objectively know who does or does not have the required skills when it comes time to select a TD staff.
Now that we have our goal, you have to determine how will the test be administered? You determine this before coming up with test questions because the types of questions you ask can be dependent on the format in which the test is given. A while ago someone, or more likely a group of someones, decided it would be an open book test (I do not know the history of the test design, I am extrapolating here).
Now that we know it is an open book test, we have to design the questions. When designing the questions, it remains pretty clear to me (without stating any of the questions publicly of course) that the original test designers understood that if the tests are to be open book that not only matches what a TD must do at an event when a rule is questioned (look it up) and swirl in the concept that scenarios are not always clear and a rule doesnât necessarily exist for every scenario that can come up.
Thus, my point is that if the test is open book and the test questionâs answers are ALWAYS located in the rulebook in perfectly clear fashion, then the tests would not be measuring a TDâs ability to solve issues on the floor but would instead be measuring their ability to look things up in a dictionary.
Therefore, I believe that just because the questions have an answer key only serves to show us that test graders are CONSISTENT in grading the tests; but, this is not a defense against the accusation that the questions often have multiple answers. Multiple answers, with some being more correct than others, is an internal-consistency measure to keep the test honest to its goal to ensure TDs are competent before moving to higher certification levels and allows the test not to be as simple as finding a word in the dictionary.
In other words, the test is designed with a compromise between not being able to administer the tests in person (ie: open book) and measuring competent TDs. If the TD culture were to change and want more emphasis on tests with more clear answers, the tests would need to be longer, they would need to be administered in person, and they would need more question-type variability (ie: essay, multiple choice, true false, etcâŚ)
Some have, but barely and rarely.
Generally a person can only pass the test closed book if they already have the experience prerequisites needed to take the test open book.
Because for those who take the Senior TD test closed book the experience requirements are reduced to being Chief TD of three Category C tournaments (some substitutions allowed.) Good luck passing it if thatâs all the experience you have.
Absolutely!!! There are so many issues covered in the test that merely (and this is difficult- reading and memorizing all the rules) studying and âknowingâ the text of the rules does not mean knowledge of how to âapplyâ the rules. It is very difficult to
bypass experience. It has been very helpful in my growth, esp, in my early growth as a TD to see and learn the approach
of so many other very capable tournament directors, and their approaches to dispute resolution, management styles, and different levels of respect for their chain of command. It is, at least it seems to me, for one to adopt their own methodologies, without the experiences of viewing the techniques of others. And as important, is the when â as WHEN should a rule be applied.
While itâs more consistent to have multiple choice, with answers as described, itâs also potentially less of a good test of a TD, and potentially less fair to the TD. I know. For the first time in decades I just took the Senior TD exam and did not pass. This was due to a few reasons.
First, in general, life has been hectic and a challenge over the past year and I allowed myself to rush through the test too much as a result. (Iâm on the tail end of recovering from a 3rd surgery [all is well] in last December, both my parents passed in 2016 and I am STILL dealing with estate issues, along with several work challenges. And Iâve been very focused on making a couple of upcoming chess events really great experiences, and have tried not to be distracted from those.) I own that this was my error in just not taking the time needed for the test seriously enough.
Second, on the prize problems (there were 2 I think) I rushed through both and made the same error on both. Yep, I blew it. I suppose in part because most of my tournaments over the last several years have had no prizes or no cash prizes. And then, there were a couple other questions that I got outright wrong. Thereâs always going to be a couple of those, with updates, and rules changes, weâll always miss something.
Even with these errors, I should have passed. The number wrong due to my own clodd-hopping sloppiness wasnât sufficient to cause a fail.
But, in addition, there were about a half dozen answers that I provided that were âwrongâ either because the particular test question was poorly worded, otherwise unclear, or in at least one instance (and arguably up to 3 cases) - actually incorrect about a rule, thus leading to a series of multiple choice âanswersâ that therefore made little sense. Frankly, Iâm not certain that I would have changed my answer on any of these half dozen, and the inability to have meaningful explanation or discussion included means that a significant amount of relevant testing information is lost.
Sometimes the poor wording isnât so much that the wording is horrible, its just that the wording isnât sufficiently crafted to direct the student to address specifically the issues that the TDCC is hoping to test with the specific question. If the wording is purposefully vague in order to ensure that the student is thinking the way the TDCC wants, then one must question whether a pure multiple choice test can get at that. Thereâs something problematic about tests with vague questions and set answers.
The result of this experience is a loss of confidence in the process and in the TDCC. Certainly, I got questions wrong, and was a little embarrassed about a couple I got wrong, and I own that.
But in this process we also must remember that US Chess is still a service organization, and an approach of âwe wonât talk to you about thisâ certainly doesnât âfeel likeâ a service organization. It really made me question the value of the process as a whole, as it was easy to see that this process wasnât necessarily providing the testing and/or learning experience I thought it was supposed to be. Maybe my understanding of the goals was wrong, but the âinstructionsâ to the test do say âThis test is designed to measure whether the applicant has the experience and judgment necessary to cope with such situations in the tournament hall at a âseniorâ level of competence.â In actuality, that isnât what the test seems to do.
I also inadvertently avoided a larger issue. I was going to point out one of the errors on the test, and had posted it to this forum, but then decided to talk to the TDCC about it instead (in spite of the stonewalling to my initial attempt.) Apparently, that post almost created a lot of trouble for me - there is apparently an unstated expectation that specific test questions never be shared, even though the test instructions donât indicate that: âThis is an âopen bookâ test, but each applicant is bound by his honor not to discuss specific questions with other persons.â is about not discussing questions with others with respect to taking the test, not with respect to follow-up questions on the test. So, for any inadvertent affront to the TDCC I sincerely apologize, but I do suggest that you be more clear in your exam instructions.
Generally, I greatly appreciate the work that the TDCC (and the Rules Committee) have done to volunteer to set up rules and processes for evaluating directors and running and maintaining those rules and processes. But the system should not be so closed to well-intended criticism that at the first sign of questioning or disagreement that it creates escalated conflict and animosity. I am concerned that may not be the case based on the responses I received.
Finally, while I understand that the TDCC wishes to be judicious of its time, I question the need for a two-month wait to retake in a situation such as this. I have nearly five decades of experience of directing chess tournaments; during most of that time (basically all the time there were testing procedures) Iâve been a Senior TD (I never wanted to be more.) Today I organize more than I actually TD, but being certified at a Senior Level is important for what I try to do. Frankly, if I made a point of listing myself in every event where I had some amount of meaningful TD involvement I wouldnât have had to re-certify. Apparently another error on my part. At any rate, it seems to me that this should be easier to waive under appropriate circumstances.
In all honesty - the senior TD test did its job. You admitted you were rushed when taking the test, missed two prize distributions questions. Does US Chess want a Senior TD that can not do prize distributions? Probably not.
If the testing instructions were unclear, you could have asked the TDCC for clarification; in regard to the testing instructions, Yes, it is a multiple choice exam with sometimes multiple correct answers but the exam does say to choose the BEST answer. I suggest re-reading the test instructions as there were partial credits to be had on the exam. For example, my answer âsheetâ to the senior TD exam was 17 pages because every answer, even trivial, was backed up with a rationale.
I would think that the reason the TDCC wants one to wait two months is hopefully the applicant would go over their exam and learn from the mistakes (not to take this exam rushed, brush up on the current 6th edition and its updated rules, etc.). Otherwise, itâs just a waste of your time to take the exam and their time to grade it and another test version you cannot take anymore.
I know itâs embarrassing experience to know you failed the exam but take it as a learning experience rather than âpuff, Iâve been Senior TD beforeâŚâ. For the record, I failed the Local TD test 8 years ago when I first took it. whatâs these pairing cards people keep talking aboutâŚ
One point that Mr. Bachler makes that struck a chord with me is the impossibility of discussing anything relating to a TD test with the powers that be at US Chess. I would characterize their attitude as aggressively hostile.
When I took the Local TD test I passed it easily, but there was one question the answer to which I was unsure of. Two choices on my exam looked equally good to me. I called US Chess and tried to find out which one was correct, pointing out to them that I had already passed the test. I was accused of trying to cheat for wanting to know this. The woman (donât know who it was) with whom I spoke took it for granted that my real reason for the call was so that I could give this answer to some friend of mine who had yet to pass the test.
First, for example, I CAN do prize distributions --in the past have done them correctly, and have gotten them correct on past tests, and have contributed to TD discussions on prize distributions. I have a long history of doing prizes.
Yes, I made a mistake here. But this mistake was not sufficient to fail the test. So the argument âDoes US Chess want a Senior TD that can not do prize distributions? Probably not.â While valid, appears to be irrelevant, given the history of the individual taking the exam. Have you ever rolled through a stoplight because you were in a hurry? The error here is more akin to that - not an inability to correctly calculate a prize - just the mistake of being less focused than necessary at this time. If Iâm actually doing a prize, and taking care of money for people, Iâd focus.
The testing instructions were not unclear - the questions were. At the time, the questions seemedclear because at the time I didnât understand that the TDCC was going for a specific target in the answer - a target that wasnât necessarily created by their question. Now that I know that, I know to look for it, and could ask for them to clarify a question during the exam. Iâve asked the TDCC multiple times since the exam, if they would be willing to do that, and so far they have not answered that question.
I did provide significant additional notes to the exam - multiple pages. It appears that had no impact on scoring, and based on what is written in this thread, had no impact on scoring. Had it had an impact, I would have imagined that I would have passed. However, my comments, which may have included elements of the âbest answerâ in choosing the âsecond bestâ didnât seem to sway anything when second best got 0 points. Further, when the question gets a rule wrong, its entirely possible that none of the 4 answers is âBestâ.
I canât imagine what would happen in the next 7 weeks that will make the next test experience better or that I will do better on the next test. I donât feel the test failure had anything to do with not being up-to-date, or unable to locate the best answer. I see the situation as one where Iâm already up-to-date, and highly motivated and focused
I can understand why it might not always be the case that a quicker retest would be ok, and for a number of reasons encouraging serial test taking is a bad idea. But I donât see that as applying here.
With respect to my chess students, Iâve always been fine with sharing embarrassing moments to help them to understand that we can learn from our mistakes. In fact, I recently shared an embarrassing loss in these forums at viewtopic.php?p=316490#p316490. Iâve already taken this test result as a learning experience, and listed the reasons in this thread. This is why I had already approached the TDCC about clarifying questions during the exam, that was clearly a large if not the largest factor here.
Iâm pretty sure the office still only has 3 versions of the club/local and the senior exams. If someone were to take all 3 tests within a few days of each other, that might make it difficult for the office to give an individual who failed the test 3 times a 4th test.
Thereâs also a presumption (or call it a hope) that the person being tested spends sufficient time between tests studying the material to prepare for another exam, whether thatâs by studying the US Chess rulebook or by working at events under a more experienced TD.
Could the TD testing process be changed into some kind of learning experience, using automated testing procedures and a Keller plan-like approach to test writing and grading? Yes, though thatâs a major change in TD training/testing philosophy and it would entail a LOT of work to prepare sufficient test questions.
In a Keller plan, exams become learning experiences rather than just knowledge assessments. That often means being told not only what the right answer is, but why the answer selected is wrong. (That makes choosing the set of possible answers more challenging, you canât or shouldnât just insert random garbage into possible answers. Think about coming up with several options for prize distributions at an event, each answer has to make sufficient sense to be a possible, though wrong, answer.)
The general rule of thumb in Keller plans is that you need a test bank that contains at least 10 X the number of questions in a typical exam. Questions are randomly selected when generating an exam, with a large test bank the odds of repeating too many questions goes down. (Most testing software would also randomize the answers, so that even if a question is asked again in a later exam, the answer might not be the same choice as before.)
So if the Senior TD exam is, say, 40 questions, that would mean we currently have a test bank of 120 questions (40X3) but need one of at least 400 questions. And it isnât clear that the existing questions would all be appropriate for a Keller learning experience, either.
Keller tests are often segmented, 3 questions on subject A, 5 questions on subject B, etc. This complicates the test-writing process. Suppose, for example, you wanted to have a question on the assigning of points (eg, half/full/zero point byes) for late entrants. Youâd need to come up with at least 10 questions on the issue to ask it once on the exam.
And if âpractice examsâ were included as part of the Keller experience, thatâd likely mean a separate set of questions (usually with annotated explanations of the answers.) I know professors who have taken a year or longer (working several hours every day) to write sufficient questions for a Keller plan course.
There are software packages that can be used to set up online tests, but thatâs a lot of work, too. Some of them are free software (Moodle comes to mind), others, like Blackboard, are very expensive packages that are sold to major universities. Iâm not sufficiently familiar with Moodle to know if it can do a full-blown Keller experience.
A question that often comes up with regards to Keller plans is this: Suppose someone get hold of the full test bank and memorizes all the questions and answers. The usual answer is: Thatâs fine, someone who has done that has probably assimilated the knowledge that the Keller plan was designed to teach. But thatâs one reason why the existing questions might not be adequate for a Keller plan approach to learning and testing.
Iâm not sure whoâs grading the multiple choice TD exams these days, possible Grant, but at times this has been a job for the office clerical staff. Iâm not sure it requires TD certification to grade a multiple choice TD exam.
If the missed questions were the prize distribution questions, then everyone should be forgiven and every answer choice should be regarded as correct, because the prize distribution algorithms are incredibly stupid anyway, and have been for at least 40 years. Usually just about any reasonably creative answer has a good chance of being better than the âcorrectâ answer.
The speed limits on the Interstate are incredibly stupid, too, but that doesnât mean I think the police should stop citing people for speeding.
A wrong answer under the rules is nothing more than a wrong answer.
if you donât like the rules, WORK TO CHANGE THEM!
Iâve always been amazed that nobody has put together a âprize distributionâ web page, which would be the ideal place to put forth alternate distribution rules and lobby for their adoption.
Prize distribution is an issue for organizers. While it is true that many TDs function as both organizer and tournament director of their club or personal events, the task of determining prizes is ultimately up to the organizer. Many tournaments have a split in functions with the TD dealing with pairings and rules, and the organizer dealing with other matters. The organizer has the responsibility to determine an equitable distribution of prizes. That may include making sure that class players are appropriately accommodated and not have prizes funneled upward into the prize pool for the top players. As I recall, one of the reasons for the switch to and the popularity of class section tournaments was because so many players complained about prize distributions in one section events.
Since there is often no âbestâ answer to some of the questions, it may be useful to rework the tests to either remove the questions , lower the score for such high stakes questions, or make them essay questions rather than multiple choice/guess. Having been a teacher and designed many a test for myself and colleagues over the years, the so-called âobjectiveâ exams give the least satisfactory depiction of the knowledge and the skills of the person taking the test.
My background is in designing tests and measures and I think I can shed some light on a few things for some people.
This was due to a few reasons.
whatâs these pairing cards people keep talking about⌠