Discussion about when ratings should become official

Running Chess Tournaments
1

I’m going to make an attempt to move this discussion out of the discussion on the sample XML file for ratings supplements. It would be nice if that thread could return to being on topic.

A few things.

Not every tournament waits two months before affecting the supplement. Tournaments held and rated just before the cut-off make the supplement that will be official about one month later. Tournaments that have problems getting rated may be held a couple of weeks prior to the cut-off and still not make it, in which case they wait until about 2.5 months before hitting the official rating. I’d guess that 1.5 months is close to the average delay.

Re-rating doesn’t just fix one players rating. It then goes through and fixes the rating of every player that was played after that fix, and their opponents, and their opponents, and their opponents, etc. Because of that re-rating a single game can cascade into thousands of ratings to be corrected. That is one reason re-rating takes a noticeable chunk of time to redo. When a correction is made to a two-year-old tournament that cascades quite a bit. I chose that example because there was an adult player that noticed that he was listed as having played in a scholastic tournament two years earlier. It turned out it was another player with the same first and last name, same state, similar ID number and similar rating. It’s possible that the scholastic player did not remember his ID so his parents printed the (incorrect) MSA page to show that the player was a member.

Prior to office procedure changes and Mike’s work on the rating process, rating tournaments wasn’t merely slow, some people would say it was glacial. Getting a tournament result mailed in the day after the tournament and then getting the tournament rated four months later was considered by many to be pretty good (though more people fantasized that maybe one day things would get faster).
Nowadays I get frustrated when a membership check made out to the USCF takes a week to get to the office and get processed so that those IDs can be created and the tournament rated. Unless you ban membership payments by check there will always be tournaments that take a week or so to get rated (or tournaments rated immediately but with non-members because the checks were not actually sent to the USCF). Some organizers have gone to the extent of requiring a valid USCF membership to even allow a player to register in a tournament, but not everybody has a market that will accept that stance.

If you use old ratings for section assignment and require more current ratings for pairings then there is a significant education effort. Considering how many complaints I’ve heard over the years over what I think are straightforward issues, this would require a lot of lead time. If you do not show the pairing-rating then you will have VERY vocal complaints about obviously inaccurate pairings. If you do show the pairing-rating then you will have EXTREMELY vocal complaints about allowing an ineligible player to play for a class prize.

Discussing this in the forums may bring out some of the issues, but won’t cause any changes. It would require Executive Board action and a Delegate’s Motion to actually get anything changed.

2

  1. While it is true in the abstract that “Fresh ratings are better predictors of outcomes than stale ratings,” over the kind of intervals you are discussing (a few weeks), the difference is literally to small to be measured.

  2. Why wouldn’t your argument imply that the TD should update the ratings every time they change, which now means every hour? If you don’t reach this conclusion, what is your basis for choosing one arbitrary “effective date” over another, other than custom and convenience?

  3. What about people who don’t use a computer to run tournaments or to download rating lists? Are you willing to make possession of a computer and Internet access a minimum requirement for being a TD? Sure, we have too many TDs already …

3

Those arguments make sense with regards to calculating new ratings - and that’s what we do. However I see two issues:

  1. Why does having ratings that are only a few games fresher have much positive effect for placing players in sections, awarding prizes and pairings? There are certainly negatives.

  2. Your argument makes sense with regards to more current “official” ratings - where re-ratings have occurred, most events in the time period have arrived (even those coming by mail), and errors have been noticed and corrected. It isn’t clear to me that more recent “unoffical” ratings are more accurate than the last “official” ratings until those things have happened.

4

I think if I had a chance to update my ratings in WinTD the day before an event with a file that just updated players membership expiration dates, updated provisional ratings (e.g. more games than on the monthly supplement) and ratings for past unrated players, then I would happily do that and use those ratings for the purpose of the event. I have always thought this would be a useful file to be able to request “on demand” from the TD area.

However, for published, non-provisional ratings, I would not want to change them until the next “official” supplement.

Chris Bird

5

Section placements should be done in whatever way is agreeable to the TD and the players. If the players wish to be paired against other players who are close to them in strength, then it makes sense to use the freshest ratings available, since those correlate with underlying strength better than stale ratings.

However, there seems to be a strong sentiment that even more important as far as section assignments are concerned than pairings in which players have reasonable chances, is letting players know in advance which section they will be in. Presumably players would be delighted to have the players with ratings recently risen above the ceiling for a section to be assigned to the next higher section. But they don’t want this to happen to themselves. So they prefer tolerate the over-ceiling players in their section, in order not to find themselves in a higher section because of recent improvement.

I’m not sure that this is true, or why it should be true (other than snarkiness about sandbagging), but some TD’s in this thread are adamant on this point, and I guess they know their customers. In that case, it makes sense to use the ratings from some date before the tournament for section assignments even though by the date of the tournament more recent ratings might be available.

If a player has played 15 games since the last “official” ratings, and only a few of them are “in”, the rating computed with the ones that have arrived is still better than the rating computed without any of them. Of course, it would be better to have more of them, or all of them, but I don’t see why it makes sense to wait until some percentage of them are in. The others will eventually come in, and the rating will get more accurate. But meanwhile the ratings are more accurate for the ones that have come in.

6

And that is why it doesn’t make sense to make all the unofficial ratings official. As it is now, everyone knows simply by looking at the date of the tournament what rating will be used. If you end that, every tla and every tournament advertisement would have to include the date which was to be used. The players would have to figure out what there rating was on that day. That might not be easy to do when re-rates of their opponents events possibly cause changes to their rating.

You seem to be worried about the atypical USCF member - someone who plays 15 or more games per month. More typical might be 4 games. Also, you appear to be assuming your claim: “rating computed with the ones that have arrived is still better than the rating computed without any of them”.

You are ignoring the possibility of mistakes which can be corrected when the players have time to notice them before the ratings become official. Any mistakes will make the new ratings less accurate.

Also, you would need to look to see if their is a pattern to the events which are rated quickly compared to those being mailed in. If the Game/30s are being rated quickly while the Game/120s are being sent in by mail, there could well be bias introduced which would make the intermediate ratings less accurate than those produced by waiting for all the results.

7

I’ve been carrying around an idea that is related to this discussion. Please forgive me if this is a bit orthogonal to Brian’s argument, though.

Nowadays, we see many tournaments being submitted promptly online, though some tournament reports straggle in more slowly, and some come by post. Reports are subject to holds due to unresolved membership issues, too. Ratings are subject to change due to “rerating”, which can occur due to a variety of reasons, such as getting rating reports out of order, or corrections to results that had already been rated. Rerating can change a rating long after it was officially published, not to mention sooner, too.

We issue “official ratings” monthly now, online; not long ago, it was only every other month, when the rating supplements were printed and mailed.

How about adapting to the new web-based world in the following way?

  1. Why not designate a period of, say, 14 days after each event as the period for submittal and review of the rating report. During this period, reratings might occur repeatedly, as out-or-order events are rated, events are rated after a wait due to membership issues, and corrections are made and accepted - a process that will become quite plausible if more and more players review the posted rating report within days of playing in an event. Post-tournament ratings on such events could be considered unofficial.

  2. Upon the end of the submittal and review period, then formalize acceptance of the rating report.

Under this model, your rating change would become official at the end of the 14-day period, though of course, it still would be subject to rerating processes for as long as USCF wishes.

Although USCF does not currently send email notification to players when rating reports containing games played by them are posted to the web site currently (because of issues related to overzealous spam-fighters, which might someday be resolved), the day may come when we can do this. If that happens, members will probably enjoy seeing their results right away, and if they spot errors, they will probably be reported quickly via the web, too. Such corrections would lead to higher quality (and make it harder to submit false rating reports, too!).

Players would probably chastise TDs who are slow to report, and reward TDs who report quickly with more loyalty to their events. TDs who do not submit online (smaller events, most likely) might find ways to do so (either via libraries, or with the help of players who have computers).

Money would be saved on rating fees, because online submission is at a lower rate per game, and by USCF on processing, too. And we’d all probably feel good about the speed and quality of USCF’s rating service, too.

8

Hal, suppose you are running an event today (August 25th.)

One of your players played in an event on Friday, August 10th.

His updated rating from that event would become official on the 25th of August.

Are you going to wait until midnight on the 25th to download an official ratings list update and start to look up the players in your event?

These kind of “Six Million Dollar Man” approaches (we have the technology, we can do it) don’t take into account the realities of tournament schedules and site conditions. There are still numerous sites that don’t have net access and there are still TDs who don’t pair by computer. (I even know one TD who still pairs by hand but submits his events online, typing them in using the online editing form on the TD/A website.)

Your 'money saved" argument is irrelevant, any change in when a rating becomes official wouldn’t change the cost savings from online submission.

You’ve been an advocate for the USCF making decisions based on sound business reasons, Hal. What are the sound business reasons for having official ratings list more frequently than monthly?

9

I agree Nolan. One of the biggest adult tournament in Seattle is paired mainly using the 3x5 card system, computers are not for everyone. Unfortunately; I believe that we need to develop processes to handle the lowest common denominator.

That said, I am excited to see the improvements that you have made in the website Hal.

Gregory

10 
 I really wonder about that.  If the entry fee were a dollar less to a tournament run by a "slow" TD, I think the players would pay the dollar less, and chastize the slow TD, and pay him the dollar less next time, and re-chastize him, and keep on paying and chastizing, paying and chastizing..
11

Your points are good ones, Mike.

I’m curious as to how long it takes to rate most events from the time the event concludes. For instance, what percentage of events are rated successfully n number of days after the event, for a few values of n that you think would be useful to share with us (if you do the analysis)?

Likewise, rerates can apparently cause a change in a rating months or even years after an event, but there must be a period during which such things are likely, and an ever-diminishing likelihood after that. Do you have data or a feeling about what the data probably tells about that?

As far as “official ratings” go, perhaps there are two or more different “needs” for them - one being the need for a TD to have a rating to use for pairing and prize purposes, and another being USCF’s need to identify specific points in time that are the basis of publishing lists such as top player lists or for eligibility and invitations for events or awards.

How do the official “top 100 lists” work now, in terms of frequency and such? (It’s been half a year since bimonthly rating supplements defined them, and those printed rating lists don’t look to be coming back, I assume.)

12

Top 100 lists are still being issued on a bi-monthly basis, because there is a lot of work the office has to do to check each new set of Top 100 lists.

As to how quickly TDs get their events in, here is a breakdown of how long it took between the ending date and the date an event was rated for the 7412 events rated during FY 2006-07, separated out by whether the event was submitted online or mailed to the USCF office. (I think the 0, 1 and 2 delay day events ‘mailed’ to the USCF office were all ones run in Crossville and brought to the office by the TD, who may have been a USCF staff member.)

[code] Delay Online By Mail

0 days 1650 3
1 day 1577 2
2 days 614 4
3-7 days 1221 132
8-14 days 469 496
15-21 days 172 197
22-28 days 139 107
29-70 days 258 197
71 or more days 82 92
[/code]

It’s harder to state what typically happens during rerates because as someone noted before a single correction in an event can cascade through hundreds of subsequent events, though that is usually not the case.

However, here’s how many records from events prior to 2007 have been updated as a result of rerates in July and August:

[code]year count

2003 63
2004 1508
2005 7822
2006 16031[/code]

By comparison, here’s how many records from events in 2007 (by month) have been updated as a result of rerates in July and August:

[code]month count

01 2418
02 3169
03 4348
04 3462
05 3221
06 4312
07 5289
08 1043[/code]

13 
I am curious if anyone else considers that every question posed to Mike takes valuable time away from his work on his list of priorities?
14

There is an issue I’ve not seen mentioned, what section you are eligible for. Some players may make a decision about playing in a given event based on what section that they are eligible for. This does not matter for in town events, but when you have to make hotel reservations, this can be significant. Another related issue is forming teams for National scholastic and pure team events such as the USAT. The National Scholastics normally use a somewhat older supplement date to avoid this problem.

Also in addition to the mentioned problem of updating the ratings at the last minute, the TD of a multiple section event also has to review the sections to see if the new ratings have made anyone ineligible for the sections.

When I worked on this question back around '98, I decided to state that if a TD wanted to use the current rating, they must have Internet access on site and use the current ratings for everyone.

15

what do you do if players’ ratings change DURING the tournament?

OK…OK…

The main issue here is that there must be an agreed upon date (and time?) on which ratings are frozen for pairing and prize purposes. When your event draws players who must travel to the event, you should announce the specific date. Usually, there is an “out” for players who are, say, UNR on that date but receive a rating that qualifies them for a class prize before the event starts. That is what is normally done at USCF National Scholastics.

But…remember that the TD can always assign a rating for pairing and prize purposes, as long as the assigned rating is not lower than the official published rating. Again - you probably do not want to do that to a player who has travelled a great distance and finds that your assigned rating makes him ineligible for the section he wanted to play in.

So, you need to: a) choose a cutoff date, b) advertise it if it will affect someone’s decision to play, c) perhaps provide a way to update some ratings (like UNR → rated), d) live with it.

In the good old days when ratings became “official” every two months, most tournaments didn’t have a problem with advertising cutoff dates. Some tournaments, held in the first week of the month, would advertise that the previous supplement would be used.

Now, with ratings calculated hourly, it’s a bit more complicated - but the principles of fairness still apply. Every player should (in principle) know what rating will be used - before they commit to travelling to the event.

for your weekly Quads, that’s the morning of the event. For a National Scholastic, that’s 2 months before the event. Your milage may vary.

16

As Ken states, you can always specify an alternate to the default of the current supplement. I believe this should remain our default position as most players are used to and expect this to be what happens. As Ken also noted, the TD keeps the right to adjust individual ratings for various reasons. Additionally, the organizer can specify MSA ratings as of a given time.
Regards, Ernie

17

A TD or organizer who states that MSA ratings as of some specified point in time will be used is IMHO asking for headaches, because MSA records are not time-stamped or archived and can change for a variety of reasons several times a day without any indication as to what the previous rating was or when the change was made or posted, much less why something changed.

In an extreme case, if someone timed it JUST RIGHT during the middle of an update cycle, the most current event shown for a player on MSA may only remain the most current event for a few seconds before a later one is posted.

Keep in mind that when MSA was designed (in 2002 or early 2003), we were still rating events in batches about once a week.