For anyone who wants to see how I generate the combined database, I have made a git repository available at http://www.sudburyriverchess.com/git/ratings. You can see the scripts behind the curtain by creating a local copy of the repository:
(Clicking on the URL directly will just give you a blank page.)
If this message makes about as much sense to you as it would if it were written in Aramaic, well, I put the āNERD ALERTā header at the top for a reason.
Iām happy to announce a new feature coming to the combined database!
Currently, the combined database contains data for exactly those US Chess members whose ID appears in the corresponding golden database. However, members do join or renew after the cutoff for the golden database. The custom rating lists I use to get the membersā full names also includes expiration dates and ID numbers as of the request for the custom rating list.
Starting with the September file, the combined database will now include up-to-date member expiration dates and will also include (almost) all US Chess ID numbers. (I have to fudge that last statement. The custom rating lists I use include all fifty states plus PR, ON, AE, AP, QC, BC, VI, and GU (the most common āstatesā in the golden database that are not actually US states). New members in other āstatesā will not appear in the combined database until they appear in the golden database.)
US Chess has been working on a new format for rating supplements which includes some FIDE information as well as some other useful fields. The format being tested is a JSON data file.
However, at this time there are no plans for it to contain FIDE ratings. Thatās because the release cycle for US Chess ratings lists and the FIDE rating lists are not compatible, so either the US Chess information will be out of date relative to the FIDE data or vice-versa. US Chessās official position is that organizers running FIDE events should go to FIDE for up-to-date ratings data. US Chess tries to maintain accurate linkages between US Chess member IDs and FIDE IDS.
Testing on this new format is proceeding, though slowly, and there is as of yet no commitment that this format will ever become an officially supported format.
Should the format change, I will continue providing this service with a combined database in the new format. As I have previously stated, the source for the FIDE ratings in my database is the FIDE rating list (FRL) that FIDE officially announces the last day of each month (to take effect the following day). It is true that I have seen ratings change during the month (and the players_list.zip file updated), but I donāt know why that happens. In any case, I feel safe using the official published ratings effective the first of the month.
Something like this really needs to be a regular thing. It doesnāt need the FIDE ratings. Thatās easy enough to merge together for those who need it. Have there been any changes or an update on this file? I sent you some notes on the file you gave me a while back. Did you see them by any chance?
Thanks a lot for doing this each month Ken. I use this database every month since it has players full name and the FIDE information is also nice for the occasional FIDE rated tournament I help out with. Unlike the version of the database provided by US Chess, your version of the database includes duplicate and deceased players IDās. Is there any way those can be taken out of the database?
If we take deceased players out of the database completely, what would happen to the name displayed in crosstables for those players? (Answer: It shows up as an error, there are around 500 IDs deleted from our database back in the 90ās for whom this already happens, because we no longer have records for those IDs.)
It doesnāt happen often, but sometimes we have to reactivate a deceased player so that an event that person was competing in can be rated. After the event is rated, we change the status of that player back to deceased.
Unfortunately, there is no āstatusā field in either the monthly or golden master rating supplement formats, probably because it did not occur to the people designing those formats many years ago to include one, and adding one would break at least one of the pairing programs which utilize those files.
What about duplicate IDs? Is it possible to take those out completely? Could the crosstables including the duplicate IDs be updated with the correct ID?
There are over 500 IDs flagged as duplicates that are still in the crosstable system, which probably means that the ratings for those IDs were never merged together.
They might need to be reviewed on a case-by-case basis, to prevent updating them without losing track of what the currently are in those events and without causing a large number of players to see a change in their rating after what would almost certainly be a massive rerate would be complicated.
Changing the player ID in an event before the rerate window (events initially rated after 1/1/2004) causes the pre and post event ratings for the changed players in those events to show as blanks.
When we looked at it a few years ago, we did not not see a reasonable solution to these issues.
In other words, it is complicated enough that doing anything might best be done as part of the re-engineered database and MSA system.
I donāt think we want to change whatās currently shown on MSA for duplicated IDs.
TDs who are using current ratings (as opposed to official published ratings) are looking those players up on MSA anyway, arenāt they? So theyāll see the duplicate ID.
One possibility might be to change the name field in the rating supplement files to show something like āDUP,SEE 99999999ā instead of the playerās name. That fits in the 17 characters currently allocated for the name field in the supplement files, but it might cause other problems so it would need further study, including consultation with TDs and the pairing program authors. And there are people who use the supplement files for things other than looking up ratings, it might interfere with that.
Iām sorry for the failure. I have an automated procedure that produces the combined database, uploads it to the web site, and posts the announcement in this thread. (No, Iām not up at 3:30 AM EDT generating the database!) Apparently something went wrong and my script did not catch the error.
In any case, the August combined database is now available. I regret any inconvenience to the users of this database. As I am at the U.S. Open, I canāt investigate what went wrong until I get back home.
Iām doing some cleanup on my web hosting server. I have old combined databases going back to 1/2014. (Wow, Iāve been doing this almost six years!) Those old databases are taking up a fair bit of space, and I really doubt anyone needs them any longer. So, Iām purging all the combined databases before 2019.
If you were counting on these being available ā¦ oops, sorry.
Thanks, Ken, for doing these. It has been a real help. How you scrape the email download for the longer names is neat. But, what a necessary ugly hack. Itād be nice if there was a better way. I know there currently isnāt.
If there is a new supplement/golden master format defined at some point, it could include a longer name field. (17 characters is kind of small, we have hundreds of members whose last name is longer than that. But that spec was defined long before I was doing IT work for the Federation.)
Be aware, though, that even in a longer name field truncation is sometimes necessary, because we limit the combined length of the prefix/title, first name, last name and suffix fields to 30 characters.
If? Weāve been asking for a better format for a long time now. In the meantime we will continue to use Kenās version of the database as it is superior to the one offered by US Chess (has players full name, has all three OTB US Chess ratings).
There is a SwissSys database configuration file for use with the combined database at https://www.sudburyriverchess.com/ratings/uscf-fide.dbp. If you havenāt already configured SwissSys to use the combined database, you can download this file, launch SwissSys, choose āDatabase Setupā under the āDatabaseā menu, and click the āLoad Settingsā¦ā button. Just change the āDatabase fileā field in the āDatabase Setupā dialog to point to the directory where you have placed the uscf-fide.dbf file and click the āOK - Applyā button. If youāve already configured SwissSys to work with the database, you donāt need the uscf-fide.dbp file.
There is a SwissSys database configuration file for use with the combined database at https://www.sudburyriverchess.com/ratings/uscf-fide.dbp. If you havenāt already configured SwissSys to use the combined database, you can download this file, launch SwissSys, choose āDatabase Setupā under the āDatabaseā menu, and click the āLoad Settingsā¦ā button. Just change the āDatabase fileā field in the āDatabase Setupā dialog to point to the directory where you have placed the uscf-fide.dbf file and click the āOK - Applyā button. If youāve already configured SwissSys to work with the database, you donāt need the uscf-fide.dbp file.
