I am writing a Pairing program and would like to provide the features of importing users from the USCF files and sending in rating reports. I was wondering if anyone knows where to get the definitions of the formats. Thanks.
I don’t suppose you’ll be supporting linux???
TIA,
AJG
Actually, I am writing it in Java for the purpose of being able to use it in both Linux and Windows.
OOO, goody!
TIA,
AJG
Well, unlike you, I don’t plan to write a pairing program, but I have written programs to do a quick look-up (independent of Swis-Sys or Win-TD) from the USCF rating supplement files.
These files, as well as those produced BY the pairing programs, are dBase3 files. Although nobody has ever explained to me the structure of dBase3 files, a little detective work (with a DEBUG-type program) convinces me that the scheme is as described below.
(1) First there is a 32-byte overall header record. This record includes such information as record count, record length, and start-of-main-data pointer.
(2) Then there are one or more 32-byte field header records. There is one such record for each field name in the main file. Each field header record includes the name and size of the field.
(3) Then there is a single byte of 0D hex (carriage return), optionally followed by a single byte of 00 hex. (The files from USCF do not include this 00 byte; those produced by Win-TD do.)
(4) The main data records follow, each of length NN, where NN is specified in the overall header record.
(5) The file ends with a single byte of 1A hex (end-of-file marker).
The 32-byte overall header record consists of:
Byte 0: Always 03 hex. This could be a version number, as we are dealing with dBASE3 files.
Bytes 1-3: File date:
Byte 1: File year minus 1900 (0-255, representing 1900-2155 respectively).
Byte 2: File month (1-12).
Byte 3: File day (1-31).
Bytes 4-7: Record count (4 bytes, low byte first). This is the total number of main data records.
Bytes 8-9: Start-of-main-data pointer (2 bytes, low byte first). This points to the byte immediately following the 0D in (3) above, or immediately following the 00 if the 00 is present.
Bytes 10-11: Record length (2 bytes, low byte first). This gives the length of each main data record. All main data records are the same length.
Bytes 12-31: Always 00 hex.
The 32-byte field header records each consist of:
Bytes 0-10: Field name, up to 10 characters, padded with trailing null characters out to length 11 (thus there is always at least one trailing null). Legal characters in the field name appear to be uppercase letters, digits, and underscores.
Byte 11: Always “C” or “D”. I don’t know what this is for. Possibly “D” means numeric and “C” means non-numeric, but it seems safe to always use “C”.
Bytes 12-15: Always 00 hex.
Byte 16: Field size (1 byte).
Bytes 17-31: Always 00 hex.
The main data records consist of a space character (I don’t know why) followed by the values (in character form) for each field. Thus, the sum of the field sizes specified in the field header records, plus one (to allow for the space), will always add up to the record length specified in the overall header record.
That ought to get you started. More to follow.
Bill Smythe
Here’s the promised follow-up. The main file produced for USCF by the pairing programs is TDEXPORT.DBF, in dBase3 format as described previously. The fields in this file are:
D_EVENT_ID – 9 characters, yymmdd followed by “001” (same in every record).
D_SEC_NUM – 2 characters, blank followed by “1” for simple tournaments (same in every record).
D_PAIR_NUM – 4 characters, nnnn, leading blanks when < 1000.
D_REC_SEQ – 1 character, always “1” for simple tournaments.
D_MEM_ID – 8 characters, USCF ID number.
D_RND01 thru D_RND10 – 5 characters each, Wnnnn or Dnnnn or Lnnnn, with leading blanks in nnnn when < 1000. Four blanks followed by “0” for round numbers beyond the number of rounds in the tournament.
You should probably get ahold of a copy of WinTD or Swis-Sys, make up a small tournament, and use a DEBUG-type utility to examine the diskette files produced by these programs.
Bill Smythe
TDEXPORT.DBF is the player table. While it has the most data, the other two tables, THEXPORT.DBF and TSEXPORT.DBF are also necessary.
All three tables are missing important information. Here are a few examples:
TDEXPORT.DBF has the player ID but not the player’s name.
TSEXPORT.DBF has a field to identify the chief TD and assistant TD, but only one of each. Large tournaments can have many assistant TDs. It also does not have any information about time controls.
THEXPORT.DBF does not have a space for the chief TD for the event.
There will be a new file format out some time this summer. It will address the issues above, plus others, such as identifying cross-round pairings. While it isn’t needed to rate the event, it would be interesting to have the color.
Thanks for the info, Mike.
Which brings us to a point I brought up in RGCP, and in private email to you:
What if some pairing program were to simply add a new field or two, while still keeping everything in dBase3 format, and keeping all existing fields exactly as they are? Would the existing software still work? I’m guessing the answer would be yes.
If so, this could ease the timetable for authors of pairing programs. They could be preparing their new versions now (for that matter, they could have done so years ago), knowing that the files they were creating would not cause existing USCF software to choke – even though the existing software would not be able to make any use of the new fields.
Then, when the USCF software is revised, tournament reports will already be pouring in with the new fields, which would smooth things tremendously in the USCF office.
Incidentally, instead of just adding the player’s name to TDEXPORT.DBF, wouldn’t it be helpful to also add the state, expiration date, and rating?
I agree it would be interesting to have the colors for each pairing – and it would also be interesting to include this information in the published crosstables, whenever available.
Bill Smythe
Are you interested in a program that runs on Linux? One exists and it has USCF pairing capabilities but not the reports in the required dBase format for electronic submission to the USCF.
Steve
Mine will support Linux, USCF capabilites and will export the necessary files. Which program are you talking about? Vega?
Smythe Dakota, thanks for the quick information on the file format. Should not be too difficult to figure out.
What kind of format will the files be in? XML? dBase3 is a very old format which seems kind of silly to keep using. Thanks for the information. Hopefully the new format comes out sometime in the next month.
Yes. Vega. I have run the pro version on both Linux and Windows and it works nicely. It also converts the USCF rating files into the Vega, fixed format. Luigi used the latest USCF rules for making the pairings. The big problem was the dBase formatted files that the USCF requires. I agree with your later post that dBase is obsolete. There was discussion previously about the new format being XML or tab delimited or CSV. I think it is still up in the air.
How far along is your program? Java would be nice as long as the Microsoft extensions are not used.
Steve
For development testing, I’m using tab-delimited files, because they’re the least work to parse and I can create them with either a text editor or something like Excel. I’m also testing uploading of the current dBase files, though I have yet to figure out how to get that data over to the current ratings system.
The only way we came up with to get the data to the old ratings computer when we tried it last week was to copy the files to a diskette on the database server system and then use that diskette as if it had been sent in by the TD. Given that the server is in a locked room, that’s not very practical.
XML strikes me as being a bit of overkill.
As far as adding fields is concerned, I don’t have any way of testing it but since the programs are written in Clipper, I think it is likely that the current ratings programs would just ignore any columns they aren’t expecting to use. I don’t know if the reverse is true for adding columns to the ratings supplement files. Tom and Thad would have to answer that question.
I don’t know what type of server is being used, but can you not use sftp or ftp to send the files to the server?
Steve
The ratings programs currently run under CLIPPER (a DOS program) and the data files are on a Novell 3.1 server.
The Linux-based database server doesn’t have the ability to connect to Novell, and I’m not sure if we set up an FTP server on a Windows system if it would be able to access the Novell drives. That might be worth trying after they get everything moved over to the new building, though.
Isn’t it also true that the Clipper programs have “A-colon-backslash” hard-wired into the file names, so that ONLY a diskette can be used, and that the source code for these programs has disappeared?
Bill Smythe
When Nancy and I tested it last week, it looked like it would also access a file on the local hard drive but not a drive on another system that was mapped to a local drive letter across the network.
I’m not sure which programs they have source code for anymore, but since I don’t do Clipper programming and don’t have any way of getting to those systems or files remotely, I think we have to look for other solutions. The best solution, of course, would be to get the new programming working, but that takes time.
I’m curious if the java project mentioned was ever completed, as I recently released my free TDX, also a java app. I’m still adding the dbase3 export stuff, though, but all the other required functionality is done.