Master-citation backfill

Done

• the study's tools (new, read-only module). It pulls record identifiers out of master citations (text, url, ancestry_url) and normalizes them. The kinds are:

  • FamilySearch persona ARK and FamilySearch image ARK;
  • Ancestry record (discoveryui and search/collections are one form), Ancestry image, and member-tree media;
  • Newspapers.com page and clipping;
  • Find a Grave memorial (URL or "memorial N");
  • Washington Digital Archives record and Missouri death certificate;
  • StABE signature, page and entry number;
  • Chronicling America, Fold3, the NPS soldiers database, Internet Archive, and online obituaries.

  Background sources have no identifier. The module also has:

  • names_person(), which decides from the citation text whether the master's person is in the record:
    • it returns named, unsure or absent;
    • it checks for relative annotations ("[his father]"), namesakes (by lifespan or age), same-named relatives (settled by suffix, age or the spouse named nearby), a wife named by her husband, and household lists;
  • fs_personas(), which labels each FamilySearch ARK (the principal, "father's entry", "also", and so on) so that a row is linked only to the person's own entry;
  • load_review() for hand decisions in the study's data. That file is not yet created.
• the study's tools (new). It runs dry by default; --report FILE writes the full list and --apply writes. Its behaviour is in the docstring:
  • each row gets its natural key through occ_ids.assign(key_override=…): fs:, fag: and mo: for the record's own person, fag:<id>|<H-ID> or mo:<file>|<H-ID> for another person named in it, and <site>:<H-ID>|<kind>:<id> for the rest;
  • an existing unlinked row of the same record is linked through edits/<date>-backfill-links.json;
  • an entry already linked to someone else is reported as a conflict;
  • masters are never touched, because only checks that the occ: IDs it finds exist and ignores them.

Dry run (latest; nothing applied)

The 4 conflicts:

• O03168 (Marie A Hegnauer) is linked to H0152, not H0023/H0069;
• O03362 (the father's entry, Leonard D.) is linked to H0027, not H0026;
• O03146 (Mary Lee Hegnauer) is linked to H0027, not H0230;
• O04348 is linked to H1474, but H1462 cites it.

Review of the add list (about 400 of 630 rows read) found no false links after these fixes:

• the Mrs./Mr. title against the person's sex;
• a wife's mention dated before her marriage;
• the subject of the memorial or certificate against a secondary person;
• namesake children (a baptism date used as the birth year).

Remaining (in order)

1. Fix before applying.
   • The "full given names" rule must ignore initials and honour "Mrs." (H0375 obitidx wrongly matched "Mrs. Richard W. Yeo").
   • Year-only lifespans should give the death year for burials.
2. Create the study's data.
   • Mark skip for the member-tree stories H0250 story and H0490 rgsm12.
   • Decide the 268 unsure items.
   • Skim the 589 context items for missed mentions ("Criss Hegnauer and wife", "his estranged wife").
3. Apply. Run python tools/backfill_citations.py --report … (dry), then --apply.
4. Extend the study's tools (task 2, not started). Using cited_records, add:
   • a WARN for a citation whose record identifiers match no occurrence;
   • a WARN for a citation that names the person with no occurrence linked to them (the review file suppresses these).
5. Link grades (task 3, not started): the study's edit files via the study's tools.
   • Seed rows O00001–O00244 go to confirmed where the image was read.
   • Unlink O00190 and O00191 from H0038.
   • O00652 and O05907 go to possible.
   • H0155's O00756, O01130, O01392, O01786, O01807, O01836 and H0017's O00311, O02543, O03358, O03424, O03936 go to confirmed.
   • O02601 gets a note.
6. Run python tools/check.py and record the counts here.