Uncertainties in metadata

Introduction

  1Uncertainties are a common challenge in scholarly work. It is fundamentally important to point out uncertainties in order to make the research process and the results transparent and comprehensible. In contrast to natural sciences (e.g. mathematics), the humanities often miss the opportunity to model uncertainties exactly. The handling of uncertainties, especially when working with historical sources, is therefore usually explained descriptively—so far, however, lacking a best practice regarding the retention of uncertainties or decision-making in digital editions.[1]

  2Basically, editors face three challenges: First, uncertainties must be identified, second, they must be modelled/made comprehensible, and third, they must be commented on or interpreted by the editor. These steps must be taken into account in the creation of editions—not only in the main text but also in the metadata. The importance of metadata is significant for proof and transparency of research, for findability, distribution and interlinking as well as for their obvious potential for evaluation methods (e.g. network visualizations).

  3The core idea of metadata lies in having a simple and accurate data set that summarizes the most important information about the particular information resource. Metadata has been playing an increasingly important role in the research process for some time, and also has become a subject of research in its own right.

  4Good metadata facilitates interoperability. Describing/mapping uncertainties in metadata does not necessarily impede this. The goal must therefore be to identify uncertainties but not to fall back to project- or case-specific solutions or data models; otherwise, such an approach threatens the accuracy of scientific research as well as the prospect of interoperability. Precision of the metadata with regard to the disclosure of uncertainties is therefore urgently needed.

Modelling uncertainties in the TEI

  5In principle, the TEI guidelines take into account two types of uncertainties: On the one hand, uncertainty regarding the use of the appropriate TEI element and, on the other hand, uncertain ways of reading text passages/words or the identification of entities like persons or places. Both areas offer opportunities to highlight uncertainty regarding entities.

  6This article concentrates on uncertain readings as well as dealing with various forms of supplementary information.[2] For the encoding of insecure readings and related information, the chapters 11.3.3.2 and 21 of the TEI Guidelines are of relevance (Chapter 11.3.3.2 "Use of the <gap>, <del>, <damage>, <unclear>, and <supplied> Elements in Combination";[3] Chapter 21 "Certainty, Precision, and Responsibility"[4]). When modelling uncertainty, it is essential to specify the responsibility (@resp or <respons>), the degree of uncertainty (@cert, @precision) and the reason (@reason or via <note>) in order to make the decisions comprehensible for readers. For this purpose, a gradual encoding of uncertainty is possible using @cert with fixed values (high, medium, low, unknown). This area seems to be well covered by the existing possibilities.

  7However, problems arise when uncertainties in metadata are to be modelled and these relate primarily to entities and datings. As we observe, there is a certain arbitrariness in the encodings, which may be a signal of inadequate documentation or uncertainty in dealing with the current guidelines. Certainly, the described characteristics of metadata (see introduction) also play a role.

  8This impression was reinforced by the discussions and the examples presented in the workshop preceding this handbook on encoding correspondence. The aim of this contribution is to inform about existing possibilities, to give examples as well as to address some existing gaps and to develop coding suggestions.

General challenges

  9The challenges of modelling uncertainties of metadata are an integral part of every modelling process in the humanities and accompany editorial projects as well as database-driven development or cataloging projects. The fundamental dilemma in this context is the requirement of metadata for information that is as granular, structured and standardized as possible, and the widespread approach of researchers to enter information that circumscribes uncertainty or to restrict themselves to one of several possible entities. Typical forms are "around 1700", "Late Middle Ages", or "located between Cologne and Dusseldorf". Thus, regardless of the following recommendations, it is necessary to understand the relevance of metadata for automated processing steps, i.e. for further processing of the data in search portals or interfaces. Only if the researchers become aware of this and are willing to engage in structured and standardized information input in the metadata, also with respect to uncertainties, the quality of metadata dealing with uncertainties may sustainably be improved.

10Another challenge is the often unsatisfactory authority data. The use of authority data is a key factor in ensuring that metadata is reusable. In particular, authority data can be used to identify entities in the metadata with added value but there is often a lack of differentiating information in the authority data. To give one example, authority datasets do not include identifiers for all territorial units. While there is a GeoNames identifier for today's state of Bavaria (2951839),[5] identifiers describing the electorate, duchy, or the scale of the early modern Upper and Lower Bavaria are missing. While authority data are used to uniquely identify an entity, they cannot be seen in the same way as content-rich biographical reference works or lexicons but the combination of an authority record with incomplete or non-scientific information poses a problem here. The only remedy is the insight that norm data serve primarily for identification and only secondarily for description.

11Thus, in order to model uncertainties in metadata, one needs to be aware, firstly, of the benefits of metadata and the fact that they are not suitable for mapping all possible interpretive variants, and, secondly, the perspective on norm data as a reliable means of identification and less as a resource for further information.

Recommendations for coding uncertainties

12In the metadata, uncertainties occur at almost any location where information from the source is recorded. If a source document is undated, multi-dated, or ambivalently dated, such as being ambiguous about the underlying calendar, editors feel compelled to identify the most plausible date and assign it to the source. Likewise, location information may be missing or may not be specified with sufficient accuracy. For instance, the resolution of an abbreviation may not be unique, the same place names may occur in different areas, or historical places or field names are no longer common. Similar problems arise with regard to the identification of senders and recipients.

13As general recommendations for the coding of uncertainties in metadata we suggest the following general rules:

1. Always identify uncertainties in the metadata; the recognition, identification and interpretation of uncertainties is a scientific achievement,
2. Render the process of coding transparent by indicating who is responsible for encoding the uncertainty,
3. Distinguish whether the uncertainty is source-related (intrinsic) or based on inaccurate or ambiguous identification options (extrinsic),
4. Name methods (@evidence with recommended values "internal", "external", and "conjecture") to make transparent in which way the uncertainty was resolved,
5. If possible, refer to external references (authority data, prosopographic databases, etc.) in order to give researchers the opportunity to familiarize themselves with the phenomenon at hand,
6. If possible, add indications on how to process the information for analytical and indexing purposes.

What about degrees of uncertainty?

14Apart from these general recommendations, the question of whether the degree of uncertainty should be encoded is a matter which researchers always have to answer anew for each edition project as there are until now no common guidelines or best practice models. The most straightforward answer to this question is the statement that an indication is either certain or uncertain and therefore establishing a degree of uncertainty is a far too subjective undertaking.[6] In our opinion, it would be advisable to dispense with the specification of granularity in the metadata and to state only generally whether the information is secure or uncertain. The (implicit) default specification reasonably being @cert="true", the statement @cert="false" would only need to be set in cases of uncertainty. However, this would require a corresponding change in the content model of teidata.certainty, which introduces these Boolean values as alternatives to the existing options (in order to allow for backward compatibility). To remain in line with the current options offered by the TEI, projects might provide an indication whether @cert="medium" shall be evaluated as certain or uncertain by applications that distinguish only these two statuses (with "unknown" and "low" defaulting to uncertainty and "high" defaulting to certainty).

Examples of uncertain information and proposed solutions

15Strategies for dealing with uncertainty often include the recording of different options and their evaluation for plausibility and possibly mutual compatibility. There is a need to record possible alternatives and to assign a priority to at least one of them, pursuing the goal to maintain transparency and select a "best option". This prioritised option may then be considered by default in the processing of metadata (such as inclusion in registers, search indices, etc.). In Chapter 16.8 "Alternation", the TEI offers several basic approaches on how to capture and prioritise alternatives.[7]

16One such mechanism is the select attribute (see the following explication), which allows one or more of the detected alternatives to be specified starting from a hierarchically superior element (in the form of space-separated tei.pointers). Because @select mostly refers to identifiers (@xml:id), this procedure is well suited for the recognition of named entities such as places or persons. For time units such as dates or periods of time we are more reluctant to use @xml:id and prefer the use of @select in combination with @n or @ana. This offers the possibility to provide more detailed descriptions. The evaluation of @select would thus use not one specific but two different reference schemes, which is not very elegant and presents additional challenges for processing routines. The appropriate schema would either need to depend on a @type or be based on the syntactic structure of the pointer.

17An alternative approach to referencing specific metadata is local tagging directly at each element. This approach seems to be more consistent. At the same time it facilitates clarity and data processing. This kind of marking can be done explicitly (e.g. via @ana) but it can also be done implicitly by creating just one element of each type—carrying the information that is deemed most precise or probable—directly within <correspAction> and specifying all alternative places, persons and data only at a deeper level. Using this approach, only one place of dispatch, address, recipient and sender as well as a date of dispatch and date of receipt would be specified directly within <correspAction> and more detailed and possibly alternative information might be located in <note> elements, (partially) structured or in prose. Given that letters may be written and sent by more than one person from and to more than one place and at more than one date, it is hardly possible to narrow the schema to allow only one element of each type directly within <correspAction>.[8] Thus we can only suggest a best practice but not enforce it formally.

<correspAction type="sent"> [...] 
	<date resp="#editor-XY" when="1753-04-20"/>
	<note>Bei <bibl>B2462</bibl> wird 1750 als Jahr angegeben,
	was aber inhaltlich nicht stimmen kann (1750 AVH nicht
	in Bern; die angekündigte Rückkehr nach Göttingen, falls
	er in Bern kein Amt erhält, passt auf Ende April
	1753).</note>
</correspAction>

<correspAction type="sent" select="#date_pref"> [...] 
	<date resp="#editor-XY" when="1753-04-20" n="date_pref"/>
	<date resp="#author-B2462" when="1750" n="date_alt1">
		<note>Bei <bibl>B2462</bibl> wird 1750 als Jahr
		angegeben, was aber inhaltlich nicht stimmen kann
		(1750 AVH nicht in Bern; die angekündigte Rückkehr
		nach Göttingen, falls er in Bern kein Amt erhält,
		passt auf Ende April 1753).</note>
	</date>
</correspAction>

<correspAction type="sent"> [...] 
	<date resp="#editor-XY" evidence="conjecture" when="1729-04-21">
		<note type="evidence">ohne Jahr, eindeutig 1729; Notiz
		von AvH: "du 19 d'avril, reçu le 22"; AvH hat einen
		Rechnungsfehler begangen, 11. ante Kal. Maij ist der
		21.4.</note>
	</date>
	<note type="alternativeDate">
		<date calendar="#roman">11. ante Kalend. Maij</date>
		<p>Die Lesung II. ante Kal. Maij kann nicht stimmen.</p>
	</note> [...] 
</correspAction>

<correspAction type="sent" select="#date_pref"> [...] 
	<date resp="#editor-XY" evidence="conjecture" when="1729-04-21" n="date_pref">
		<note type="evidence">ohne Jahr, eindeutig 1729; Notiz
		von AvH: "du 19 d'avril, reçu le 22"; AvH hat einen
		Rechnungsfehler begangen, 11. ante Kal. Maij ist der
		21.4.</note>
	</date>
	<date calendar="#roman" n="date_alt1">11. ante Kalend. Maij
		<note>die Lesung II. ante Kal. Maij kann nicht
		stimmen</note></date> [...]
</correspAction>

<correspAction type="sent"> [...] 
	<date resp="#editor-XY" evidence="internal external conjecture" when="1768-05-01">
		<note type="evidence">Ohne Jahr; Jahr gemäss alter
			Sign., <bibl>GEH</bibl> und Inhalt (O. schickt den
			dritten, vierten und fünften Band seiner
			Zeitschrift, die AvH dann ab dem 13.4.1769 in den
			GGA rezensiert. Wie AvH dort schreibt, umfasst Bd. 5
			die späten Monate von 1766 und den ersten von 1767.
			Da er erwähnt, dass der 6. Band (der also die
			späteren Monate von 1767 und den ersten von 1768
			enthalten sollte) leider noch nicht fertig sei, muss
			der Brief (wie auch bei Thormann angegeben) 1768 zu
			datieren sein).</note>
			<note type="alternativeDate">
				<date calendar="roman">Cal.Maij</date>
			</note>
	</date> [...] 
</correspAction>

<correspAction type="sent" select="#date_pref"> [...] 
	<date resp="#editor-XY" evidence="internal external conjecture" when="1768-05-01" n="date_pref">
		<note type="evidence">Ohne Jahr; Jahr gemäss alter
			Sign., <bibl>GEH</bibl> und Inhalt (O. schickt den
			dritten, vierten und fünften Band seiner
			Zeitschrift, die AvH dann ab dem 13.4.1769 in den
			GGA rezensiert. Wie AvH dort schreibt, umfasst Bd. 5
			die späten Monate von 1766 und den ersten von 1767.
			Da er erwähnt, dass der 6. Band (der also die
			späteren Monate von 1767 und den ersten von 1768
			enthalten sollte) leider noch nicht fertig sei, muss
			der Brief (wie auch bei Thormann angegeben) 1768 zu
			datieren sein).</note>
	</date>
	<date calendar="roman" n="date_alt1">Cal.Maij</date> [...]
</correspAction>

<correspAction type="sent"> [...] <date resp="#editor-XY" cert="high" evidence="conjecture" when="1777-12-24">
		<note type="evidence">Eindeutig als 1771 zu lesen. 1771
			jedoch nicht möglich, da im Brief der
			<bibl>Usong</bibl> (dt. 1771, frz. 1772) und der
			<bibl>Alfrzed</bibl> (dt. 1773, frz. 1775) erwähnt
			werden. Auch erhält C., der als Chan. de Brioude
			unterschreibt, diese Funktion erst 1775. Also wohl
			doch 1777 (wie bei <bibl>GEH</bibl>), dafür spricht
			auch die alte Signatur. [hs/1/96]</note>
	</date> [...] 
</correspAction>

<correspAction type="sent" select="#date_pref"> [...] <date resp="#editor-XY" cert="true" evidence="conjecture" when="1777-12-24" n="date_pref">
		<note type="evidence">Eindeutig als 1771 zu lesen. 1771
			jedoch nicht möglich, da im Brief der
			<bibl>Usong</bibl> (dt. 1771, frz. 1772) und der
			<bibl>Alfrzed</bibl> (dt. 1773, frz. 1775) erwähnt
			werden. Auch erhält C., der als Chan. de Brioude
			unterschreibt, diese Funktion erst 1775. Also wohl
			doch 1777 (wie bei <bibl>GEH</bibl>), dafür spricht
			auch die alte Signatur. [hs/1/96]</note>
	</date>
	<date resp="#author-Creyssent" cert="false" evidence="internal" when="1771-12-24" n="date_alt1"/>
	[...] 
</correspAction>

<correspAction type="sent"> [...] <date resp="#editor-XY" evidence="external" cert="false" when="1759-06-23">
		<note type="evidence">Nach
			<bibl>Grotefend</bibl>.</note>
	</date>
	<note type="alternativeDate">
		<date calendar="roman">9.Cal.Quint.</date>
		<date when="1759-04-23">
			<note>Mit "Quint" könnte auch Mai gemeint sein, das
			ergäbe den 23.4. als Briefdatum.</note>
		</date>
	</note> [...]
</correspAction>

<correspAction type="sent" select="#date_pref"> [...] <date resp="#editor-XY" evidence="external" cert="false" when="1759-06-23" n="date_pref">
		<note type="evidence">Nach <bibl>Grotefend</bibl>. Mit
			"Quint" könnte auch Mai gemeint sein, das ergäbe den
			23.4. als Briefdatum.</note>
	</date>
	<date resp="#editor-XY" evidence="external" cert="false" when="1759-04-23" n="date_alt1">
		<note type="evidence">Wenn mit "Quint" Mai gemeint sein
			soll, ergibt sich der 23.4. als Briefdatum.</note>
	</date>
	<date calendar="roman" n="date_alt2">9.Cal.Quint.</date>
	[...] 
</correspAction>

<correspAction type="sent"> [...] <placeName key="#place_00603" resp="#editor-XY" cert="true" evidence="conjecture">Hannover
			<note>Ohne Ortsangabe im Original.</note>
	</placeName> [...] </correspAction>

<correspAction type="sent" select="#date_pref #place_00603"> […]
		<placeName key="#place_00603" resp="#editor-XY" evidence="conjecture">Hannover</placeName>
	<placeName evidence="internal"/> […] </correspAction>

<correspAction type="sent"> [...] <placeName key="#place_01130" source="#letter_13916" evidence="conjecture" resp="#editor-XY">Wien <note>Wien nach <rs key="#letter_13916">AvH.s Brief an S. Tissot [K] vom
				16.6.1763</rs></note>
	</placeName>
</correspAction>

<correspAction type="sent" select="#date_pref #place_01130"> […]
		<placeName key="#place_01130" source="#letter_13916" resp="#editor-XY" evidence="conjecture">Wien <note>Wien nach
				<rs key="#letter_13916">AvH.s Brief an S. Tissot [K]
				vom 16.6.1763</rs></note>
	</placeName>
	<placeName evidence="internal"/> […] </correspAction>

<correspAction type="received"> […] <persName ref="http://d-nb.info/gnd/118534262" resp="#editor-XY">Fontane, Theodor</persName> […] </correspAction>
<correspAction type="received"> […] <persName ref="http://d-nb.info/gnd/1023503034" resp="#editor-XY" evidence="external" cert="medium">Hochenburger, Anna von
			<note>Vgl. <bibl>Berbig, Theodor Fontane. Chronik, S.
				3078</bibl>.</note>
	</persName> […] </correspAction>

<correspAction type="received" select="#recipient_pref"> […]
		<persName n="recipient_pref" ref="http://d-nb.info/gnd/1023503034" resp="#editor-XY" evidence="conjecture" cert="medium">Hochenburger, Anna
		von</persName>
	<persName n="pers_alt1">unknown</persName> […] </correspAction>