Text coding assigns a category or a value to each passage of a text. Delach et al. coded the listing documents for 459 species by hand, recording for each one whether climate change was named as a threat and whether any action to address it was planned. A coding scheme is the set of questions we ask of each passage and the answers we allow.
Laya applies a coding scheme that we write. Its answers tell us as much about our scheme as about the text, so the scheme is the thing we design and test.
Workflow¶
Coding a corpus with Laya takes six steps, and each one is a choice we make and should report.
Corpus. We download the full text of each listing rule from the Federal Register. A rule includes much more than the threat analysis: legal background, responses to public comments, and descriptions of critical habitat.
Unit. We split each rule into passages. Here a passage is a paragraph, found at blank lines. A sentence or a whole section would be other reasonable units.
Selection. We keep only the passages that mention a climate-related term (climate, sea level, flood, storm or drought). Passages that are never selected are never coded.
Coding scheme. We write the questions and the allowed answers.
Coding. Laya answers every question for every selected passage, one passage at a time.
Aggregation. We combine the passage codes into a code for each rule, for example the highest or the average probability, and compare the result with hand coding.
Question types¶
Laya answers three types of question, and each one fits a different kind of code.
A noul question has a yes or no answer, and Laya returns the probability of yes. We use it when a passage either has a property or does not.
A choice question picks one option from a list that we write, and Laya returns a probability for each option. We use it when every passage belongs in exactly one of several categories.
A score question places a passage on an ordered scale of levels that we write. We use it when the property comes in degrees.
Each question has an instruction, and each option or level has a short description. Those descriptions are the only definitions Laya has, so writing them is the main work of designing a coding scheme.
The request¶
Each request to Laya has two parts. The state is the passage being coded, and the questions are the coding scheme. Each question has a name that we choose, a type, an instruction, and for choice and score questions a set of criteria. A request with two of our questions looks like this:
state: "Sea level rise, more frequent tidal flooding ... (one paragraph)"
questions:
threat_noul:
type: noul
instructions: Does this paragraph say that climate change threatens the species?
threat_choice:
type: choice
instructions: Which threat does this paragraph describe as the most important?
criteria:
climate: sea level rise, flooding, drought, heat or storms
development: habitat loss from building, roads or land conversion
other: any other threat
none: the paragraph does not describe a threatLaya sees nothing except the state and the questions. It does not know the species, the rule, or which section of the rule the passage came from, unless we put that information in the state or in the instructions. Laya returns one answer for each question, under the same name.
A passage¶
Our example is the final rule that listed the Florida Keys mole skink as endangered in July 2026. This paragraph, from the body of the rule, summarizes the threats to the species:
Sea level rise, more frequent tidal flooding (increase of tides above the mean high tide), and increasing intensity of storm events (such as hurricanes) are the predominant threats to the Florida Keys mole skink and its habitat. Other threats to the Florida Keys mole skink include habitat loss and degradation that result from development and habitat disturbance. We also evaluated existing regulatory mechanisms and ongoing conservation measures. In the SSA report, we considered additional threats: overutilization due to recreational, educational, and scientific use; disease; oil spills; saltwater intrusion; and nonnative species. We concluded that, as indicated by the best available scientific and commercial information, these additional threats are currently having little to no impact on the Florida Keys mole skink, and thus their overall effect now and into the future is expected to be minimal. For full descriptions of all threats and how they impact the Florida Keys mole skink, please see the SSA report (Service 2024, pp. 25-46).
Notice that the paragraph never uses the words “climate change”.
One code, three ways¶
We code two ideas from Delach et al., a climate threat and an action to address it, using different question types. Each version records something different.
Climate threat as yes or no (noul). Does this paragraph say that climate change threatens the species? This is closest to the hand coding of Delach et al., who recorded whether a threat was named at all.
Climate threat as the main threat (choice). Which threat does this paragraph describe as the most important? The options are:
climate: sea level rise, flooding, drought, heat or storms
development: habitat loss from building, roads or land conversion
other: any other threat
none: the paragraph does not describe a threat
This version asks for a ranking among threats, which a yes or no question cannot record.
Climate threat by degree (score). How much weight does this paragraph give to climate-related threats? The levels, from lowest to highest, are: not mentioned; mentioned in passing; one of several important threats; the main threat.
Action as yes or no (noul). Does this paragraph describe an action to reduce a climate-related threat? This is the second code of Delach et al.
Action by degree (score). How specific are the actions this paragraph describes to reduce climate-related threats? The levels are: no actions; actions in general terms; specific actions; specific actions with a schedule or a responsible agency.
Laya’s answers for the example paragraph:
Source
import os
import re
import pathlib
import requests
import pandas as pd
from dotenv import load_dotenv
load_dotenv()
laya_url = os.environ["LAYA_URL"]
laya_auth = {"Authorization": f"Bearer {os.environ['LAYA_API_KEY']}"}
def decide(texts, scheme):
response = requests.post(f"{laya_url}/v1/systemone/batch", headers=laya_auth,
json={"states": texts, "questions": scheme}, timeout=300)
response.raise_for_status()
return [result["answers"] for result in response.json()["results"]]
rules = pathlib.Path("rules")
rules.mkdir(exist_ok=True)
def download(document_number):
path = rules / f"{document_number}.txt"
if not path.exists():
meta = requests.get(
f"https://www.federalregister.gov/api/v1/documents/{document_number}.json",
params={"fields[]": "raw_text_url"}, timeout=60).json()
path.write_text(requests.get(meta["raw_text_url"], timeout=60).text)
return path
def paragraphs(path):
text = re.sub(r"<[^>]+>", "", path.read_text())
paras = [re.sub(r"\s+", " ", p).strip() for p in re.split(r"\n\s*\n", text)]
return [p for p in paras if len(p) > 200]
skink = paragraphs(download("2026-14636"))
example = next(p for p in skink if p.startswith("Sea level rise, more frequent"))
scheme = {
"threat_noul": {
"type": "noul",
"instructions": "Does this paragraph say that climate change threatens the species?"},
"threat_choice": {
"type": "choice",
"instructions": "Which threat does this paragraph describe as the most important?",
"criteria": {
"climate": "sea level rise, flooding, drought, heat or storms",
"development": "habitat loss from building, roads or land conversion",
"other": "any other threat",
"none": "the paragraph does not describe a threat"}},
"threat_score": {
"type": "score",
"instructions": "How much weight does this paragraph give to climate-related threats?",
"criteria": ["Not mentioned.", "Mentioned in passing.",
"One of several important threats.", "The main threat."]},
"action_noul": {
"type": "noul",
"instructions": "Does this paragraph describe an action to reduce a climate-related threat?"},
"action_score": {
"type": "score",
"instructions": "How specific are the actions this paragraph describes "
"to reduce climate-related threats?",
"criteria": ["No actions.", "Actions in general terms.", "Specific actions.",
"Specific actions with a schedule or a responsible agency."]},
}
def summarize(answer):
if answer["type"] == "noul":
return round(answer["noul"], 2)
if answer["type"] == "choice":
return answer["choice"]
return round(answer["score"], 2)
answers = decide([example], scheme)[0]
pd.Series({name: summarize(answer) for name, answer in answers.items()})threat_noul 0.86
threat_choice development
threat_score 2.28
action_noul 0.08
action_score 1.6
dtype: objectThe versions do not agree. The noul question finds a climate threat with probability 0.86, but the choice question names development as the most important threat. The two action questions also point in different directions. Each answer follows from how we wrote its question and options.
The choice question returns a probability for every option, which shows how close the others came:
Source
pd.Series(answers["threat_choice"]["probabilities"])climate 0.2644
development 0.3958
other 0.2407
none 0.0991
dtype: float64A folder of rules¶
A coding scheme is written once and applied to every passage in a corpus. Here we add a second rule, which lists the Peñasco least chipmunk, and save both rules as text files in one folder. We then follow the workflow above: split each rule into paragraphs, select the paragraphs that mention a climate-related term (17 in each rule), code each one with the same scheme, and average the codes for each rule.
Source
download("2024-28338")
rows = []
for path in sorted(rules.glob("*.txt")):
paras = [p for p in paragraphs(path)
if re.search(r"climat|sea level|flood|storm|drought", p, re.I)]
for text, answer in zip(paras, decide(paras, scheme)):
rows.append({"rule": path.stem, "text": text} |
{name: summarize(a) for name, a in answer.items()})
coded = pd.DataFrame(rows)
codes = ["threat_noul", "threat_score", "action_noul", "action_score"]
coded.groupby("rule")[codes].mean().round(2)Questions¶
Which of these questions comes closest to what Delach et al. coded by hand, and what would you change to bring it closer?
Read the paragraphs that score highest on the yes or no threat question. Does each one meet your own definition of a threat finding? If not, is the cause the wording, the question type, or the passage you sent?
What changes if you send a sentence or a whole section in place of a paragraph?
The code for each step is folded above each result. The Laya documentation and the Jev documentation describe the full question format.