You extract ACTION SPANS from a caption describing a video. Captions are often several sentences long and describe events in order. For every action EVENT the caption states, return one entry holding the exact substring(s) of the caption that express it — a caption may mention the same event more than once — with exact caption spans, event grouping, and action type. ## Input format CAPTION: ## What counts as an action Any predicate asserting a visually meaningful physical ACTION, PROCESS, EVENT, or held bodily posture: - actions/events: runs, throws, opens, scratches, hits - postures and bodily states: sitting, standing, seated, lying, perched, squatting - processes: moving, floating, drifting, falling A word being grammatically a VERB does NOT by itself make it an action. The verb must semantically express an actual visible action/process/event in the caption. For example: "He tries to jump." -> "jump", NOT "tries" "He prepares to run." -> "run", NOT "prepares" "a falling ball" -> "falling" "a fallen ball" -> nothing for "fallen" `try`, `attempt`, `prepare`, `plan`, `intend`, `want`, and similar verbs that merely introduce an intended, attempted, or prepared action are not themselves action events for this tier. Extract the actual embedded action instead. However, when both verbs independently denote real physical actions, extract BOTH: "He runs to hit the desk." -> "runs", "hit" "She bends to pick up the bag." -> "bends", "pick up" Mere presence, visibility, static spatial arrangement, mental state, intention, or resulting state is NOT an action (see the exclusion list below). ## Camera/viewpoint exclusion The filming camera and camera viewpoint are out of scope. If the caption says that "the camera" or the viewpoint pans, zooms, moves, tracks, shakes, or cuts, do NOT return that predicate as an action; do not annotate camera/viewpoint motion. Return actions [] if that is the only apparent motion. This rule does not exclude a physically visible camera device when it is an ordinary scene object that is actually handled, moved, or dropped. Process EVERY sentence. Actions in subordinate clauses, coordinated clauses and reduced relative clauses count too. Every distinct event gets its own entry. ## POS guidance Use Penn Treebank POS tags as an important guide when identifying action heads and particles. Main action-head candidates: VB, VBD, VBG, VBN, VBP, VBZ Important neighboring tags: RP = particle · IN = preposition / subordinating conjunction · TO = infinitival "to" RB/RBR/RBS = adverb · MD = modal POS is a guide, not the final decision. A VB* token must still pass the semantic action test above. An RP token is strong evidence for a phrasal particle, but words such as up, down, in, out, on and off can have different grammatical functions in different contexts. Use the syntactic function and meaning in the current sentence: - genuine phrasal particle -> may be part of the action span - ordinary direction -> exclude - preposition heading a PP -> exclude If you reason about dependencies, a particle relation such as `prt` / `compound:prt` is strong evidence that the word is a genuine phrasal particle. ## One event, many mentions Captions often re-describe the SAME action of the same participants — a restatement, an elaboration, a continuation. One event = ONE entry, whose `mentions` array lists EVERY expression of it, in caption order (each as its own exact substring): "A man is pushing a cart. He pushes it into the garage." -> ONE action with mentions ["is pushing", "pushes"] (same person performs both mentions) "A dog chases a ball. The dog keeps chasing it until it stops." -> ONE action with mentions ["chases", "chasing"] ("keeps" stays out) Each mention is the verb group of that expression (aux included, per the span rule), or its complete light-verb + action-noun construction when that exception applies; control/aspectual verbs like "keeps" are never part of a mention. Return SEPARATE entries when the caption describes distinct events, even of the same kind by the same participant: - explicit repetition: "He waves. He waves again." -> two actions - a new occurrence after other events: "She jumps, lands, then jumps onto the box." -> two jump actions Most actions are mentioned once: then `mentions` has a single element. ## Span boundaries -- the core rule The action span is the VERB GROUP of the core lexical action: its auxiliary chain (the VB/VBD/VBG/VBN/VBP/VBZ forms of be, get and have that belong to the same verb group) plus the lexical action verb, plus any genuine PHRASAL-VERB PARTICLE. For LIGHT-VERB / ACTIVITY-NOUN constructions, also include the noun phrase that names the action, exercise, gesture, sport, or physical activity (see below). Modals (MD: can, could, must, will, would, may, might, shall, should) are NOT part of the span — they express possibility or obligation, not the action itself. Neither are control/aspectual/intention verbs (see below). The goal is NOT to extract the entire verb phrase. Exclude participant noun phrases from the action span; annotate only the action predicate itself. Also exclude modals, control verbs, ordinary directions, adverbs, ordinary prepositions, prepositional phrases, infinitival "to", and clause-linking function words. Auxiliary do/does/did is excluded, in all inflected forms: "He did fall." -> "fall"; "He does not move." -> "move". This boundary rule alone does not assert that a negated event occurred. Lexical do in "does a flip" is different: it supports an action noun and stays. INCLUDE in the span: - the verb group: VB* auxiliaries + the core lexical action verb "A fish is eating." -> "is eating" "Children are clapping." -> "are clapping" "Honey is fed using a syringe." -> "is fed" "The door gets pushed." -> "gets pushed" "The door is being pushed." -> "is being pushed" "The ball was kicked." -> "was kicked" but never a modal (MD is not a VB* tag): "She can swim." -> "swim" - a genuine phrasal particle that belongs to the lexical verb "Person opens dishwasher and pulls out rack." -> "opens", "pulls out" "Woman puts on black helmet." -> "puts on" "He picks up the ball." -> "picks up" "She turns on the light." -> "turns on" ### Light verbs and nouns naming physical activities Keep the WHOLE minimal contiguous construction when a verb such as do, perform, execute, make, give, take, or play combines with a noun phrase that NAMES the visible action, movement, exercise, gesture, maneuver, sport, or activity. The noun is the semantic action, not a separate concrete entity acted upon. Judge by meaning in context; this verb list and the examples are not exhaustive. Apply the rule to all inflections and progressive/perfect auxiliary chains. "A person performs jumps." -> "performs jumps", type light_verb "He does a flip." -> "does a flip", type light_verb "They do squats." -> "do squats", type light_verb "She did a somersault." -> "did a somersault", type light_verb "He is doing three push-ups." -> "is doing three push-ups", type light_verb "He does sports." -> "does sports", type light_verb "She has performed a backflip." -> "has performed a backflip", type light_verb "He executes a forward roll." -> "executes a forward roll", type light_verb "She makes a jump." -> "makes a jump", type light_verb "He gives a thumbs-up." -> "gives a thumbs-up", type light_verb "She takes a step." -> "takes a step", type light_verb "Two people play table tennis." -> "play table tennis", type nonvisual_object "A person is playing golf." -> "is playing golf", type nonvisual_object The light-verb rule is about grammatical function, not one fixed surface form. Apply it to all ordinary tense, person/number, and auxiliary-chain variants of the same construction. Preserve the exact contiguous words that appear in the caption: "He does sports." -> "does sports" "They did sports." -> "did sports" "She is doing sports." -> "is doing sports" "He has done sports." -> "has done sports" "They have been doing push-ups." -> "have been doing push-ups" Do not strip do/does/did/doing/done when the form is the lexical light verb followed by an action/activity noun. This is different from auxiliary do in "He did fall", where only "fall" is the action span. Include determiners, possessives, numbers and modifiers INSIDE the action-noun phrase, because they are part of the exact contiguous expression: "does a flip", "performs two backward somersaults". Exclude trailing location, equipment, manner and time adjuncts: "does a flip on the mat" -> "does a flip". An intervening adverb may stay only when required for one contiguous substring: "does slowly a flip" -> "does slowly a flip" (ungrammatical, low confidence). Do not rewrite "does a somersault" as "somersaults". Do not return a separate action for the same included noun, or label that noun separately. Concrete recipients and equipment stay outside the action span: "gives a thumbs-up to a man" -> action span "gives a thumbs-up". For coordinated action nouns under ONE shared verb, retain one contiguous phrase and do not create overlapping mentions: "does squats and lunges" -> "does squats and lunges". Independently expressed/repeated events stay separate. Contrast concrete entities; these nouns stay OUT of the span: "He hits the desk." -> "hits" "She gives a ball to a child." -> "gives" "He plays the piano." -> "plays" "She takes a cup." -> "takes" Mental/abstract nouns are not automatically physical actions: "makes a decision" or "has an idea" does not become a visible action just because it has a light verb. "performs uniform" has no clear physical activity noun in ordinary English: do not automatically label it a light_verb action. If the intended action is unclear, omit it; never silently replace it with "performs jumps" or another phrase. ### Lexicalized physical-event complements Some physical events are expressed by a lexical verb together with an abstract complement rather than a concrete entity. Keep the minimal complete predicate when the combination names the visible event. In particular, keep the full balance-loss expression, including an overt possessive when present: "The kid loses balance." -> "loses balance" "The kid loses its balance." -> "loses its balance" "The kid lost balance." -> "lost balance" "The kid is losing balance." -> "is losing balance" Do not reduce these to only "loses", "lost", or "is losing". The abstract complement "balance" is part of this lexicalized action expression, not a separately groundable entity object. If an event-noun phrase is pronominalized, do not invent the missing noun text. Only keep "does it" as a light-verb mention if "it" clearly refers to that same physical event (e.g. "does a flip, then does it again"); otherwise keep the verb or omit an unclear event. Each repeated occurrence remains a separate event. A phrasal particle is different from an ordinary directional word or preposition: "He picks up the ball." -> "picks up" "He picks the ball up." -> "picks" + separated particle "up" But: "He drives up the hill." -> "drives" "He walks to the left." -> "walks" "He looks at the dog." -> "looks" "He sits on the chair." -> "sits" The deciding question: does the neighboring word belong to the lexical phrasal verb itself, or does it merely express direction/location or introduce a prepositional phrase? Only the first case belongs in the action span. ### Directional modifiers Do NOT include directional modifiers merely because they distinguish where the motion goes: "The car is driving forward." -> "is driving" "He flips backwards." -> "flips" "She falls backwards." -> "falls" "The kid turns left." -> "turns" "A man glides forward." -> "glides" "A man falls down." -> "falls" Direction belongs outside the action span unless the word is genuinely a lexical phrasal particle. ### Prepositions and prepositional phrases Do NOT include an ordinary preposition in the action span, including a preposition selected by the verb. The preposition and the noun phrase it introduces stay outside the action span: "Man looks at a dog." -> "looks" "Woman waits for a train." -> "waits" "A person points at a laptop." -> "points" "Man walks in the park." -> "walks" "He is sitting on a chair." -> "is sitting" "A boy sits next to a woman." -> "sits" "He looks to the left." -> "looks" Do NOT keep a dangling preposition: "looks at" -> wrong · "sits on" -> wrong · "walks to" -> wrong Compare a true particle: "turns on the light" -> "turns on" with a preposition: "sits on the chair" -> "sits" ### Infinitival clauses Infinitival `to` is not part of an action span. If an intention/attempt/preparation verb introduces a genuine action, extract the embedded action only: "Man prepares to throw discus." -> "throw" "Man tries to jump." -> "jump" "Helicopter attempts to take off." -> "take off" Do NOT return "prepares to throw", "tries to jump", "attempts to take off". But if both predicates are independently real actions, return both separately: "He runs to hit the desk." -> "runs", "hit" "He steps forward to grab the bag." -> "steps", "grab" ### Participles VBG and VBN must be interpreted semantically, not mechanically. An ongoing visible process expressed by VBG counts: "a falling ball" -> "falling" "a moving car" -> "moving" A VBN that merely describes a completed/resulting state does not count: "a fallen ball" -> nothing for "fallen" "a broken chair" -> nothing for "broken" if it only describes the chair's state But an eventive passive DOES count, and keeps its auxiliary chain: "The ball was kicked." -> "was kicked" "The door is being pushed." -> "is being pushed" A bare participle with no auxiliary in the text stays bare: "a stick tied to a rope" -> "tied" The question is whether the caption asserts an action/event, rather than merely describing a resulting state. ## EXCLUDE entirely — these are NOT actions, return no entry for them: - STATIVE COPULAR RELATIONS: a copula plus a location, orientation, visibility or existence complement describes a configuration of the scene, not an action. "the two men are next to the black car." -> nothing "Two grey cars are in front." -> nothing "A woman in white is at a basin." -> nothing "A kid is on a balance bike." -> nothing "A white table is visible." -> nothing "a moving car is present." -> nothing for "is present" ("moving" is still an action) "A ping-pong table is in view." -> nothing "The bike is backward." / "is upside down." -> nothing "There are four pillows in front of it." -> nothing Contrast progressives and postures, which DO count with their auxiliary chain ("is walking", "is sitting", "standing"), and eventive passives, which are real events ("was thrown", "is fed"). The test: copula + VB* participle = a verb group = action; copula + location/adjective/existence = configuration = nothing. - intention, attempt and preparation predicates when they merely frame another action "He tries to jump." -> no action for "tries" "He prepares to throw." -> no action for "prepares" "She plans to run." -> no action for "plans" - result-state participles that do not assert a visible event "a fallen ball" -> no action for "fallen" "a broken chair" -> no action for "broken" when merely descriptive ## EXCLUDE from the span: - modals (MD) — but keep the VB* auxiliary chain "He is walking." -> "is walking" "She can swim." -> "swim" (modal out) "He must go now." -> "go" (modal out) - control / aspectual verbs that frame the core action "The dog keeps chasing it." -> "chasing" "He starts running." -> "running" "She continues to spin." -> "spin" This excludes aspectual start/stop only when it frames an embedded action: "She stops walking." -> "walking" Lexical start/stop acting on a concrete entity stays: "He starts the engine." -> "starts" "She stops the car." -> "stops" Standalone start/stop stays if it clearly describes a physical event. - subordinating conjunctions (while, as, before, after, when) "A parrot spreads its wings while standing on the edge of a bowl." -> "spreads", "standing" NOT "while standing" - temporal and sequencing adverbs (then, finally, suddenly, at the same time, again) "He finally lands." -> "lands" "They squeeze the balloon at the same time." -> "squeeze" - manner, degree and ordinary directional adverbs "He walks angrily." -> "walks" "The car moves forward." -> "moves" "He hits him badly." -> "hits" - reciprocal pronouns (each other, one another) "Two high-speed trains pass each other." -> "pass" - concrete visible entities; they stay outside the action span "A bald man in a suit plays the piano." -> "plays" "A cat scratches a rooster." -> "scratches" - nouns and pronouns naming independently groundable entity participants; EXCEPTION: nouns naming the action/activity in the light-verb rule stay in the action span, and are part of the action predicate "He hits the desk." -> "hits" "He looks at me." -> "looks" "He picks up the ball." -> "picks up" "Two people playing table tennis." -> "playing table tennis" "A person performs jumps." -> "performs jumps" - ordinary prepositions and their noun phrases "Man looks at a dog." -> "looks" "He is sitting on a chair." -> "is sitting" "Man walks in the park." -> "walks" "Two men squatting on the ground." -> "squatting" - subjects, adverbials, and nominal events introduced by "during" "A girl grabs a red ball during a color sorting activity." -> "grabs" The test for the ACTION SPAN is: 1. Find the lexical verb that expresses the actual visible action. 2. Reject grammatical verbs that only express intention, attempt, preparation, planning, modality, or auxiliary information. 3. Include a genuine lexical phrasal particle; for light-verb/activity-noun constructions, include the minimal full action-noun phrase as well. 4. Exclude ordinary directions, prepositions, PPs, adverbs and entity nouns/pronouns, except internal modifiers/function words needed by the light-verb construction. ## Separated particles Each selected character range must be ONE exact contiguous substring. A mention may use multiple ordered ranges when an excluded adverb interrupts an eligible verb group. When a participant phrase splits a phrasal verb, include only the verb in the mention's `text` and put a separated particle in `particle`: "The man picks a red bag up from the ground." -> mention {"text": "picks", "particle": {"text": "up", "occurrence": 1}} When the verb and particle are adjacent, keep them together in `text` ("picks up") and set particle's text to null. A V5 mention may reconstruct its `text` from multiple ordered ranges only to exclude an intervening adverb; each range itself must be contiguous. Do not use `particle` for an ordinary direction or preposition: "The man drives up the hill." -> "drives", NOT separated particle "up" "He looks at the dog." -> "looks", NOT separated particle "at" ## More worked examples "Cranes fly and walk, wings spread wide." -> "fly", "walk", "spread" "Two men squatting on the ground." -> "squatting" "He looks at a dog and then picks up a ball." -> "looks", "picks up" "A man falls down and a car hits him badly." -> "falls", "hits" "A falling ball passes a fallen branch." -> "falling", "passes" ("fallen" is NOT an action) ## Fields mentions every expression of this one action event, in caption order. Usually one element. Each mention has: text the exact substring, copied character for character from the caption occurrence 1 if this is the first occurrence of that exact substring in the caption, 2 for the second, and so on -- counted over the WHOLE caption, even for text inside other words. Repeated verbs and pronouns make this matter: in "He waves. He waves again." the second "waves" has occurrence 2. particle a separated phrasal-verb particle ({"text": null, "occurrence": 1} when there is none). See "Separated particles". type one of: simple, phrasal, prepositional, intention, light_verb, nonvisual_object, posture, passive, other confidence high, medium or low. Use low when the caption is ungrammatical or the action / particle / semantic judgement is genuinely unclear. ## Hard requirements - Every `text` must be reconstructable verbatim from its selected caption ranges. Do not normalise, lemmatise, correct spelling, expand contractions, or change capitalisation. - Every individual span range must be contiguous; V5 permits one mention to use multiple ordered ranges only when excluding an intervening adverb. Separated particles go in `particle`. - Action mentions must not overlap each other or any particle -- one stretch of the caption expresses one mention of one action. - Two mentions belong to the same entry ONLY if they describe the same single event; "again" / "once more" / a later repetition is a NEW entry. - Process every sentence; do not stop after the first. - If the caption states no action at all, return {"actions": []}. Do not invent one. - Captions are sometimes ungrammatical. Extract what is actually there and mark confidence low. Most importantly: - POS tags identify candidates; they do not determine actionhood by themselves. - Extract the actual semantic action, not every grammatical verb. - The action span is the VERB GROUP (VB* auxiliary chain + core action verb) + genuine PHRASAL PARTICLE; include the ACTION-NOUN PHRASE for light-verb or activity-noun constructions. Modals, auxiliary do and control verbs are out. - Do not include ordinary directions or prepositions. - Entity nouns/pronouns stay out; action nouns stay in light-verb constructions. - `does a flip` -> `does a flip` · `do squats` -> `do squats` - `performs jumps` -> `performs jumps` · `gives a thumbs-up` -> `gives a thumbs-up` - `plays golf` -> `plays golf` · `plays the piano` -> `plays` - `is walking` -> `is walking` · `was kicked` -> `was kicked` · `can swim` -> `swim` - `keeps chasing` -> `chasing` · `starts running` -> `running` - `look at it` -> `look` · `look to the left` -> `look` - `is sitting on the chair` -> `is sitting` - `pick up the ball` -> `pick up` · `turn on the light` -> `turn on` - `try to jump` -> `jump` · `prepare to throw` -> `throw` · `run to hit` -> `run`+`hit` - `falling` can be an action; result-state `fallen` is not. ## RDCap V5 ACTION AND SPAN CLARIFICATIONS — AUTHORITATIVE - FACE orientation: when `face`, `faces`, `faced`, or `facing` is the lexical verb explicitly describing a person's body/head orientation, annotate that predicate as `posture`. This V5 exception overrides the generic static-copular-orientation exclusion. Preserve its full auxiliary verb group, for example `is facing`; exclude ordinary direction words from the span (for example, `face-front` -> `face`), and do not annotate the noun `face` in `scratches his face`. Apply this only to an explicitly stated person-orientation predicate, not mere location, visibility, or object arrangement. Localize it only when sampled-frame evidence supports it; otherwise use `uncertain` with empty `segments`. - Generic light-verb phrases: omit an underspecified phrase when `do/doing` or `perform/performing` is followed only by a generic label such as `physical activity`, `hand actions`, `some actions`, or `an activity`. Do not annotate the light verb alone, the generic noun phrase, or inferred unspecified actions. Keep specific activity names such as `doing push-ups` and `performing a backflip`. - Completeness pass: inspect every sentence, clause, coordinated predicate, action noun, and repeated mention. Return every eligible action; exclude each non-action only for a stated rule. Do not stop at the most salient action. ## RDCap V5 ACTION-SPAN OVERRIDE — AUTHORITATIVE; OVERRIDES CONFLICTING V2 ADVERB EXAMPLES - Keep the full auxiliary verb group for an eligible predicate, including BE: `is walking` -> `is walking`; `was kicked` -> `was kicked`; `is being pushed` -> `is being pushed`; `is facing` -> `is facing`. Auxiliary `do/does/did` remains excluded as specified above, unless it is a lexical light verb naming a specific action. - Adverbs and other modifiers are not part of an action span. Keep the caption wording unchanged. If a modifier interrupts the auxiliary group and core predicate, represent the action with multiple ordered exact ranges: in caption `is quickly walking`, use mention text `is walking` and select `is` and `walking` in separate ranges; leave `quickly` unselected. For `is quickly doing three push-ups`, use mention text `is doing three push-ups` and select the auxiliary, light verb, and action-naming phrase in ordered ranges while leaving `quickly` unselected. Do not rewrite the caption to force a contiguous span. - A mention may have one or more ordered, non-overlapping, zero-based half-open character ranges. Each range must select an exact caption substring. Join the selected substrings with one space to obtain `mentions[].text`. Keep exact surface form, tense, spelling, and capitalization. - For a qualifying light-verb action, keep the full eligible auxiliary group plus the minimal action-naming phrase. Keep a specific activity name such as `doing three push-ups`; apply the generic-light-verb exclusion above. - `occurrence` is 1-based and counts ordered, non-overlapping matches of the selected span components in the entire caption, case-insensitively and with whole-word boundaries for every component. Unselected modifiers may occur between components; do not count substrings inside other words. - This override governs spans only. Keep the V2 actionhood, event grouping, completeness, genuine particle, and light-verb semantic distinctions unless an explicit V5 clarification above changes them. # Text Span Annotation Guidelines This document describes how the text side of the **bidirectional concept correspondence** dataset is annotated. The task is defined over an image–text pair: the goal is to recover all correspondences between visually referential text spans and instance-level image masks. The image side is a set of binary instance segmentation masks (one per visual entity). These guidelines cover the **text side only** — what counts as a valid text span and how spans are grouped by entity instance. --- ## Overview The text annotation for one image–text pair is a set of **groups**: ``` { G1: [[start, end), [start, end), ...], ← all text spans mentioning entity 1 G2: [[start, end), [start, end), ...], ← all text spans mentioning entity 2 ... } ``` Each group corresponds to **one visual entity instance** and collects every character span in the text that mentions that entity — including the first noun phrase, subsequent pronouns, and any repeated references. A group maps one-to-one with one instance mask on the image side. **Character spans** are `[start, end)` half-open intervals over the text string. Indices are **0-based Unicode code points**, equivalent to Python `str` indexing (`text[start:end]` extracts the span text). Not UTF-8 bytes. **Multiple spans per group** is the norm, not the exception. A caption like "A woman sits at a table. She reads a book." produces a Woman group with two spans: `"A woman"` and `"She"`. There is no restriction to a single contiguous span — all character regions that mention the same entity belong together. **Cross-group overlap** is allowed and expected. A shared type anchor like "Two dogs" appears in both Dog-A and Dog-B groups. A possessive like "his" overlaps with "his phone" across groups. The key invariant is **within a single group, no two spans may overlap**. --- ## 1. What to Annotate Annotate all **concrete, tangible, visually groundable** entities — entities that correspond to a segmentable region in the image. ### Annotate - **People and animals**: man, woman, child, dog, cat, bird, horse - **Objects**: car, phone, umbrella, ball, cup, book, surfboard - **Furniture and structures**: table, bench, chair, building, fence, bridge - **Tools and vehicles**: hammer, bicycle, truck, boat - **Natural features (countable)**: river, tree, mountain, hill, lake, rock, cloud — treated as **things** (see §3.3) - **Tangible body parts**: hand, face, leg, eye, wing, beak — annotated as **separate entities** with their own groups (see §3.6) - **Stuff classes**: sky, water, grass, road, snow, sand, ocean, pavement, floor, ceiling, wall, ground, field, fog, mud, ice — always a **single group** per stuff type (see §3.3) - **Named entities**: John, Mary, London; multi-word proper nouns with `of` ("University of Washington", "Statue of Liberty") as a single span (§2.8.4) - **Indefinite pronouns (existential)**: `someone`, `something`, `somebody` — refer to a specific (unknown) entity in the scene - **Reflexive / reciprocal pronouns**: `himself`, `herself`, `themselves`, `each other` — placed in the antecedent's group via coreference (§3.4) - **Demonstrative pronouns (standalone)**: `this`, `that`, `these`, `those` used pronominally ("these are nice", "that is a dog") — placed in the referent's group via §3.4 - **First / second person pronouns**: `I`, `me`, `we`, `us`, `you` — valid spans when they refer to a visible person in the scene. Skip generic / impersonal `you` ("you can see the dog" — no specific referent) ### Do NOT Annotate - **Meta-references to the image medium**: photograph, image, picture, frame, photo, shot, capture - **Abstract/spatial concepts**: background, foreground, scene, corner, side, center, area, edge, the left, the right, view, atmosphere, mood, setting - **Universal / negative indefinite pronouns**: `nobody`, `nothing`, `no one`, `everyone`, `everybody`, `everything` - **Negative-determiner NPs**: `no dog`, `no people`, `not a single hat` - **Verbs and function words**: running, is, are, there (existential) - **Pleonastic pronouns**: "it" in "it is raining" - **Punctuation** **Gerunds**: Annotate a `-ing` form only when it functions as a noun referring to a concrete entity ("the building" the noun). Do not annotate when it is a verb ("they are building a house"). **Never skip annotation.** Every text containing at least one concrete entity must be annotated. Vague plurals, generic references, and stuff classes all receive groups. --- ## 2. Span Boundaries Each span covers **the head noun phrase plus its premodifiers only**. This section defines exactly what goes inside a span and what does not. ### 2.1 Span Types | Type | Pattern | Examples | |------|---------|---------| | **Noun phrase** | (article \| quantifier \| count \| demonstrative \| possessive)? + attribute(s)? + head noun | "the dog", "a red car", "an umbrella", "Two dogs", "Several people", "another man", "this dog", "all the people", "a couple of dogs", "ice-cream", "John" | | **Pronoun** | Personal pronoun or standalone demonstrative as entity reference | "he", "she", "it", "they", "him", "her", "them", "that" (in "that is mine") | | **Selector** | Selector alone or with article — no spatial modifier, no PP, no participle | "one", "other", "another", "the other", "the third", "the first", "this one", "that one" | | **Possessive** | Possessive determiner or `'s` form — alone or combined with a head noun | "his", "his phone", "her", "her bag", "John's", "John's phone" | Premodifiers (article, quantifier, count, demonstrative, possessive, attributes) are all part of a **single span** with the head noun. "the red car" is one span `[start_of_the, end_of_car)`, not three separate spans. ### 2.2 Article Inclusion Articles (`the`, `a`, `an`) are **always included** in entity spans when they directly precede the noun phrase. | Text | Span | NOT this | |------|------|---------| | "the dog" | "the dog" | ~~"dog"~~ | | "a red car" | "a red car" | ~~"red car"~~ | | "an umbrella" | "an umbrella" | ~~"umbrella"~~ | | "the other" | "the other" | ~~"other"~~ | | "the third" | "the third" | ~~"third"~~ | | "a cup of coffee" (cup as entity) | "a cup of coffee" | ~~"cup of coffee"~~ | Demonstratives ("this", "that") are determiners, not articles — already included in v1 and continue to be. "this dog" is one span; "that one" is one span. Quantifier + article: "all the people" → one span "all the people"; "both the dogs" → one span "both the dogs". ### 2.3 Premodifier-Only Rule A span covers the head noun and its **premodifiers only**: ``` [ article | quantifier | count | demonstrative | possessive ]? [ attribute(s) ]? HEAD NOUN × nothing after the head noun ``` Everything **after** the head noun is excluded: PPs, postmodifying participles, postmodifying spatial words, relative clauses, and further appended attributes. | Text | Span | Why | |------|------|-----| | "a barking dog" | `a barking dog` | Premodifier participle — included | | "a dog barking" | `a dog` | Postmodifier participle — excluded | | "a tall man" | `a tall man` | Premodifier attribute — included | | "a man with a hat" | `a man` + `a hat` | Postmodifier PP — excluded; hat is its own entity | | "one in front" | `one` | Postmodifier spatial — excluded | | "the third on the left" | `the third` | Postmodifier PP — excluded | | "a man holding a phone" | `a man` + `a phone` | Postmodifier participle — excluded | | "the dog that I saw" | `the dog` | Relative clause — excluded | The only postmodifiers ever kept inside a span are the two compound-NP patterns in §2.8. Also exclude from all spans: - **Prepositions**: on, in, near, between, to, at - **Conjunctions**: and, but, or, while - **Standalone spatial regions**: "the left", "the right", "the background" - **Existential "there"**: as in "there are three dogs" - **Pleonastic "it"**: as in "it is raining" - **Punctuation and whitespace** ### 2.4 Within-Group No-Overlap Within a single group, no two spans may overlap or nest. When two spans would be generated where one subsumes the other, keep only the **longer** span. | Text | Wrong | Right | |------|-------|-------| | "this tree" | ["this tree", "tree"] | ["this tree"] | | "the dog" | ["the dog", "dog"] | ["the dog"] | | "his phone" | ["his phone", "phone"] | ["his phone"] | Cross-group overlap is always allowed — see §2.5 (possessives) and §2.8 (compound NPs). ### 2.5 Possessives (Cross-Group Overlap) Possessive determiners and possessive nouns create cross-group overlap: - **Possessive alone** ("his", "her", "its", "John's") → goes in the **possessor's** group - **Possessive + noun phrase** ("his phone", "John's phone") → goes in the **possessed entity's** group Example: - "his" [start_of_his, end_of_his) → in Person's group - "his phone" [start_of_his, end_of_phone) → in Phone's group - These spans overlap across groups — this is expected and correct ### 2.6 Compound NPs — Two Exceptions to §2.3 These are the **only** postmodifier exceptions. #### 2.6.1 Counter + Content — Fixed Single Span ` of ` → whole phrase is one span for the content entity: - "a pair of glasses" → `a pair of glasses` in Glasses group - "a stack of books" → `a stack of books` in Books group - "a herd of cows" → `a herd of cows` in Cows group - "all of them" / "all of the dogs" → single span (universal quantifier `all` covers the whole set — contrast with partitive selectors in §2.7) No choice here: counter phrases always stay attached. #### 2.6.2 Container + Content — Context-Dependent (Three Patterns) ` of ` where both are visually groundable ("a cup of coffee", "a bottle of wine") — choose one of three patterns: **Pattern A — Container only**: "She lifted **a cup** of coffee" → span `a cup` in Cup group only. **Pattern B — Content only**: "She sipped the **coffee**" → span `the coffee` in Coffee group. **Pattern C — Both entities, full phrase shared**: "A cup of coffee sits on the table" → span `A cup of coffee` appears in Cup group AND Coffee group (cross-group overlap). When in doubt and both are visually grounded, use Pattern C. Other PPs are never combined this way ("the man with a dog" → `the man` + `a dog` separately). #### 2.6.3 Named Entities with `of` — Single Span ` of ` proper nouns are a fixed single span: - "University of Washington" → `University of Washington` - "Statue of Liberty" → `Statue of Liberty` - "the Statue of Liberty" → includes the article per §2.2 ### 2.7 Partitive Constructions — Two Spans Required ` of ` produces **two separate spans**, never combined: | Text | Spans | Notes | |------|-------|-------| | "one of them" | `one`, `them` | `them` co-refers to its antecedent group | | "one of the dogs" | `one`, `the dogs` | | | "two of the dogs" | `two`, `the dogs` | Cardinals behave like selectors | | "the third of them" | `the third`, `them` | Ordinal selector | | "some of the cars" | `some`, `the cars` | | | "both of them" | `both`, `them` | | | "each of the children" | `each`, `the children` | | **Trigger forms:** cardinals (one, two, …, ten), ordinal selectors (first, second, …, tenth, other, another, former, latter), quantifier selectors (some, many, few, several, both, each, none, most, any, either, neither). **Exception — `all`**: "all of them" → **single span** (§2.6.1). `all` covers the whole set; it is not a subset selector. Both sides must be annotated. If only one side is present, the other is a missing span. --- ## 3. Grouping Rules A group = one visual entity instance = one image mask on the image side. All text spans that mention the same instance go in the same group. ### 3.1 Same-Type Instance Splitting When a text mentions a specific count of same-type entities AND provides instance markers, create **separate groups** sharing a type anchor: - **Trigger**: explicit count + instance selectors (one/other, first/second/third, this/that) - **Shared type anchor**: The count phrase (e.g., "Two dogs") appears in **all** groups for that entity type - **Instance markers**: Each selector goes only in its respective group **Count without selectors** ("Two dogs are playing") → single group, one span "Two dogs". Count alone is not enough to split. Example: "Two dogs, one is big and the other is small" - G1 (Dog A): spans `"Two dogs"` and `"one"` - G2 (Dog B): spans `"Two dogs"` and `"the other"` #### 3.1.1 Distributive Adjective Coordination When coordinated adjectives modify a shared head noun and the modifiers are mutually exclusive (one entity cannot have both properties), the construction refers to **two distinct entities** with the head noun elided on one side: | Caption | Span 1 | Span 2 | |---------|--------|--------| | "the rightmost and leftmost bus" | `rightmost bus` (Bus A) | `leftmost` (Bus B, elided head) | | "the wooden and metal chair" | `wooden chair` (Chair A) | `metal` (Chair B, elided head) | Contrast: "red and white fringe" → 1 Fringe (co-occurring properties on the same entity). Annotate as a single span. ### 3.2 Vague/Generic Plurals No explicit count or distinguishing markers → **single group**: - "Several people" → 1 group - "Dogs playing in the yard" → 1 group - "many cars", "few children", "all the people" → 1 group each ### 3.3 Stuff vs. Things **Stuff classes** (amorphous, uncountable): sky, water, grass, road, snow, sand, pavement, floor, ceiling, wall, ocean, ground, field, fog, mud, ice. Always a single group: - "the sky" → 1 group; "the snow on the ground" → 2 groups (Snow, Ground) **Things, not stuff** (reclassified from v1): river, mountain, tree, hill, lake, cloud, rock. Follow §3.1 / §3.2: - "a river" → 1 thing group - "Two mountains, one on the left, one on the right" → 2 thing groups Forest is treated as stuff. Individual trees are things. ### 3.4 Coreference — Multiple Spans Per Group All mentions of the same entity go in the same group: - **Repeated references**: "A dog ran by. The dog was fast." → `"A dog"` + `"The dog"` both in Dog's group - **Pronouns**: "A woman sat down. She smiled." → `"A woman"` + `"She"` in Woman's group - **Synonyms**: only when they unambiguously co-refer in this text Pleonastic pronouns ("it is raining") have no referent and are not annotated. **Anaphoric `one` / `ones`** is NOT coreference. It introduces a NEW INSTANCE: - "the red ball, not the blue **one**" → `the red ball` in Ball-A group; `the blue one` in Ball-B group (different instances) Personal pronouns (`it` / `they` / `them`) DO refer to the same instance and merge via coreference. ### 3.5 Compound Possessives Apply §2.5 to any possessor, including non-human entities: - "the dog's" → Dog group; "the dog's tail" → Tail group ### 3.6 Body Parts Body parts are separate entities with their own groups: - "A man raises his hand" → Man group + Hand group (separate) - "his hand" → full phrase in Hand's group; "his" alone in Man's group **Paired body parts** follow thing rules (§3.1 / §3.2): - "his eyes" / "her hands" (bare plural, no markers) → **1 group** each - "his two hands, one holding a cup and the other a phone" → 2 Hand groups --- ## 4. Examples All spans below are verified: `text[start:end]` equals the listed text. ### Example 1: Multiple Spans Per Group (Coreference) **Text**: `"A woman sits at a table. She reads a book."` | Span text | Positions | |-----------|-----------| | A woman | [0, 7) | | a table | [16, 23) | | She | [25, 28) | | a book | [35, 41) | **Groups:** - G1 (Woman): `[[0,7], [25,28]]` — "A woman" + "She" - G2 (Table): `[[16,23]]` - G3 (Book): `[[35,41]]` **Key points:** The Woman group has **two spans** at different positions because "She" co-refers to "A woman". This multi-span group maps to a single woman mask on the image side. --- ### Example 2: Shared Type Anchor (Cross-Group Overlap) **Text**: `"Two dogs are playing near a river while one chases a ball and the other watches."` | Span text | Positions | |-----------|-----------| | Two dogs | [0, 8) | | a river | [26, 33) | | one | [40, 43) | | a ball | [51, 57) | | the other | [62, 71) | **Groups:** - G1 (Dog A): `[[0,8], [40,43]]` — "Two dogs" + "one" - G2 (Dog B): `[[0,8], [62,71]]` — "Two dogs" + "the other" - G3 (River): `[[26,33]]` - G4 (Ball): `[[51,57]]` **Key points:** "Two dogs" at [0,8) appears in BOTH G1 and G2 — this is cross-group overlap. Each dog group maps to its own instance mask. --- ### Example 3: Three Instances with Selectors **Text**: `"Three people stand on a sidewalk; one holds an umbrella, another checks a phone, and the third looks away."` | Span text | Positions | |-----------|-----------| | Three people | [0, 12) | | a sidewalk | [22, 32) | | one | [34, 37) | | an umbrella | [44, 55) | | another | [57, 64) | | a phone | [72, 79) | | the third | [85, 94) | **Groups:** - G1 (Person A): `[[0,12], [34,37]]` - G2 (Person B): `[[0,12], [57,64]]` - G3 (Person C): `[[0,12], [85,94]]` - G4 (Sidewalk): `[[22,32]]` - G5 (Umbrella): `[[44,55]]` - G6 (Phone): `[[72,79]]` --- ### Example 4: Selector + Noun Combined **Text**: `"A man walks down a street while another man waits at a bench."` | Span text | Positions | |-----------|-----------| | A man | [0, 5) | | a street | [17, 25) | | another man | [32, 43) | | a bench | [53, 60) | **Groups:** - G1 (Man A): `[[0,5]]` - G2 (Man B): `[[32,43]]` - G3 (Street): `[[17,25]]` - G4 (Bench): `[[53,60]]` **Key points:** "another man" is one combined span. No shared type anchor — each man is referred to individually. --- ### Example 5: Attributes in Spans **Text**: `"A red car is parked next to a blue car near a tall building."` | Span text | Positions | |-----------|-----------| | A red car | [0, 9) | | a blue car | [28, 38) | | a tall building | [44, 59) | **Groups:** - G1 (Car A): `[[0,9]]` - G2 (Car B): `[[28,38]]` - G3 (Building): `[[44,59]]` --- ### Example 6: Vague Plural **Text**: `"Several people are walking near some trees and a fountain."` | Span text | Positions | |-----------|-----------| | Several people | [0, 14) | | some trees | [32, 42) | | a fountain | [47, 57) | **Groups:** - G1 (People): `[[0,14]]` — single group, vague plural - G2 (Trees): `[[32,42]]` — single group, vague plural thing - G3 (Fountain): `[[47,57]]` --- ### Example 7: Stuff Classes **Text**: `"A dog runs across a sandy beach under a cloudy sky near the ocean."` | Span text | Positions | |-----------|-----------| | A dog | [0, 5) | | a sandy beach | [18, 31) | | a cloudy sky | [37, 50) | | the ocean | [56, 65) | **Groups:** - G1 (Dog): `[[0,5]]` - G2 (Beach): `[[18,31]]` — stuff class, 1 group - G3 (Sky): `[[37,50]]` — stuff class, 1 group - G4 (Ocean): `[[56,65]]` — stuff class, 1 group --- ### Example 8: Possessives and Body Parts **Text**: `"A man raises his hand to wave his phone."` | Span text | Positions | |-----------|-----------| | A man | [0, 5) | | his (1st) | [13, 16) | | his hand | [13, 21) | | his (2nd) | [30, 33) | | his phone | [30, 39) | **Groups:** - G1 (Man): `[[0,5], [13,16], [30,33]]` — "A man" + both "his" - G2 (Hand): `[[13,21]]` - G3 (Phone): `[[30,39]]` **Key points:** "his" at [13,16) is in Man's group AND overlaps with "his hand" at [13,21) in Hand's group (cross-group overlap). The Man group has three spans. --- ### Example 9: Compound Noun Phrases **Text**: `"A cup of coffee sits on a wooden shelf next to a pair of glasses."` **Pattern C (both salient):** | Span text | Positions | |-----------|-----------| | A cup of coffee | [0, 15) | | a wooden shelf | [24, 38) | | a pair of glasses | [47, 64) | - G1 (Cup): `[[0,15]]` - G2 (Coffee): `[[0,15]]` — same span, different group (cross-group overlap) - G3 (Shelf): `[[24,38]]` - G4 (Glasses): `[[47,64]]` — counter+content fixed span (§2.6.1) --- ### Example 10: Selectors with Excluded Postmodifiers **Text**: `"Two cyclists ride past a café; one in front is faster than one behind."` | Span text | Positions | |-----------|-----------| | Two cyclists | [0, 12) | | a café | [23, 29) | | one (1st) | [31, 34) | | one (2nd) | [59, 62) | **Groups:** - G1 (Cyclist A): `[[0,12], [31,34]]` - G2 (Cyclist B): `[[0,12], [59,62]]` - G3 (Café): `[[23,29]]` **Key points:** "in front" and "behind" are excluded (postmodifier spatial words, §2.3). The two `"one"` spans at [31,34) and [59,62) are distinguishable by position. "é" in "café" is 1 Unicode code point — use code-point indices. --- ### Example 11: Paired Body Parts **Text**: `"A woman covers her eyes with her hands."` | Span text | Positions | |-----------|-----------| | A woman | [0, 7) | | her (1st) | [15, 18) | | her eyes | [15, 23) | | her (2nd) | [29, 32) | | her hands | [29, 38) | **Groups:** - G1 (Woman): `[[0,7], [15,18], [29,32]]` - G2 (Eyes): `[[15,23]]` — 1 group (paired body part, vague plural) - G3 (Hands): `[[29,38]]` — 1 group (paired body part, vague plural) --- ### Example 12: Possessive Noun (`'s`) **Text**: `"John's dog chases Mary's cat across the lawn."` | Span text | Positions | |-----------|-----------| | John's | [0, 6) | | John's dog | [0, 10) | | Mary's | [18, 24) | | Mary's cat | [18, 28) | | the lawn | [36, 44) | **Groups:** - G1 (John): `[[0,6]]` - G2 (Dog): `[[0,10]]` - G3 (Mary): `[[18,24]]` - G4 (Cat): `[[18,28]]` - G5 (Lawn): `[[36,44]]` --- ## 5. Quick Reference | Pattern | Rule | Span | |---------|------|------| | "the dog" | Article included | "the dog" | | "a red car" | Article + attributes | "a red car" | | "an umbrella" | Article included | "an umbrella" | | "the other" | Article + selector | "the other" | | "the third" | Article + selector | "the third" | | "another man" | Selector + noun = one span | "another man" | | "one" (standalone) | Selector alone | "one" | | "one in front" | Postmodifier excluded (§2.3) | "one" | | "one behind" | Postmodifier excluded (§2.3) | "one" | | "this one" | Demonstrative + selector = one span | "this one" | | "this dog" | Demonstrative + noun = one span | "this dog" | | "that" (entity ref) | Demonstrative pronoun | "that" | | "a cup of coffee" | Container + content (§2.6.2) | A/B/C per context | | "a pair of glasses" | Counter + content (§2.6.1) — fixed | "a pair of glasses" | | "a stack of books" | Counter + content (§2.6.1) — fixed | "a stack of books" | | "all of them" | Universal quantifier — single span (§2.6.1) | "all of them" | | "one of them" | Partitive — TWO spans (§2.7) | "one" + "them" | | "one of the dogs" | Partitive — TWO spans (§2.7) | "one" + "the dogs" | | "two of the dogs" | Partitive — TWO spans (§2.7) | "two" + "the dogs" | | "some of the cars" | Partitive — TWO spans (§2.7) | "some" + "the cars" | | "both of them" / "each of them" | Partitive — TWO spans (§2.7) | quantifier + "them" | | "someone" / "something" | Indefinite pronoun — annotate | the pronoun | | "nobody" / "everyone" / "nothing" | Universal/negative — NOT annotated | — | | "no dog" / "not a single hat" | Negative-determiner NP — NOT annotated | — | | "himself" / "each other" | Reflexive/reciprocal — in antecedent's group | the pronoun | | "the blue one" (anaphoric) | New instance — SEPARATE group from antecedent | "the blue one" | | "the rightmost and leftmost bus" | Distributive ADJ (§3.1.1) — TWO spans | "rightmost bus" + "leftmost" | | "these" / "those" (standalone) | Demonstrative pronoun | the demonstrative | | "I" / "me" / "you" (referential) | 1st/2nd person — valid when referring to visible person | the pronoun | | "you" (generic) | Generic — NOT annotated | — | | "University of Washington" | Named entity with `of` — single span (§2.6.3) | "University of Washington" | | "a barking dog" | Premodifier participle — included | "a barking dog" | | "a dog barking" | Postmodifier participle — excluded | "a dog" | | "a man with a hat" | PP excluded; hat is its own entity | "a man" + "a hat" | | "the dog that I saw" | Relative clause excluded | "the dog" | | "Two dogs" | Count + noun = one span | "Two dogs" | | "2 dogs" | Digit count behaves the same | "2 dogs" | | "Several people" | Quantifier + noun | "Several people" | | "all the people" | Quantifier + article + noun | "all the people" | | "a couple of dogs" | Multi-word quantifier | "a couple of dogs" | | "his" | Possessive → possessor's group | in Person group | | "his phone" | Possessive + noun → possessed entity's group | in Phone group | | "John's phone" | Possessive noun + noun → possessed entity's group | in Phone group | | "she" / "he" / "it" | Pronoun → merged into referent's group | same group | | "Two dogs" only (no markers) | Count alone → single group | 1 group | | "Two dogs" + selectors | Count + markers → split | separate groups, shared anchor | | "sky" / "water" | Stuff class → 1 group | always 1 group | | "river" / "mountain" / "tree" | Things (reclassified v2) — follow §3.1/§3.2 | per instance count | | "his eyes" / "her hands" | Paired body part → 1 group (vague plural) | 1 group per type | | "ice-cream" / "T-shirt" | Hyphenated entity = one span | whole token | | "the building" (noun) | Gerund-as-noun → annotated | "the building" | | "they are building" (verb) | Gerund-as-verb → not annotated | — | | "photograph" / "image" | Meta-reference — not annotated | — | | "background" / "the left" | Abstract/spatial — not annotated | — | | "it is raining" ("it") | Pleonastic — not annotated | — | | "café" length | "é" = 1 code point | use code-point indices | ## VideoLoom task input and mask handling Apply the RDCap V5 action/span rules above to the fixed VideoLoom caption. Apply the TEXT_ANNOTATION.md rules above for entity selection, span boundaries, grouping, and coreference. Preserve the caption exactly; do not generate or rewrite it. The input consists of a video duration, the fixed caption, clean RGB frames sampled at nominal 1 FPS, their actual timestamps, and the corresponding binary mask image for each sampled frame when available. Each mask is a separate binary image paired with the clean RGB frame at the same timestamp; it is not embedded or composited into the RGB frame or video. Provide the clean RGB frame and its mask as separate image inputs. Do not use an overlay preview as a substitute for either input. Inspect the masks to identify the entity instance they mark and link the corresponding text entity group. Use clean RGB frames, not the mask's shape or movement, as evidence for action visibility and timing. The available VideoLoom masklet tracks the selected person only; it is not a complete set of instance masks for every entity in the caption. A text entity without its own mask may be an object, body part, or scene entity; a null source-mask link does not mean that another person performed an action. Annotate every text-groundable entity required by TEXT_ANNOTATION.md. Link a group to a mask only when the supplied mask visibly identifies that entity; leave the source-mask link null for groups without a supplied mask. Do not infer or invent missing masks. Annotate an action only when its subject is identified by the supplied mask, and apply the RDCap V5 action/span rules to those mask-linked actions. Do not annotate an action whose subject has no corresponding supplied mask, even when the caption states that action. Do not transfer an unmasked subject's action to the selected masked person. If the action subject cannot be confidently linked to the supplied mask, omit that action rather than guessing. The original LoomData caption record supplies a source-video time interval [SOURCE_START_S,SOURCE_END_S). Pass this original interval to the model along with the immutable caption. It defines the clip boundaries, not action ground truth. The clip-relative duration is SOURCE_END_S - SOURCE_START_S. Give each sampled RGB/mask pair both its actual clip-relative timestamp and its corresponding source-video timestamp. For action temporal support, use the sampled clean-frame timestamps and return half-open intervals in clip-relative seconds [start_s,end_s). Treat them as sparse support intervals, not exact continuous-video onset/offset boundaries. Use only frames that visually support the action. If the supported action reaches the clip's right boundary, end the interval exactly at the original caption clip end (local end = clip duration; absolute end = SOURCE_END_S); do not extend it by adding one sampling interval after the last frame or beyond the original endpoint. Do not force an interval to reach the boundary when the final frames do not support continuation. Return one JSON object with `caption`, `entity_groups`, `actions`, and `notes`. Keep the caption unchanged. `entity_groups` must contain every text-groundable entity grouped by instance according to TEXT_ANNOTATION.md; include exact caption span offsets as zero-based, half-open character ranges and a source-mask link or null. `actions` must contain every eligible RDCap V5 action with its required exact mention spans, type, subject/object references, confidence, temporal status and intervals, and concise clean-frame evidence. Add `mask_relation` only to say whether an action subject matches the supplied selected-person mask. `notes` should report relevant evidence or missing per-entity masks. Do not include COIN step indices or other COIN-only fields. ## VideoLoom input fields Original LoomData source-video time interval in seconds: [{SOURCE_START_S}, {SOURCE_END_S}) Clip-relative duration in seconds: {VIDEO_DURATION_S} Caption (immutable; verbatim source caption): {CAPTION} Sampled frames: for every sample, provide the clean RGB frame and its separate paired binary mask when available, with both the actual clip-relative timestamp and source-video timestamp. The nominal sampling interval is 1 second. The source-video timestamp equals SOURCE_START_S plus the clip-relative timestamp. Span reminders already governed by RDCap V5: when `face` is the lexical verb for a person's orientation, include it as `posture` and preserve BE (`is facing`). If an adverb interrupts an eligible span, omit the adverb but preserve ordered exact ranges (for example, `is quickly facing` -> `is facing`, selecting `is` and `facing` separately). Preserve qualifying light-verb action phrases such as `do a flip`, `does a flip`, `did a flip`, and `is doing a flip`; when an adverb interrupts the eligible phrase, omit the adverb with separate exact ranges. Do not retain auxiliary `do/does/did` when it is auxiliary rather than a light verb.