Glossary
Data cleaning glossary
Short, exact definitions, each linked to a lesson or an exercise where you can practise it.
- Boolean parsing
- Turning answers like Y, yes, TRUE or 1 into true/false with a declared list, rejecting anything else.
- Cardinality
- How many rows on one side of a relationship match rows on the other: one-to-one, one-to-many or many-to-many.
- Character encoding
- How text is stored as bytes, such as UTF-8 or Windows-1252.
- Composite key
- A key made of several columns, such as (order_id, line_id).
- Controlled vocabulary
- An approved list of category values that every record must use.
- Data cleaning
- Turning raw records into a dataset you can trust, by applying stated rules and keeping evidence of every change.
- Data dictionary
- A document that defines every column: meaning, type, allowed values and who owns it.
- Data leakage
- Letting information from evaluation data influence how the data is prepared.
- Data profiling
- Measuring a table before cleaning it: row counts, missing values, distinct values, ranges and patterns per column.
- Deduplication
- Keeping one row per key with a deterministic survivor rule, and recording the rest.
- Delimiter and quoting
- The character that separates fields (comma, semicolon, tab) and the quotes that protect separators inside a value.
- Empty string
- Text that was recorded and is empty (""). Not the same as missing.
- Entity resolution
- Deciding whether two records describe the same real-world thing.
- Exact decimal
- A number type that stores base-10 digits exactly, used for money.
- Exception report
- The list of rows that failed a rule, each with the reason, kept next to the clean output.
- Idempotency
- Running a step twice gives the same result as running it once.
- Identifier
- A value that names a thing (a code, an id). Compared exactly, never as a number.
- Imputation
- Filling missing values with an estimate, flagged as filled.
- Join fan-out
- Rows multiplying in a join because a key matches more than one row on the other side.
- Leading zeros
- Zeros at the start of an identifier like 005642. They are part of the identifier.
- Lineage
- A record of where each row came from (file, system, batch).
- Locale
- The conventions a value was written in: 1,234.50 (en-US) versus 1.234,50 (de-DE), 03/04 as month/day or day/month.
- Missing value (null)
- A cell where nothing was recorded. Different from empty text and from zero.
- Null key
- A join key that is missing. It should never match another missing key.
- Nullable dtype (pandas)
- pandas types such as Int64, boolean and string that hold <NA> without changing the column type.
- Orphan row
- A row whose foreign key points to nothing in the referenced table.
- Outlier
- A value far from the rest. Possibly an error, possibly the most important row.
- Pivot
- Turning rows (entity, month, value) into one row per entity with a column per month.
- Quarantine
- Moving an invalid row to an exceptions table with a reason, instead of deleting it.
- Reconciliation
- Checking that clean + quarantined (+ filtered) rows equal the input rows.
- Referential integrity
- Every reference points at a row that exists, such as a payment's account.
- Rounding (half-even)
- A declared rule for dropping digits; half-even rounds 2.345 to 2.34 and 2.355 to 2.36.
- Row grain
- What one row represents: an order, an order line, a line version.
- Schema
- The declared columns and types of a table, including which are required and which optional.
- Schema drift
- A source file changing its columns or types between deliveries.
- Survivor rule
- The ordering that decides which duplicate row is kept.
- Timestamp offset
- The +01:00 or Z at the end of a timestamp that ties the clock time to a real instant.
- Train/test split
- Keeping evaluation data separate so nothing learned from it influences the model or the cleaning.
- Trim
- Remove a declared set of characters from both ends of a value.
- Type inference
- A tool guessing column types from the values it reads, for example turning 02139 into the number 2139.
- Unicode normalization (NFC)
- Rewriting text into one canonical form so visually identical strings compare equal.
- UTC
- Coordinated Universal Time, the reference clock used to compare instants from different offsets.
- Validation rule
- A testable statement a value must satisfy, with a reason name for failures.
- Wide and long format
- Wide: one column per measurement (Jan, Feb, Mar). Long: one row per measurement with a column naming it.
- Window function
- A SQL calculation over related rows that keeps every row, such as ROW_NUMBER() OVER (PARTITION BY … ORDER BY …).