Exercise · intermediate · 15 min
Append monthly files under a declared schema
Three files share most columns. Stack the ones that match the schema, record which file each row came from, and reject a file that's missing a required column.
The rules
These rules decide what a correct result is. Your answer is checked on this dataset and on hidden datasets that follow the same rules.
Inputs (three files, all text) jan(id, amount) feb(id, amount, channel) bad(id, currency) Rules Schema: required columns id and amount; optional column channel. No other columns are allowed. A file missing a required column, or containing a column outside the schema, is rejected as a whole. Keep every row of accepted files. A missing optional column gives a missing channel. source_file is the name of the file the row came from (jan, feb, …). amount is always a whole number in accepted files. Output (row order does not matter) clean(id text, amount int64, channel text, source_file text)
The data
jan · 6 rows
| id | amount |
|---|---|
| iz6d | 8 |
| rna5 | 262 |
| s38t | 410 |
| pk5w | 345 |
| r8id | 426 |
| azjp | 425 |
feb · 4 rows
| id | amount | channel |
|---|---|---|
| irsl | 91 | web |
| byjg | 143 | store |
| fate | 28 | missing |
| ujtb | 207 | store |
bad · 3 rows
| id | currency |
|---|---|
| o8wv | EUR |
| vmuv | EUR |
| yt74 | EUR |
What you practise
- combining files
- schemas
- lineage
Your answer is also checked on hidden datasets generated from the same rules.
Sign in to bookmark this and keep private notes.