Data Cleaning Exercises

Module 13

Combining files

Schema union, required and optional fields, source lineage and rejected files.

  1. Append files under a declared schema

    Stack monthly files under required and optional columns, reject files that break the schema, and keep each row's source file. · 8 min

  2. Lineage for appended files

    Record which delivery every row came from so problems can be traced and files reprocessed. · 7 min

  3. Required and optional columns

    Append files whose columns differ by declaring which columns are required and which optional. · 7 min

  4. Duplicate deliveries

    Detect a file delivered twice before its rows are appended. · 6 min

Module 13 quiz

Practice questions: choose an answer and check it to see the explanation. Nothing here is timed or graded.

  1. 1. Why add a source_file column when appending monthly files?
  2. 2. One monthly file is missing the required amount column. What should happen?
  3. 3. pandas.concat of a file with columns [id, amount] and one with [id, amount, channel]: what happens to channel for rows from the first file?
  4. 4. Required versus optional columns: why declare both?
  5. 5. Files are appended by position, not by column name. One file has amount and id swapped. What happens?
  6. 6. The same monthly file is delivered twice. How do you avoid double-counting?