Data Cleaning Exercises

Module 3

Missingness

Null versus blank versus zero versus unknown; keep the reason and handle each on purpose.

  1. Missing is not empty (and neither is zero)

    Tell missing, empty text and zero apart, and convert between them only when a rule says so. · 8 min

  2. Fill labels, not measurements

    Decide when a missing value may be replaced (a reporting label for a category) and when it must stay missing (a measurement). · 7 min

  3. Sentinel values are missing values in disguise

    Recognise codes such as -999, 9999 or "N/A" that stand for missing data, and convert them explicitly before any statistic. · 8 min

  4. Keep the reason a value is missing

    Distinguish not collected, not applicable and refused, and keep that reason as data. · 6 min

Module 3 quiz

Practice questions: choose an answer and check it to see the explanation. Nothing here is timed or graded.

  1. 1. In SQL, what does NULL = NULL evaluate to?
  2. 2. A duration column has missing values. The report needs an average. What is the safest approach?
  3. 3. pandas: s = pd.Series([1, None, 3]); what is s.mean()?
  4. 4. Why keep a reason when a value is missing (not collected, not applicable, refused)?
  5. 5. SQL: SELECT AVG(x) over values 2, NULL, 4 returns?
  6. 6. A pipeline replaces every missing category with "Other" before reporting. What is the risk?