Data Cleaning Exercises

Exercise · intermediate · 25 min

Validate quantities and prices against ranges

An inventory feed has impossible quantities, zero prices and unknown countries. Apply the validation rules in order and report the first rule each row breaks.

The rules

These rules decide what a correct result is. Your answer is checked on this dataset and on hidden datasets that follow the same rules.

Input
  inventory(id, sku, quantity, unit_price, country) — text. id is unique; sku is always present.

Normalize
  country: remove ASCII whitespace at both ends, then uppercase.

Quarantine each invalid row with the FIRST failing reason, checked in this order:
  1. missing_quantity        quantity is missing
  2. bad_quantity            quantity is not a whole number: optional "-" then digits only
  3. quantity_out_of_range   quantity is less than 1 or greater than 500
  4. bad_price               unit_price is missing, or is not: optional "-", digits, optionally "." and 1–2 digits
  5. price_out_of_range      unit_price is zero or negative
  6. bad_country             country is not one of US, CA, MX (a missing country is also bad_country)

Outputs (row order does not matter)
  clean(id text, sku text, quantity int64, unit_price decimal(10,2), country text)
  exceptions(id text, reason text)

The data

inventory · 21 rows, first 12 shown

idskuquantityunit_pricecountry
invybqzSKU-62521760.28·CA
invf1geSKU-7738501580.05MX
invoi41SKU-000312-1.00USA
invereeSKU-519123572.57·CA
inv29z8SKU-000100.00UK
invvnhmSKU-4924434524.17US
invkmmsSKU-44967.5573.38CA
inveec2SKU-684710us
invezdpSKU-0936436956.62US
invgt2uSKU-92594401,200.00MX
inv4nh6SKU-6643721,200.00MX
invi6w4SKU-0002missingabcmissing

What you practise

  • validation rules
  • ranges
  • type conversion
  • allowed values
  • quarantine

Your answer is also checked on hidden datasets generated from the same rules.

Download this dataset (CSV) with its data dictionary

Sign in to bookmark this and keep private notes.