Pythonic data handling: comprehensions, JSON and None

The Monday after Diwali, Meera walks into the tech room with three questions on a sticky note. “Which items sold the most last week? How much did each store make? And why do some orders have no delivery time on them? A customer called about that.”

Arjun is feeling confident after last week’s pre-order script. Priya exports Diwali week from the online shop and drops the file in the team folder: diwali_orders.json. Arjun opens it and stops smiling. These aren’t tidy (item, qty) tuples. Each order is a dictionary with a customer dictionary inside it, a list of item dictionaries inside that, and, here and there, the word null.

“Real data,” Priya says. “Most of the code you’ll write as an AI engineer looks like this: take something messy and nested, pull out the parts you need, and summarise them. Models return JSON. APIs return JSON. Your evaluation results will be lists of dictionaries. Let’s learn to handle it properly.”

The export file

To follow along, run this once. It creates the same diwali_orders.json file Arjun received: six orders from Crumb’s three Pune stores. Don’t worry about the with open(...) line yet; the files lesson explains it. All it does here is write the text to disk.

import json

orders = [
    {"order_id": "CR-2001", "store": "Baner",
     "customer": {"name": "Ananya", "allergies": ["nuts"]},
     "items": [{"name": "Eggless chocolate cake", "qty": 1, "price": 650},
               {"name": "Butter croissant", "qty": 4, "price": 85}],
     "delivery_slot": "18:00-19:00", "delivery_fee": 30},
    {"order_id": "CR-2002", "store": "FC Road",
     "customer": {"name": "Rohan"},
     "items": [{"name": "Diwali hamper", "qty": 2, "price": 1499}],
     "delivery_slot": None, "delivery_fee": 0},
    {"order_id": "CR-2003", "store": "Baner",
     "customer": None,
     "items": [{"name": "Butter croissant", "qty": 6, "price": 85},
               {"name": "Kaju katli", "qty": 1, "price": 650}],
     "delivery_slot": "10:00-11:00", "delivery_fee": 0},
    {"order_id": "CR-2004", "store": "Kothrud",
     "customer": {"name": "Kabir", "allergies": []},
     "items": [{"name": "Plum cake", "qty": 1, "price": 450},
               {"name": "Butter croissant", "qty": 2, "price": 85}],
     "delivery_slot": None, "delivery_fee": 30},
    {"order_id": "CR-2005", "store": "FC Road",
     "customer": {"name": "Diya", "allergies": ["gluten"]},
     "items": [{"name": "Kaju katli", "qty": 2, "price": 650}],
     "delivery_slot": "12:00-13:00", "delivery_fee": 0},
    {"order_id": "CR-2006", "store": "Kothrud",
     "customer": {"name": "Farah"},
     "items": [{"name": "Butter croissant", "qty": 3, "price": 85}],
     "delivery_slot": "08:00-09:00", "delivery_fee": 60},
]

with open("diwali_orders.json", "w") as f:
    json.dump(orders, f, indent=2)

print("Wrote", len(orders), "orders")

Wrote 6 orders

Look closely at the data, because it has the quirks real data has. Order CR-2003 has "customer": None: a guest checkout, with no account. Rohan’s customer record has no "allergies" key at all, while Kabir’s has an empty list. Two orders have no delivery slot. And three orders have a delivery fee of 0, because they qualified for free delivery.

Arjun’s first attempt

Arjun starts with what he knows from the last lesson: a loop.

import json

with open("diwali_orders.json") as f:
    orders = json.load(f)

for order in orders:
    name = order["customer"]["name"]
    print(order["order_id"], name)

CR-2001 Ananya
CR-2002 Rohan
TypeError: 'NoneType' object is not subscriptable

Two orders in, it crashes. Order CR-2003 has no customer, so order["customer"] is None, and you can’t look up ["name"] inside nothing. “Not subscriptable” is Python’s way of saying “you used square brackets on something that doesn’t support them”.

He patches it with an if, then moves on to revenue. He knows some orders might be missing a fee, so he adds a “sensible default” of the standard ₹60:

import json

with open("diwali_orders.json") as f:
    orders = json.load(f)

revenue = 0
for order in orders:
    for item in order["items"]:
        revenue = revenue + item["qty"] * item["price"]
    revenue = revenue + (order.get("delivery_fee") or 60)

print("Revenue: ₹", revenue)

Revenue: ₹ 7623

He sends Meera ₹7,623. The real answer is ₹7,443. He has invented ₹180 of delivery fees that nobody paid. We’ll see exactly why in a moment, because it’s one of the most common bugs in Python.

Priya reads his code. “Two problems. You trusted that every value would be there, and then you didn’t trust the values that were there. Let’s slow down and look at the shape of the data first.”

Boxes inside boxes: reading nested data

“Think about a Diwali hamper,” Priya says. “You open the hamper, and inside there’s a box of kaju katli. Open that, and there are the sweets. To get one sweet, you open three things, in order.”

Nested data works the same way. Each pair of square brackets opens one box:

import json

with open("diwali_orders.json") as f:
    orders = json.load(f)

first = orders[0]                       # open the list: the first order
print(first["customer"])                # open the order: the customer box
print(first["customer"]["name"])        # open the customer: the name
print(first["items"][1])                # the order's items, then position 1
print(first["items"][1]["qty"])         # ...and that item's quantity

{'name': 'Ananya', 'allergies': ['nuts']}
Ananya
{'name': 'Butter croissant', 'qty': 4, 'price': 85}
4

Read every access chain left to right: first["items"][1]["qty"] means “the items of the first order, then position 1 of that list, then its qty”. When a chain crashes, check each box one at a time: which one was missing, or None?

JSON: the shape everything travels in

The file on disk isn’t Python. It’s JSON (JavaScript Object Notation), a text format for sending structured data between programs. Crumb’s website sends orders as JSON, every AI API replies in JSON, and when you ask an LLM for structured answers, you’ll usually ask for JSON. It’s the shipping box: a standard shape that every system knows how to pack and unpack.

Python’s built-in json module converts between JSON text and Python objects:

import json

raw = '{"order_id": "CR-2002", "paid": true, "delivery_slot": null, "items": [{"name": "Diwali hamper", "qty": 2}]}'
print(type(raw))                 # still plain text

order = json.loads(raw)          # "load string": JSON text -> Python dict
print(type(order))
print(order["paid"], order["delivery_slot"])

order["delivery_slot"] = "17:00-18:00"
print(json.dumps(order))         # "dump string": Python dict -> JSON text

<class 'str'>
<class 'dict'>
True None
{"order_id": "CR-2002", "paid": true, "delivery_slot": "17:00-18:00", "items": [{"name": "Diwali hamper", "qty": 2}]}

json.loads and json.dumps work on strings. json.load and json.dump (no s) work directly on open files, which is what we used above.

Following order CR-2001 through the report

Before writing the report, Priya has Arjun trace one order by hand, the way we did with Ananya’s pre-order last lesson. Order CR-2001 arrives as JSON text in the file. Here’s every step of what the finished report will do to it:

Every step is one of four moves: look inside (steps 2, 5, 8, 9), transform each item (step 3), combine many into one (steps 4, 6, 7), or keep only some (step 8). The rest of this lesson gives you the clean Python tool for each move.

The conveyor belt: comprehensions

“Here’s step 3 the way you’d write it,” Priya says, and types:

items = [{"name": "Eggless chocolate cake", "qty": 1, "price": 650},
         {"name": "Butter croissant", "qty": 4, "price": 85}]

line_totals = []
for item in items:
    line_totals.append(item["qty"] * item["price"])
print(line_totals)

[650, 340]

“Four lines to say one thing. Picture a conveyor belt of cookies. Each cookie passes under a stamp that ices it, and maybe through a sieve that drops the broken ones. Python lets you write the belt in one line.”

items = [{"name": "Eggless chocolate cake", "qty": 1, "price": 650},
         {"name": "Butter croissant", "qty": 4, "price": 85}]

line_totals = [item["qty"] * item["price"] for item in items]
print(line_totals)

big_lines = [item["name"] for item in items if item["qty"] * item["price"] > 500]
print(big_lines)

[650, 340]
['Eggless chocolate cake']

That’s a list comprehension. Read it as a sentence: “give me item["qty"] * item["price"] for each item in items”, and in the second one, “…if the line is over ₹500”. The stamp comes first, then the belt, then the optional sieve.

The same pattern builds dictionaries and sets:

menu = {"Butter croissant": 85, "Plum cake": 450, "Kaju katli": 650, "Brownie": 90}

# Dict comprehension: a 10% post-Diwali sale on everything
sale = {name: round(price * 0.9) for name, price in menu.items()}
print(sale)

# Only the treats under ₹100
cheap = {name: price for name, price in menu.items() if price < 100}
print(cheap)

# Set comprehension: the unique first letters, no duplicates
print(sorted({name[0] for name in menu}))

{'Butter croissant': 76, 'Plum cake': 405, 'Kaju katli': 585, 'Brownie': 81}
{'Butter croissant': 85, 'Brownie': 90}
['B', 'K', 'P']

For step 4, you don’t need the list at all. Pass the belt straight into sum:

items = [{"name": "Eggless chocolate cake", "qty": 1, "price": 650},
         {"name": "Butter croissant", "qty": 4, "price": 85}]

subtotal = sum(item["qty"] * item["price"] for item in items)
print(subtotal)

990

With round brackets (or none, inside a function call), it’s a generator expression: it produces values one at a time without building a list in memory. On six orders that doesn’t matter. On six million log lines, it does.

Sorting, counting and the rest of the toolbox

Meera’s first question was “which items sold the most?” That needs combining (count every item across every order) and then sorting.

Counting with Counter

import json
from collections import Counter

with open("diwali_orders.json") as f:
    orders = json.load(f)

sold = Counter()
for order in orders:
    for item in order["items"]:
        sold[item["name"]] += item["qty"]     # += means "add to the current value"

print(sold.most_common(3))

[('Butter croissant', 15), ('Kaju katli', 3), ('Diwali hamper', 2)]

A Counter is a dictionary that starts every key at zero, so you can add to keys that don’t exist yet. most_common(3) sorts and picks the top three for you. Fifteen croissants: that’s Meera’s first answer.

Sorting by any field

sorted() gives back a new, sorted list. To sort dictionaries, you tell it what to compare with key=:

import json

with open("diwali_orders.json") as f:
    orders = json.load(f)

def order_value(order):
    return sum(i["qty"] * i["price"] for i in order["items"])

biggest = sorted(orders, key=order_value, reverse=True)
print([o["order_id"] for o in biggest[:3]])

by_store = sorted(orders, key=lambda o: o["store"])
print([o["store"] for o in by_store])

['CR-2002', 'CR-2005', 'CR-2003']
['Baner', 'Baner', 'FC Road', 'FC Road', 'Kothrud', 'Kothrud']

lambda o: o["store"] is a lambda: a tiny function with no name, written inline. It means exactly the same as def get_store(o): return o["store"]. Use a lambda for one-line keys; use a named function, like order_value, when the logic deserves a name.

One gotcha: text sorts by character code, and capital letters come before lowercase ones. sorted(["banana bread", "Croissant", "almond tart"]) puts "Croissant" first. Pass key=str.lower to sort the way a human expects.

list.sort() also exists. It sorts the list in place and returns None, so orders = orders.sort() destroys your data. Use sorted() unless you specifically want to change the original.

Filter, map, reduce

You’ll hear these three words in every data and AI codebase:

• Filtering keeps only the items that pass a test (the sieve).
• Mapping turns each item into something else (the stamp).
• Reducing combines many items into one value (the cookies into one box count).

Python has filter() and map() built in, and reduce() in the functools module. Comprehensions and built-ins like sum, min, max, any and all usually read better:

from functools import reduce

fees = [30, 0, 0, 30, 0, 60]

# The functional style
paid = list(filter(lambda fee: fee > 0, fees))
doubled = list(map(lambda fee: fee * 2, fees))
total = reduce(lambda running, fee: running + fee, fees, 0)
print(paid, doubled, total)

# The Pythonic style, same results
print([fee for fee in fees if fee > 0], [fee * 2 for fee in fees], sum(fees))

# Yes-or-no questions about a whole collection
print(any(fee == 0 for fee in fees))   # is at least one free?
print(all(fee <= 60 for fee in fees))  # is every fee at most ₹60?

[30, 30, 60] [60, 0, 0, 60, 0, 120] 120
[30, 30, 60] [60, 0, 0, 60, 0, 120] 120
True
True

reduce walks the list keeping a running value: it starts at 0, then 0 + 30, then 30 + 0, and so on. You’ll see it in other people’s code, but for sums, sum says the same thing more clearly.

Empty shelves: None, missing keys and defaults

Now, Arjun’s two bugs. Both are about absence.

None is Python’s way of saying “nothing here”. It isn’t zero, it isn’t an empty string, and it isn’t False. Priya calls it a shelf with a label that says “deliberately empty”. You’ll see it constantly: a guest checkout with no customer, an unrated support ticket, an LLM reply that contains no tool call, a search function that found nothing.

There are three different kinds of “missing” in the export, and they need different handling:

dict.get(key, default) returns the default when the key is missing. For nested data, you can chain .get() calls, using an empty dictionary as the fallback so the next .get() has something to work on:

order = {"order_id": "CR-2003", "customer": None}
guest = order.get("customer") or {}            # None becomes an empty dict
print(guest.get("name", "Guest"))
print(order.get("payment", {}).get("method", "unknown"))
print(order.get("delivery_slot"))              # missing key, no default: None

Guest
unknown
None

Always check for None with is None or is not None. It’s both correct and the style every linter expects.

The or trap

Here’s Arjun’s revenue bug. a or b gives back a if a is truthy, and b otherwise. Falsy values, the ones Python treats as False in an if, are None, False, 0, 0.0, "", [] and {}. Everything else is truthy.

So order.get("delivery_fee") or 60 doesn’t mean “60 if the fee is missing”. It means “60 if the fee is missing, or zero, or anything else falsy”. Three orders had a real, deliberate fee of 0. Each became ₹60. That’s the invented ₹180.

for fee in [30, 0, None]:
    wrong = fee or 60
    right = fee if fee is not None else 60
    print(f"fee={fee!r:5}  wrong={wrong:3}  right={right}")

fee=30     wrong= 30  right=30
fee=0      wrong= 60  right=0
fee=None   wrong= 60  right=60

({fee!r:5} shows the value as Python would write it, padded to 5 characters, so the columns line up.)

The rule of thumb: or is fine when an empty value really should count as missing, like a blank name becoming “Guest”. When 0, False or an empty list are valid answers, use an explicit is None check.

The finished report

With every tool in hand, Arjun rewrites the report. It answers all three of Meera’s questions, plus one she didn’t ask but should have.

import json
from collections import Counter

with open("diwali_orders.json") as f:
    orders = json.load(f)

# 1. Revenue per store: items plus the fee that was actually charged
revenue = {}
for order in orders:
    subtotal = sum(i["qty"] * i["price"] for i in order["items"])
    fee = order.get("delivery_fee")
    fee = fee if fee is not None else 60     # only a truly missing fee gets the default
    revenue[order["store"]] = revenue.get(order["store"], 0) + subtotal + fee

# 2. Best sellers
sold = Counter()
for order in orders:
    for item in order["items"]:
        sold[item["name"]] += item["qty"]

# 3. Orders with no delivery slot
no_slot = [o["order_id"] for o in orders if o.get("delivery_slot") is None]

# 4. Customers with allergies (guest checkouts have no customer)
allergies = {
    o["customer"]["name"]: o["customer"]["allergies"]
    for o in orders
    if o["customer"] is not None and o["customer"].get("allergies")
}

print("Revenue by store:", revenue)
print("Total revenue:", sum(revenue.values()))
print("Top sellers:", sold.most_common(2))
print("No delivery slot:", no_slot)
print("Allergy flags:", allergies)

Revenue by store: {'Baner': 2180, 'FC Road': 4298, 'Kothrud': 965}
Total revenue: 7443
Top sellers: [('Butter croissant', 15), ('Kaju katli', 3)]
No delivery slot: ['CR-2002', 'CR-2004']
Allergy flags: {'Ananya': ['nuts'], 'Diya': ['gluten']}

Check Baner against the hand trace: CR-2001 was ₹1,020, and CR-2003 is 6 × 85 + 650 = 1,160 with free delivery. Together, ₹2,180. The trace and the code agree, which is the whole point of tracing.

Look at the allergy filter: o["customer"].get("allergies") returns None for Rohan (no key) and [] for Kabir (empty list). Both are falsy, so neither is flagged. Here, treating “missing” and “empty” the same is what we want. That’s the difference between using truthiness on purpose and by accident.

Where it breaks

The report works on six orders. Here’s what experienced engineers watch for as it grows to sixty thousand:

• One unexpected None stops everything. A single guest checkout crashed Arjun’s first loop. Data from outside your program (exports, APIs, model outputs) will eventually contain every shape you didn’t plan for. In two lessons you’ll learn Pydantic, which checks the shape once, at the door, instead of with ifs scattered everywhere.
• Changing a list while looping over it. Removing orders from orders inside for o in orders: silently skips items, because the list shifts under the loop. Build a new list with a comprehension instead.
• Comprehensions nobody can read. Two fors, two ifs and a function call in one line is a puzzle, not a comprehension. If you’d need a comment to explain it, write a loop.
• Very large files. json.load reads the whole file into memory at once. For millions of records, teams use JSON Lines (one JSON object per line, read line by line) or a database. You’ll meet JSON Lines again when you prepare data for fine-tuning.
• Data that changes shape over time. If the website team renames delivery_fee to deliveryFee, .get("delivery_fee") quietly returns None forever, and the default kicks in for every order. A loud error would have been kinder. Defaults hide bugs as well as preventing crashes.

Arjun explains it to Meera

Arjun sends the corrected numbers: ₹7,443, not ₹7,623. Meera wants to know what happened.

“Each order in the export is like a hamper: boxes inside boxes,” he says. “I was opening the customer box without checking whether it existed. Guest checkouts don’t have one. Then I gave every order a ₹60 fee if its fee looked empty, but Python treats zero as empty too. So every free delivery got charged on paper.”

“And the missing delivery times?”

“Two orders, CR-2002 and CR-2004. The shop let people check out without picking a slot. That’s a website fix, not a data fix. And croissants won the week: fifteen sold, five times more than anything else.”

“Fifteen? In six orders?” Meera laughs. “We’ll need a bigger test set. But the logic is right?”

“The logic is right. I traced one order by hand and the code agrees.”

Try it yourself

Crumb’s support inbox exports tickets like this:

tickets = [
    {"id": 101, "category": "late_delivery", "rating": 2, "resolved": False, "hours_open": 30},
    {"id": 102, "category": "allergy", "rating": None, "resolved": False, "hours_open": 2},
    {"id": 103, "category": "refund", "rating": 0, "resolved": True, "hours_open": 0},
    {"id": 104, "category": "late_delivery", "rating": 1, "resolved": False, "hours_open": 52},
    {"id": 105, "category": "allergy", "rating": 5, "resolved": True, "hours_open": 0},
]

Write code that prints:

1. The number of tickets per category, using Counter.
2. The average rating, ignoring unrated tickets but keeping the zero.
3. The IDs of the two unresolved tickets that have been open longest.

Expected:

{'late_delivery': 2, 'allergy': 2, 'refund': 1}
Average rating: 2.0
Most urgent: [104, 101]

from collections import Counter

tickets = [
    {"id": 101, "category": "late_delivery", "rating": 2, "resolved": False, "hours_open": 30},
    {"id": 102, "category": "allergy", "rating": None, "resolved": False, "hours_open": 2},
    {"id": 103, "category": "refund", "rating": 0, "resolved": True, "hours_open": 0},
    {"id": 104, "category": "late_delivery", "rating": 1, "resolved": False, "hours_open": 52},
    {"id": 105, "category": "allergy", "rating": 5, "resolved": True, "hours_open": 0},
]

print(dict(Counter(t["category"] for t in tickets)))

ratings = [t["rating"] for t in tickets if t["rating"] is not None]
average = round(sum(ratings) / len(ratings), 2) if ratings else None
print("Average rating:", average)

open_tickets = [t for t in tickets if not t["resolved"]]
urgent = sorted(open_tickets, key=lambda t: t["hours_open"], reverse=True)[:2]
print("Most urgent:", [t["id"] for t in urgent])

{'late_delivery': 2, 'allergy': 2, 'refund': 1}
Average rating: 2.0
Most urgent: [104, 101]

Common mistakes

• == None instead of is None. It usually works, but is None is the correct check and the one linters expect.
• Calling json.loads on something that’s already a dict. It raises TypeError: the JSON object must be str.... Check with type(x) when unsure which side of the conversion you’re on.
• orders = orders.sort(). .sort() returns None. Use sorted(orders) for a new list.
• Forgetting the default in .get() on nested data. order.get("customer").get("name") still crashes when customer is None. Use (order.get("customer") or {}).get("name").
• Using or for numeric defaults. 0, 0.0 and False are real values. Use is None.

Recap

• Nested data is boxes inside boxes: read access chains left to right, and check each box when one fails.
• JSON is the text format data travels in; json.loads/json.dumps convert strings, json.load/json.dump work with files, and null becomes None.
• Comprehensions filter and transform in one readable line; generator expressions feed sum, any and all without building lists.
• sorted(items, key=...) sorts by any field, Counter counts, and filter, map and reduce are the three moves behind most data code.
• None, a missing key and an empty list are three different kinds of “missing”; or treats 0 as missing too, so use is None when zero is a real answer.

The next morning, the billing script that Arjun wrote last month gives a customer a ₹1,200 cake for free. The cause is a single typo in a dictionary key. “Dictionaries will hold any key you throw at them, spelled however you like,” Priya says. “It’s time to give our data a proper shape.” Next: classes, objects and dataclasses.

Go deeper

• Python tutorial: data structures, including list comprehensions and dictionaries.
• The json module in the Python standard library.