Episodes

A weekly log of what I am learning, newest first. Each one ends with what I am learning next.

Season 1: Understanding S3

  1. Season 1, Episode 5

    Versioning and lifecycle in S3

    Part 5 of 6: How to bring a file back after a delete with versioning, and move old files to the archive safely with lifecycle rules.

    Every pipeline overwrites and deletes files: a rerun writes the same key, a lookback window rewrites the last three days, a teammate deletes the wrong file. Without versioning, each of those is gone for good. With versioning on, S3 keeps every write as its own version and turns a delete into a marker on top, so the file can be brought back. The kept versions bill in full until a lifecycle rule deletes them, and the same rules move files nobody opens down to cheaper storage, where they have to be restored before anyone can read them.

    Next time: Querying the lake.

  2. Season 1, Episode 4

    S3 keys and the lock for a data engineer

    Part 4 of 6: What happens when two jobs write the same file at once, and how to make S3 refuse the second.

    Every piece of a pipeline promises to deliver at least once: the scheduler, the queue, the retry. At least once means sometimes twice, and the place the two runs finally collide is the write to S3. S3 has no lock. Two writers to one key, and the last one wins, silently. It does have a 412, and since 2024 the two request headers that turn a plain write into “only if nobody has”, which is enough to build the lock yourself.

    Next time: How to bring a file back after a delete with versioning, and move old files to the archive safely with lifecycle rules.

  3. Season 1, Episode 3

    S3 networking for a data engineer

    Part 3 of 6: five routes from a VPC to a bucket.

    In AWS, there are services that do not live inside a VPC. S3 is one of them. This part walks through how S3 communicates with the services inside a VPC, across its different subnets, with five scenarios designed for a data engineer.

    Next time: What happens when two jobs write the same file at once, and how to make S3 refuse the second.

  4. Season 1, Episode 2

    S3 permissions for a data engineer

    Part 2 of 6: roles, policies, and the two locks on your data

    In part 1, a file that moves from one place to another in production is moved by a worker, and AWS calls that worker compute. To do its job the worker needs permission from S3, and it gets that by wearing a role. On every request S3 reads the role and the bucket’s own rules, and a deny on either side wins. The data is encrypted in transit and at rest.

    Next time: Networking.

  5. Season 1, Episode 1

    What is S3 for a data engineer?

    Part 1 of 6: the bucket, the key, the object, and the workers that fill it in production.

    S3 is a dictionary the size of a planet: buckets hold objects by key. In production, nobody uploads a file by hand. Something runs your code and does the writing, on a schedule, every night. AWS calls that something compute, four kinds matter to a data engineer, and they make three moves. Part 1 is the workers. Part 2 is permissions. Part 3 is networking, and the bill.

    Next time: Permissions.