Practice: File feeds over SFTP
Generated by Claude (Fable 5.1) by hand from the identity and integration ops spike, Scout learning center on 2026-10-14.
Main tool: atmoz/sftp with paramiko and gpg · atmoz/sftp builds a chrooted, key-only OpenSSH feed account from one line, paramiko lets a producer and a consumer be written in a few dozen lines with the exact rename semantics a real client has, and gpg is the OpenPGP tool customers and their security teams expect; together they reproduce the whole feed pattern on one machine, including the two failure modes the learning center's run found. · Alternatives: AsyncSSH runs an SFTP server in process for the no-Docker route; sftpgo is a full server with users, quotas and object-store backends when you need the server side for real; the sftp command with -b batch files tests a feed by hand; openpgpjs and gopenpgp are OpenPGP libraries for JavaScript and Go when gpg cannot be shelled out to; a managed endpoint (AWS Transfer Family, Azure Blob SFTP) is what a cloud customer will hand you.
Every step links into the tool's own documentation; links were checked on 2026-10-14 · version 1
Setup (one time)
1. Generate an ed25519 key pair for the feed account with ssh-keygen (no passphrase for the exercise; a passphrase or a secrets manager in production).
Key-only accounts are the baseline for a feed; the public half is all the server ever holds.
Explained in: OpenSSH: ssh-keygen(1) (man.openbsd.org), checked 2026-10-14
Done when: feed_ed25519 and feed_ed25519.pub exist and ssh-keygen -l prints the fingerprint.
2. Run atmoz/sftp in Docker bound to 127.0.0.1:12222 with the public key mounted into the account's keys folder and the user specification feed::1001::inbox,outbox,archive (no password). No-Docker alternative: an AsyncSSH in-process SFTP server in the same scratch environment.
The image builds the chrooted, key-only layout a customer's server owner would; its README documents the mounts.
Explained in: atmoz/sftp README (github.com), checked 2026-10-14
Done when: docker ps shows the container with 127.0.0.1:12222->22 and the image's log shows the user and folders created.
3. Create a scratch Python virtual environment with paramiko (and asyncssh for the no-Docker route); install gpg from your package manager if it is missing, or decide to do the encryption steps with a gpg on another machine.
paramiko is the client in the proof; gpg is the encryption tool the lesson teaches.
Explained in: Paramiko: SFTP client API (docs.paramiko.org), checked 2026-10-14
Done when: python -c 'import paramiko' exits 0 and gpg --version prints a version (or you have noted which steps you will read instead).
Steps
1. Connect with the key and list the root; then try a password login; then try to write a file at the root; then try to list /etc.
These four checks prove key-only, root-owned chroot and no shell escape; run them against every feed account a customer gives you.
Explained in: OpenSSH: sshd_config(5) (man.openbsd.org), checked 2026-10-14
Done when: The listing shows .ssh, inbox, outbox and archive; the password login is refused with an authentication failure; the root write fails with permission denied; /etc is not found.
2. Read the server's effective configuration (docker exec into the container and grep sshd_config) for Subsystem, ForceCommand and ChrootDirectory.
The three directives are the server side of the feed; recognising them lets you review a customer's configuration.
Explained in: OpenSSH: sshd_config(5) (man.openbsd.org), checked 2026-10-14
Done when: You can quote the three lines and explain what each one removes from the account's abilities.
3. As the producer, write a five-row CSV, upload it as inbox/orders-1.csv.part and rename it to inbox/orders-1.csv; list the inbox between the two operations from a second connection.
Drop-then-rename is the convention that makes partial files visible as partial; seeing it from the consumer's side once fixes the rule.
Explained in: OpenSSH: sftp(1) (man.openbsd.org), checked 2026-10-14
Done when: The second connection sees the .part name during the upload and the final name only after the rename.
4. Write a manifest (orders-1.manifest.json) listing the file name, byte count, SHA-256 of the uploaded bytes and the row count, and drop it with the same drop-then-rename after the data file.
The manifest is the batch's completeness signal and its integrity check; dropping it last is what makes multi-file batches safe.
Explained in: RFC 4180: Common Format and MIME Type for CSV Files (datatracker.ietf.org), checked 2026-10-14
Done when: The manifest's hash and size match what a fresh download of the data file computes.
5. Generate the consumer's OpenPGP key pair with gpg, export the public key, and as the producer import it and encrypt the CSV to it (orders-2.csv.gpg); add a plain_sha256 field (the SHA-256 of the plaintext) next to the wire hash in the manifest and drop batch 2. (gpg was not run by the learning center; the commands follow the handbook.)
Encryption to the consumer's public key is what customers mean by 'PGP'; the second hash exists because the ciphertext differs on every run.
Explained in: The GNU Privacy Handbook (www.gnupg.org), checked 2026-10-14
Done when: gpg --list-keys shows the consumer key, the encrypted file decrypts back to the identical CSV, and the manifest holds both hashes.
6. Write the consumer: list the inbox, ignore any .part file, for each manifest verify every listed file's size and wire hash, decrypt with gpg (or read plain files), verify the plaintext hash and row count, ingest the rows into a local table or file with an upsert on the order id, record the plaintext hash in a ledger, and archive both files with a rename.
This is the whole consumer; every later step breaks one assumption of it.
Explained in: Paramiko: SFTP client API (docs.paramiko.org), checked 2026-10-14
Done when: Both batches are ingested with the row counts the manifests state, the ledger holds two hashes, and the inbox holds nothing but partial files.
7. Drop batch 1 again (same content, freshly encrypted) and run the consumer.
The two findings of the proof in one step: the ciphertext hash is not an identity, and version-3 RENAME refuses an existing target.
Explained in: SSH File Transfer Protocol (draft-ietf-secsh-filexfer-02) (datatracker.ietf.org), checked 2026-10-14
Done when: The consumer reports the batch as already ingested by its plaintext hash and skips it, and the archive rename gets a unique name (a timestamp suffix) because the original name already exists there.
8. Upload half a file as inbox/orders-99.csv.gpg.part and never rename it; run the consumer twice.
A producer that died mid-upload is a normal Tuesday; the consumer's rule is to never touch a partial file.
Explained in: OpenSSH: sftp(1) (man.openbsd.org), checked 2026-10-14
Done when: The consumer lists the partial file as skipped both times and never opens it; your alerting note says after how long a stale .part file pages someone.
9. Read OpenSSH's PROTOCOL document on posix-rename@openssh.com and check which rename your client sends (paramiko: rename versus posix_rename); write down which one the producer and the consumer use and why.
Knowing which rename you issue is the difference between an atomic archive step and a silent overwrite.
Explained in: OpenSSH PROTOCOL: posix-rename@openssh.com and other extensions (github.com), checked 2026-10-14
Done when: Your note names the two operations and states that the producer never overwrites a final name.
10. Write the feed contract: account and key handling, folder layout, file naming, drop-then-rename, manifest fields, encryption key exchange and rotation, the CSV rules from RFC 4180 plus encoding, delimiter, header, dates, numbers and nulls, the schedule, the deadline alert, retention of the archive, and what happens on a re-sent batch.
The contract is the deliverable; the practice exists so every line in it has been seen to matter.
Explained in: Enterprise Integration Patterns: File Transfer (www.enterpriseintegrationpatterns.com), checked 2026-10-14
Done when: A two-page document a customer's integration owner could implement the producer from, with every rule above present.