Eugene Yan

Building a Strong Baseline Recommender in PyTorch, on a Laptop

Item-item matrix factorization, run pair-by-pair in PyTorch on a 16GB laptop, produces a strong recommender baseline without materializing the full sparse matrix in memory — and the main lessons concern what AUC-ROC hides rather than what it reveals.

The data is Julian McAuley's Amazon dataset: 418k electronics products with 4M co-purchase/co-view pairs, and 1.9M book products with 26M pairs. With no user IDs, the user-item matrix becomes item-item. Each pair gets a weighted score (bought together = 1.2, also bought = 1.0, also viewed = 0.5). Negative samples are generated by a shuffle-and-slice trick — stride through a shuffled product array rather than calling `random` millions of times — about 100x faster. Split is 2:1 train/validation.

The PyTorch model looks up two product embeddings, computes their dot product, passes through a sigmoid, and minimizes BCE or MSE loss with L2 regularization. Training runs 5 cosine-annealing epochs (lr 0.01 decaying to near zero, then reset). Each reset collapses AUC-ROC back toward 0.5 before it recovers, suggesting one epoch is already near-optimal.

Binary labels yield AUC-ROC 0.808; continuous labels (preserving the bought-together/also-bought/also-viewed gradient) reach 0.923. Both suffer a "cliff of death" on precision-recall: at threshold ~0.5, precision drops from ~1.0 to 0.5 nearly instantly. Adding per-product bias terms lowers AUC-ROC on paper (to 0.795 and 0.832) but flattens the cliff, making the model safer to deploy. The AUC-ROC drop is an artifact — without bias, false positives cluster just above 0.5 in a way the ROC curve obscures.

On books, the approach scales super-linearly (22 hours, 5 epochs) and fails entirely — AUC-ROC never leaves 0.5. Graph and NLP approaches tackle this in the follow-up.

Beating the Baseline Recommender with Graph & NLP in Pytorch

Product-to-product recommendations improve from AUC-ROC ~0.80 to above 0.97 — roughly 21% relative — by converting co-purchase pairs into a graph, running random walks to generate sequences, and training word2vec embeddings on those sequences.

The pipeline: build a weighted product graph using networkx, generate random-walk sequences, train skipgram embeddings. The key hurdle is scale. The electronics graph has 420k nodes at 99.99% sparsity; a dense NumPy adjacency matrix exhausts memory. The fix is a sparse adjacency matrix converted to a row-normalized transition matrix, cached as a dictionary for O(1) lookup. The off-the-shelf Node2Vec library fails even on a 64GB instance because it traverses networkx directly; the sparse approach is orders of magnitude faster.

Four implementations are benchmarked. Gensim word2vec on walk sequences hits AUC-ROC 0.9082 overall (0.9735 on seen products) in 2.58 minutes — the new baseline. A custom PyTorch skipgram adds subsampling of frequent products and negative sampling (5 negatives, unigram¾ distribution), reaching 0.9554 overall / 0.9855 seen-only and beating the AUC of 0.9327 reported by Alibaba on a similar electronics dataset. One epoch is sufficient. Extending word2vec with side-information embeddings (brand, category, price) collapses AUC to ~0.45; metadata sparsity (39% of products have any metadata; brand 51% empty) is the likely cause, and the model fails to downweight uninformative embeddings automatically.

A diagnostic separates data format from model choice: feeding walk sequences into the unchanged matrix-factorization model jumps its AUC from 0.7951 to 0.9320, because a window-size-5 walk provides 5× more training signal per product. On a 2M-product books dataset, gensim word2vec reaches 0.9701 in 16 minutes while matrix factorization stalls at ~0.50 after 22 hours.

Sequence-format training data is the dominant lever. Skipgram adds further gains; gensim suffices unless cold-start initialization for unseen products is needed.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

Improving Recommendation Systems & Search in the Age of LLMs

LLMs are reshaping recommendation and search systems across four areas, each with production results.

On architecture, the core shift is replacing hash-based item IDs with content-derived representations. YouTube's Semantic IDs compress video content via RQ-VAE (8 quantization levels, 2048-entry codebook), outperforming raw embeddings and hash IDs in cold-start. Kuaishou's M3CSR clusters visual, text, and audio features into ~1,000 trainable K-means IDs, gaining 3.4% clicks and 3.6% cold-start coverage in A/B tests. Huawei's FLIP aligns ID models with LLMs via masked tabular/language co-training; joint cross-modal reconstruction is the critical design choice. Google's CALRec finetunes PaLM-2 in two stages—category-agnostic then category-specific—with textual predictions matched to catalog via BM25.

On data generation, Bing used GPT-4 to create metadata for 2M webpages, distilled into Mistral-7B; a MiniLM cross-encoder ensembled with LightGBM cut clickbait 31% and duplicate content 76%. Indeed finetuned GPT-3.5 on 200 GPT-4-reviewed labels (GPT-4 quality at 25% cost) to produce eBadMatch (AUC 0.86), cutting bad job-match emails 17.7% and lifting application rates 4.1%. Spotify's LLM-generated synthetic queries raised exploratory-intent searches 9% and max query length 30%.

On training paradigms, sequential recommender performance follows a power law: a 75.5M-parameter model needs half the data a 98.3K model needs for equal loss. Pinterest's two-stage contrastive pretraining solves the one-epoch overfitting problem, yielding +2.2% engagement. DLLM2Rec distills a Llama2-7B recommender into lightweight students, averaging 47.97% recall improvement at 1.6 seconds versus 3-6 hours. Alibaba's MLoRA attaches domain-specific LoRAs to a frozen backbone, gaining +1.49% CTR across 10 domains.

On unified architectures, LinkedIn's 360Brew (150B Mixtral-based MoE) handles 30+ ranking tasks with prompt engineering replacing feature engineering, matching specialized models once training data is tripled. Netflix's UniCoRn serves search and recommendations from one contextual ranker, adding 10% lift. Zalando's platform—composable candidate generation, multi-task ranker, steerable policy layer—delivered +15% engagement and +2.2% revenue across four A/B tests.

Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs

Standard recommender systems predict what users will click next but can't be steered by natural language. LLMs have world knowledge and can discuss products, but are unaware of your catalog and suffer from popularity bias. Semantic IDs bridge the gap: instead of opaque hash IDs, items get hierarchical token sequences generated by a Residual Quantized VAE (RQ-VAE). Similar items share common token prefixes, forming a tree structure the LLM can natively reason over.

The pipeline uses 66k Amazon Video Games products embedded with Qwen3-Embedding-0.6B, then encoded into 4-token semantic IDs by an RQ-VAE with three 256-code quantization levels (89% unique IDs; a sequential fourth token resolves collisions). Cleaning descriptions with Gemini 2.5 Flash Lite — halving average length from 1,038 to 538 characters — improved codebook utilization and uniqueness. Commitment weight β=0.5 outperformed the β=0.25 used in the original Semantic ID papers. A SASRec trained on semantic IDs trails the item-ID baseline (Hit@10: 0.20 vs. 0.28) due to the harder generative task of predicting four tokens per item, but gains cold-start generalization via shared prefixes — a capability the baseline entirely lacks.

Qwen3-8B is then fine-tuned in two phases: first, 1,027 new tokens are added to the vocabulary and only the embedding layers are trained (1.23B parameters, 1,000 steps); then full fine-tuning runs on all 8.3B parameters over 4.2 million conversational examples covering ID-to-title mapping, next-item prediction, category relationships, and multi-hop reasoning.

The result is a model that generates items as token sequences without any retrieval system or intent router. It follows platform constraints ("Xbox games similar to Zelda" → Halo 4, Fallout: New Vegas), explains choices, and shifts mid-conversation to creative tasks (naming a bundle "Xbox Racing Legends: Speed & Style Pack"). Precision is lower than a specialized recsys, but the unified interface collapses search, recommendation, and chat into a single model.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

An LLM-as-Judge Won't Save The Product—Fixing Your Process Will

Product evals fail not because teams lack better tools or a smarter LLM-as-judge, but because they skip the underlying process — the scientific method applied to AI systems.

The cycle starts with observation: examining inputs, outputs, and user interactions to find where the system fails. Those failure modes drive annotation — labeling a balanced dataset of passes and failures (ideally 50/50) that becomes the foundation for targeted evals. From there, teams hypothesize why specific failures occur: retrieval not returning relevant context, models struggling with conflicting instructions, and so on. Experiments then test those hypotheses — rewriting prompts, updating retrieval, switching models — each with a defined outcome and a baseline to compare against. Measuring results is the hardest step: unlike casual vibe checks, it requires quantifying whether accuracy actually increased, whether defects dropped, whether the new version wins pairwise comparisons. If the experiment succeeds, integrate; if not, refine the hypothesis and try again. This iterative loop is how evals become a data flywheel.

Eval-driven development (EDD) operationalizes this cycle the way test-driven development works in software engineering. Before building a feature, teams define success criteria via evals, run a baseline benchmark, then evaluate every subsequent change — prompt tweaks, retrieval updates, model swaps — against that baseline. ML teams have worked this way for decades using validation and test sets; EDD brings the same discipline to LLM product development.

LLM-as-judge scales monitoring but doesn't replace human oversight. Teams must still periodically sample and annotate outputs, collect implicit feedback through product interactions, and use explicit user feedback to recalibrate automated evaluators against human judgment. Organizational discipline — maintaining the annotation and feedback loop continuously — determines whether automated evaluators stay aligned.

Buying another evaluation tool solves nothing if the process underneath is absent.

Evaluating Long-Context Question & Answer Systems

Long-context Q&A evaluation fails when you conflate two orthogonal dimensions: faithfulness (the answer relies only on the source document) and helpfulness (it is relevant, comprehensive, and concise). A faithful answer can be useless — "Clause 4.2 addresses missed payments" — and a helpful-sounding answer can be hallucinated. Models must also know when to say "I don't know"; failure modes split into false positives (hallucinated answers) and false negatives (incorrect refusals when information is present).

Good eval datasets need question diversity: fact recall, definitions, summarization, multi-hop inference, and "no-info" questions the document cannot answer. Evidence position should vary to expose attention failures. The NarrativeQA and QASPER construction method — generate questions from summaries or abstracts, answer from full text — prevents shallow extraction and forces genuine comprehension.

N-gram metrics like BLEU and ROUGE correlate poorly with human judgment; L-Eval demonstrated this directly. LLM-as-Judge is the better approach. For faithfulness, decompose answers into atomic claims, verify each against the document, and compute the proportion supported — this catches partial hallucinations and distinguishes retrieval from generation failures (as in SummaC, QAFactEval, RefChecker). For helpfulness, pairwise comparisons outperform absolute ratings; GPT-4 judgments correlated with human preferences in NovelQA at Cohen's Kappa ~89%. Human annotation remains necessary to calibrate LLM evaluators and measure their recall and precision.

Key findings across six benchmarks: NarrativeQA (46,765 pairs from novels and movie scripts) and NovelQA (2,305 questions on 89 novels, exceeding 200K tokens) cover narrative comprehension — NovelQA found accuracy drops when evidence exceeds 100K tokens. QASPER (5,049 questions on 1,585 NLP papers) reveals models answering correctly while misidentifying supporting passages. HELMET confirmed synthetic tasks like Needle-in-a-Haystack correlate weakly with real performance. Loong (multi-document, up to 250K tokens) found RAG degraded overall scores — it helped spotlight retrieval but hurt comparison, clustering, and chain-of-reasoning tasks where evidence is dispersed.

Product Evals in Three Simple Steps

Building product evals comes down to three steps — label data, align LLM evaluators, run a harness — and the payoff is speed: four weeks of upfront investment lets a team run hundreds of experiments without waiting on human review.

Label data first. Apply binary pass/fail or win/lose labels to sampled inputs and outputs. Likert scales fail because annotators can't calibrate "3" vs "4" consistently, and stakeholders who request granular scores always end up asking for a threshold anyway. Aim for 50–100 fail cases out of 200+ total. The best source is smaller, less capable models — they produce organic failures reflecting production. Synthetic defects from strong models are out-of-distribution and train evaluators on the wrong failure modes.

Align one evaluator per dimension. Split labeled data 75/25 into development and test sets. Build separate evaluators for faithfulness, relevance, conciseness, etc. — not one "God Evaluator." Multi-dimension prompts never calibrate cleanly and make it impossible to isolate which criterion failed. Combine individual evaluators with simple heuristics (pass only if all pass). For win/lose, run each pair twice with order swapped to cancel position bias; a flip between runs means call it a tie. Measure with precision, recall on fails, and Cohen's Kappa (0.4–0.6 is substantial; 0.7+ excellent). Human inter-rater Kappa often falls to 0.2–0.3, and annotators miss ~50% of defects from fatigue — beating those numbers is the real bar.

Run the harness with every config change. Wire evaluators in parallel to the experiment pipeline so changing a model, prompt template, or retrieval config triggers automatic evaluation. At 200 samples and 3% observed defects the 95% CI is 0.6–5.4% — too wide to confirm a 5% release requirement; 400 samples tightens the upper bound to 4.7%. Margin of error falls with the square root of sample size, so returns diminish quickly.

Simpler Experimentation with Jupyter, Papermill, and MLflow

Duplicating Jupyter notebooks for each experiment variation violates DRY, and tracking results across scattered CSVs and artifact directories is slow and error-prone. A three-tool stack — Jupyter, Papermill, and MLflow — eliminates both problems.

The workflow starts with a single base notebook. In the demo, it loads stock index data from Yahoo Finance, engineers moving-average features, runs five ML models (logistic regression, decision trees, and others), and evaluates them on AUC, recall, precision, and F1. One end-to-end pass takes 3.5 seconds.

Papermill then takes this single notebook and executes it repeatedly with different parameters — in the demo, five indices: S&P 500, Gold, SSE, Hang Seng, Nikkei. The only setup required is tagging a cell with the `parameters` label in Jupyter's cell toolbar, then writing a runner notebook that loops through values and calls `pm.execute_notebook(input_path='basic.ipynb', parameters={'INDEX': index}, output_path='basic_{}.ipynb')`. Each run produces its own self-contained output notebook (`basic_SNP.ipynb`, `basic_GOLD.ipynb`, etc.) with code, visualizations, and metrics inline.

That leaves the problem of comparing results across all runs. Five indices times five models produces 20 metrics, 10 graphs (ROC and precision-recall curves per model), and 5 model binaries. MLflow handles consolidation: wrapping each training block in `with mlflow.start_run()` and calling `mlflow.log_param()`, `mlflow.log_metric()`, and `mlflow.log_artifact()` pushes everything to a local server at `127.0.0.1:5000`. The UI lets you sort by any metric (e.g., AUC descending), filter by parameter (e.g., show only logistic regression runs), and browse each run's artifacts alongside its metrics in one view. The server can also be hosted centrally for team use.

The conclusion from the demo: logistic regression outperforms more complex models on this stock-prediction task, a finding that was easy to surface precisely because all runs were comparable in the same dashboard rather than spread across notebooks and directories.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

Building AI Reading Club: Features & Behind the Scenes

Sparked by a Karpathy–Collison Twitter thread about wanting AI that could answer questions about a highlighted passage mid-read, Eugene Yan built AI Reading Club (aireadingclub.com) — a working prototype that puts an AI companion named Dewey inside a browser-based reader seeded with Project Gutenberg books.

Dewey operates on two tiers of context: explicit (the selected text or current page, shown during the conversation) and implicit (the rest of the book, retrieved silently to enrich answers). On top of that, it supports five interactions: answering ad-hoc queries to keep the reader in flow, generating quizzes/flashcards on selected passages, recapping the book up to the current page when resuming after a break, looking up a term or character from earlier in the text, and surfacing past discussions via unobtrusive "sticky" icons next to paragraphs rather than inline highlights.

UX principles were minimalist throughout: the reading pane shows clean text only; Dewey stays hidden until called by text selection or a button; one-click predefined options reduce keystrokes for common queries; past discussions are accessible without obscuring the text.

The build followed a Claude-assisted product workflow: MoSCoW requirements from the Karpathy thread, SVG wireframes iterated via prompting (Claude's initial library screen was too dense; a follow-up prompt simplified it), and a database schema collapsed from a multi-table relational design to a single books table after explicitly asking for simplicity. v0.dev built the skeleton frontend, then Cursor handled backend integration. The stack is Next.js, Supabase (PostgreSQL), Railway for deployment, and Gemini Flash as the default LLM — chosen for its long context window and generous free tier, with Anthropic and OpenAI as alternates.

Planned extensions include voice I/O, chapter-level context selection via natural language, and fiction-specific features like character family trees — with the caveat that those require hallucination and spoiler guards to be reliable.

Building News Agents for Daily News Recaps with MCP, Q, and tmux

A main agent coordinating three parallel sub-agents — each in its own tmux pane, each owning a slice of RSS feeds — can produce a cross-source daily news digest with minimal glue code, and MCP is the cleanest way to wire the data tools together.

The system uses Amazon Q CLI as the agentic runtime and FastMCP as the tool layer. Six feeds are defined in `feeds.txt`: Hacker News, WSJ Tech, WSJ Markets, TechCrunch, AI News (smol.ai), and Wired AI. Each feed gets its own Python fetcher and parser registered as an MCP tool via the `@mcp.tool()` decorator — `get_hackernews_stories` fetches RSS XML with httpx, parses it with ElementTree, and returns formatted stories. Feed tools are exposed alongside Q's built-ins (`fs_read`, `fs_write`, `execute_bash`); none are trusted by default, so `--trust-all-tools` skips confirmation prompts.

Orchestration relies on two markdown instruction files rather than any framework primitive. The main agent reads `feeds.txt`, splits feeds into three chunks, then spawns sub-agents via `execute_bash` tmux commands, passing each its chunk and the `sub-agent.md` instructions. Sub-agents run concurrently in separate panes: they call the MCP feed tools, categorize stories (AI/ML, business, cybersecurity, trade policy, etc.), write per-feed summaries to `summaries/`, and print completion status. The main agent polls for completion, reads the summaries, and merges them into `main-summary.md`.

The May 4, 2025 run processed 124 items across 6 sources and identified 42 categories. AI/ML dominated at 25% (31 stories); Business/Finance followed at 14.5%. Top cross-source trends: AI agent collaboration, tariff impact on consumer tech, DOGE deploying AI across US government agencies, and Gemini scoring worse than predecessors on safety benchmarks.

Remote MCP hosting proved non-trivial on a weekend timeline and was deferred. Planned applications include design-doc parsing, COE post-mortems, and multi-agent writing workflows. Code is at `github.com/eugeneyan/news-agents`.

How to Work and Compound with AI

Effective AI collaboration compounds: every finished artifact becomes context for the next session, every correction updates a config that reduces future errors. Five practices enable this.

Context as infrastructure. Organize work into predictable paths so models navigate by grep and glob. An annotated `INDEX.md` beats bare URLs — annotations do the work once. Treat `CLAUDE.md` as an onboarding doc: glossaries, reading order. Split memory into `~/vault` (project facts) and `~/.claude` (preferences).

Taste as configuration. `~/.claude/CLAUDE.md` is a behavioral contract — directness, pushback norms, error-handling — scoped by directory: global, repo root, project subfolder. When it grows long, split into lazily-loaded guides. Frequent tasks become skills: markdown files encoding steps and the judgment of which apply. The `/polish` skill branches — run evals if there's a metric, inspect in Chrome if it renders, read output otherwise. Bootstrap by doing the task once and asking the model to extract the pattern.

Verification for autonomy. Verification is a ladder: deterministic hooks (ruff on save) at the bottom, LLM review at the top. Give models feedback loops — Docker errors, eval transcripts, Chrome rendering. For long sessions, run a secondary session against spec to catch execution drift (wrong steps) often and direction drift (wrong goal) occasionally.

Scaling via delegation. Specify intent, constraints, and success criteria upfront; let the model execute end-to-end. Three to six parallel sessions shift the bottleneck to writing specs and reviewing outputs — use git worktrees to keep sessions isolated.

Closing the loop. Mine transcripts for phrases like "still wrong" — each flags a missing step or stale config. Scanning ~2,500 past turns drove systematic `CLAUDE.md` updates. Refactor so each rule lives in one place; conflicting rules cause silent ignoring. The system trains a collaborator one feedback at a time — a principle that governs agent harnesses and team norms.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

Advice for New Principal Tech ICs (i.e., Notes to Myself)

At the principal level, the work that got you here becomes the side task. The core job is now technical vision, design feedback, sponsorship, and connecting dots across orgs. Writing code stays important for staying grounded, but nothing is not your job.

Being right covers less than half the battle. Convincing others to act is harder. Teaching the org to value something it doesn't yet care about is some of the hardest work — one mentor noted that three of ten pitched docs getting acted on is a great outcome. There's a category of work that won't happen without you, at the intersection of what you care about most and what you do best. Often the highest-leverage move is connecting who needs the work to who has done it.

Scaling through others is the primary lever. Spend one to two hours weekly with ICs you're grooming — success is when the org makes the same decisions you would without you present. When you hand off work it becomes theirs — including the right to take an approach you wouldn't. Intervene only at high-risk, one-way doors.

Structure your charter in three buckets: owner (~50% on 1–2 projects), sponsor (~20% on 2–3, building alignment), and consultant (reviews and guidance). In breadth roles every hour fills with escalations. Guard unscheduled thinking time — you can't see ahead going meeting to meeting.

Your title carries credibility even when unearned; offhand comments trigger large efforts. Share the "why" behind positions so people reason from your model rather than parroting conclusions. Silence in a meeting implies approval.

To reach principal you put yourself on the critical path; to be effective beyond it, actively remove yourself. The org should benefit from you without depending on you. Keep learning — if projects teach you nothing, you're moving backward.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

Data Science and Agile (What Works, and What Doesn't)

Agile works with data science on its engineering-adjacent side and breaks down on its research side — the challenge is knowing which is which.

Sprint planning transfers cleanly. One- or two-week sprints with explicit stakeholder engagement force trade-offs into the open: stakeholders see their "data effort budget" and bear the cost of context-switching in story-point terms, reducing the failure mode of a team pulled in five directions mid-sprint.

Defining tasks with deliverables upfront matters because data scientists' curiosity sends work down rabbit holes. For an NPS drop investigation, specifying in advance that deliverables cover delivery, product, customer service, and app metrics gives the work a stopping condition. The same logic applies to model-building: data extraction, preparation, feature engineering, validation, ML, and AB testing can be scoped as discrete milestones even when effort per step is uncertain.

End-of-sprint retrospectives and demos have outsized return on time — 30 to 60 minutes that compound team growth. Retrospectives structured around "enjoyable / frustrating / puzzling" generate improvement that, at 5% per sprint, reaches 12x over a year. Demo sessions build organisational visibility and lift junior members beyond their own tickets.

Four failure modes cut against agile. Estimation is genuinely hard: a ranking problem has open questions about modelling approach (learning-to-rank, classification, regression), success metric, and data quality — none of which resolve before the work starts. Stakeholder priorities shift as data reveals surprises, making sprint plans obsolete mid-execution. PMs from engineering backgrounds expect tangible deliverables each sprint; findings and null results don't fit that template. The most insidious failure is a team too disciplined at Scrum: stakeholder-set priorities push toward the urgent-but-not-important, and a clean burn-down chart can mask neglect of the 10x innovation only the data team is positioned to pursue.

Data Science and Agile (Frameworks for Effectiveness)

Standard agile breaks in data science because estimation variance is too high, scope shifts constantly, and stakeholders expect working software at sprint boundaries. The parts worth keeping — periodic planning, retrospectives, demos — need structural support to survive.

The central adaptation is time-boxed iterations across four sequential stages. Feasibility (2–4 weeks) answers one question: can existing data hit a reasonable performance floor? If the target is 95% accuracy and the ballpark is 90%, continue; at 70%, park the project and improve data first. POC (4–8 weeks) builds a minimal working model validated via local testing or A/B test, to decide whether production investment is warranted. Production deployment takes 3–9 months — a cited engineer with Google/Microsoft/Amazon/Oracle experience put the ratio at 2 man-months for a research prototype versus 117 for production quality. Operational maintenance follows. Two explicit go/no-go checkpoints at feasibility and POC prevent over-commitment of resources.

Sprint rituals are preserved but tightened: tasks are scoped to no more than 2 days of effort, the backlog is deliberately over-filled before stakeholders reprioritize (which forces them to confront the real cost of context switching), and end-of-sprint demos raise the team's bus factor by spreading knowledge.

Before any project begins, a one-page write-up is non-negotiable. The template covers: current situation and its cost, stakeholder intent, success metric (e.g., 95% top-3 precision at 40 classifications/second), deliverable, quantified downstream business benefit, dependencies, and constraints including language, framework, and latency ceiling. Answering these converts unknown unknowns into known unknowns before any code is written.

Innovation time — 20% or 2–3 weeks per quarter — must be explicitly budgeted, linked to an organizational outcome, and have a deliverable of documented learnings, or sprint pressure crowds it out entirely. The frame throughout is that valuable output (working models, negative findings, 10x ideas) matters more than procedural compliance.

What does a Data Scientist really do?

Machine learning makes up less than 20% of a working data scientist's day — the rest is problem framing, data wrangling, pipeline engineering, and shipping products to production. The public perception that the job requires a PhD, olympiad-level math, and constant model training is driven by availability bias: the data scientists who appear in Forbes Turing Award profiles (Hinton, LeCun, Bengio) and DeepMind AlphaStar posts represent under 1% of the field.

The useful split is between Type A (Analysis — statisticians working with data statically) and Type B (Building — engineers who ship data products). For Type B, the actual work moves through five stages. First, problem framing: understanding the business problem, identifying constraints like data refresh rate and security, defining optimization metrics, and flagging ethical risks. Second, data acquisition and preparation: sourcing data, cleaning nulls and outliers, joining across disparate systems, and visualizing signals. Third, framework and pipeline construction: building proper validation splits (random k-fold vs. time-based), data processing and feature engineering pipelines configurable via files rather than hardcoded, and ML experiment runners that work across model types. Fourth, experimentation: comparing broad approaches (trees, regression, SVMs, neural nets), running hyperparameter sweeps, checking learning curves for underfitting/overfitting, and running online experiments. Fifth, production: making pipelines scalable and robust, creating model APIs, scheduling jobs, monitoring input data drift and validation results, communicating results to decision-makers, planning rollbacks, and managing the ethics of how predictions get used.

When interviewing Chief Data Officers, CTOs, and Heads of Data Science — asking what separates rockstar data scientists — the consistent answer was not math ability or cutting-edge technique: it was "using data to deliver measurable value." That answer is harder to operationalize than studying algorithms, which is part of why the gap between public perception and actual practice persists.

My Journey from Psych Grad to Leading Data Science at Lazada

A psychology degree is not a barrier to leading data science — self-teaching, delivery, and communication beat credentials at every stage.

Eugene Yan spent two bored years as a government investment analyst before Coursera and EdX changed his direction. IBM was running a mid-career hiring cohort; he cleared the minimum technical bar via online coursework, accepted a 33% pay cut, and joined 20 hires — 10 of them PhDs — as the least technical in the group. At IBM he moved to workforce analytics and picked up Python, machine learning, and Spark on his own time.

A top-3% finish in Kaggle's Otto Product Classification challenge (200k products, 9 categories, gradient-boosted trees plus neural networks) led to a Singapore meetup talk, 80 attendees. John Berns, building Lazada's data team, heard about it and invited Yan for a "chat" — it became a job offer the next day.

At Lazada he built a product classifier for hundreds of millions of SKUs across thousands of categories, served via low-latency API with continuous retraining — production scale far beyond Kaggle. Volunteering to patch a cold-start problem in email recommendations ballooned into site-wide product ranking. His first A/B test lost millions in revenue in one week. Two weeks of overnight iteration — fresher behavioral logs, simpler model to reduce overfitting — turned it positive; the public failure-then-recovery raised the team's standing.

When growth plateaued, mentors two or three levels above said the priority wasn't deeper ML — it was communication. He dropped jargon with stakeholders, switched to fraud savings and conversion lift, and volunteered for the internal newsletter, overseas roadshows, and conferences. Promoted twice in one year to VP of Data Science leading 12+, his boss's rationale: 3× the measurable output of an average data scientist, and the one non-technical stakeholders could actually talk to.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

Writing is Learning: How I Learned an Easier Way to Write

Writing is not downstream of thinking — it is the thinking. Treating writing as a vehicle for sharing original ideas creates a false prerequisite that blocks most people from writing regularly. Sönke Ahrens's *How to Take Smart Notes* reframes it: writing is the medium of learning, not what follows it. That shift changes the whole process.

The practical system has three stages — read, take notes, write — with note-taking as the load-bearing connector most people skip.

On reading: don't wait for long blocks. Keep three books running simultaneously — one heavy (technical or self-improvement, for weekends or rested evenings), one lighter biography or narrative, one lightest fiction. This matches reading to available mental bandwidth, filling ten-minute pockets every day. Read with intent to write: ask what the key claim is and what you would teach from it; this filters filler and sharpens attention.

On note-taking: Ebbinghaus's forgetting curve puts retention at roughly 10% after seven days without reinforcement. Summaries in Evernote help but leave each book as an isolated island. The Zettelkasten method fixes this by linking ideas rather than filing books: each idea gets a card with a few sentences, linked to related idea cards, clustered in topic boxes. Topic boxes surface everything on a subject; card links let you follow chains of related thinking across sources.

On writing: originality is not the threshold. Mark Twain's point holds — ideas are recombinations of existing pieces. Pull up your topic box, see what others argued, identify where you agree, disagree, or can extend, and write from that scaffold rather than a blank page. Gaps surface during drafting and drive more reading. The cycle closes on itself.

Writing then produces a shareable artifact almost as a side effect of learning you would have done anyway.

Stop Taking Regular Notes; Use a Zettelkasten Instead

Regular note-taking fails not because notes are bad but because they stay isolated — connections between ideas are never made by default. When you review a note, related ideas don't surface. The result is scattered information, not retrievable knowledge.

The fix comes from German sociologist Niklas Luhmann, who built a Zettelkasten ("slip-box") of 90,000 handwritten index cards and used it to publish more than 70 books and 500 scholarly articles over 40 years. His core insight: a note is only useful in context — alongside the other notes it connects to. The system forces those connections at capture time, before you need to retrieve them.

The digital implementation runs on two note types. A literature note covers every useful book, article, or paper: tagged, sourced, and summarized as key ideas written in your own words, each elaborated with a few bullets.

A permanent note is made only for ideas meeting one of two criteria: you'll explore it further in your own work, or you can connect it to an existing permanent note. In Roam Research, wrapping text in `[[ ]]` creates the permanent note; you add topic tags (`#machine learning`, `#recsys`), a link back to the literature note, and explicit links to related permanent notes with a short explanation of each connection. Writing those explanations in your own words is the key cost — it forces genuine understanding rather than passive collection.

Topic navigation falls out automatically. Filtering by a topic tag surfaces all permanent notes in that domain. Over weeks the result is a knowledge graph that presents related material when you need it rather than requiring you to remember it exists.

The payoff comes at writing time: instead of starting from scratch, you query by topic and find ideas already articulated and pre-linked, ready to synthesize.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.

eugeneyan

Building a product classification API—one that accepts a product title and returns the top three most likely categories—requires more data preparation work than modeling. This post covers the first stage: getting clean, labeled training data from Julian McAuley's Amazon product metadata dataset (9.4 million products, 3.1 GB zipped), chosen over scraping because scraping alone would consume roughly 30% of total project effort.

The metadata fields include asin, title, price, image URL, related products (also_bought, also_viewed, bought_together), salesRank, brand, and categories. Categories are stored as a list of lists, since products are cross-listed across multiple hierarchies—an Onitsuka Tiger running shoe appears under both "Clothing, Shoes & Jewelry → Men → Shoes → Fashion Sneakers" and "Sports & Outdoors → Exercise & Fitness → Running → Footwear." The API uses only the primary (first) category, flattening the nested list to an arrow-delimited path string.

The cleaning pipeline proceeds through five steps with explicit product counts at each stage. Starting at 9.43 million products: dropping rows missing a title or category path leaves 7.98 million; removing categories where a title carries no categorical signal—Books, CDs & Vinyl, Movies & TV, which get classified by ISBN or content ratings rather than title keywords—leaves 5.59 million; keeping only the deepest category paths (filtering out "Clothing → Men → Shoes" when "Clothing → Men → Shoes → Fashion Sneakers" exists in the same category tree) leaves 4.61 million across roughly 15,000 categories, down from 17,600; finally, dropping any category with fewer than 10 products—the minimum needed for a 50/50 train-test split—leaves 4.59 million.

Part 2 addresses title-specific cleaning, which is the remaining preprocessing step before training the classifier.