Data Preparation for Machine Learning: Cleaning, Formatting, and Splitting Datasets

Why "Garbage In, Garbage Out" remains the first law of AI—and how to transform messy raw records into high-performing training datasets.

---

Ask any experienced machine learning engineer where they spend the majority of their time, and the answer is rarely "tweaking neural network architectures." In real-world AI projects, 70% to 80% of the engineering effort happens in data preparation—collecting, cleaning, normalizing, converting, and splitting datasets before a model ever sees a single training batch.

A state-of-the-art algorithm trained on noisy, biased, or leaked data will fail in production. Conversely, a simple baseline model trained on clean, well-engineered features often delivers remarkable accuracy.

If you are learning AI and machine learning, mastering the data preparation pipeline is the highest-leverage skill you can build. In this guide, we walk through every stage of preparing datasets for machine learning and LLM workflows.

---

1. Exploratory Data Analysis (EDA) and Auditing Raw Data

Raw data collected from production databases, web scrapers, IoT sensors, or user forms is almost never ready for mathematical modeling. Before writing a single preprocessing script, inspect your dataset's shape and distribution:

• Distribution Shape & Skewness: Plot numerical features using a Histogram to see whether values follow a normal bell curve, are heavily skewed toward one tail (like income or latency data), or contain multimodal peaks.
• Schema & Format Inspection: Raw exports often arrive in mixed formats—spreadsheet dumps, nested API responses, or flat CSV files. Converting between tabular and hierarchical structures using tools like our JSON ↔ CSV Converter or JSON ↔ Excel Converter makes it easy to inspect columns and flatten nested attributes.

---

2. Handling Missing Values and Outliers

Real-world datasets are full of gaps (null, NaN, empty strings, or placeholder values like -999 and "N/A") and extreme anomalies. Most machine learning algorithms cannot compute gradients when inputs contain NaN.

Strategies for Missing Data

1. Drop Rows or Columns: If a column is missing 85% of its values and carries little signal, drop the feature. If only a tiny fraction of rows lack a critical label, drop those rows.
2. Statistical Imputation: Replace missing numerical values with the mean (for normally distributed data) or median (for skewed data, since the median resists extreme outliers). For categorical features, impute with the mode (most frequent category) or a dedicated "MISSING" category.
3. Indicator Features: Sometimes the fact that a value is missing is itself predictive! Adding a binary column is_income_missing = 1 preserves that signal for the model while imputing the numeric column.

Detecting and Treating Outliers

A single corrupted sensor reading (e.g., a human age recorded as 950 instead of 95) can drastically pull the weights of a linear regression or neural network.

• Z-Score Method: Flag data points that lie more than $3$ standard deviations ($\pm 3\sigma$) away from the mean. You can verify the mean, variance, and automatic outlier thresholds of any numerical sample using our Standard Deviation Calculator.
• Interquartile Range (IQR) Method: Calculate $IQR = Q_3 - Q_1$ and clip or inspect values falling below $Q_1 - 1.5 \times IQR$ or above $Q_3 + 1.5 \times IQR$.

---

3. Feature Scaling and Encoding Categorical Variables

Machine learning models compute distances (like K-Nearest Neighbors and K-Means) or step along gradients (like Logistic Regression and Neural Networks).

Suppose your dataset has two features:

• years_of_experience: ranges from 0 to 40
• annual_salary: ranges from 30,000 to 250,000

Without scaling, annual_salary is thousands of times larger in magnitude than years_of_experience, causing gradient descent to oscillate wildly and distance metrics to ignore experience entirely.

Two Essential Scaling Techniques

1. Standardization (Z-Score Normalization): Rescales features to have a mean of $\mu = 0$ and a standard deviation of $\sigma = 1$:

$$z = \frac{x - \mu}{\sigma}$$

2. Min-Max Normalization: Compresses values into a fixed range, typically $[0, 1]$:

$$x_{\text{scaled}} = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$$

Note: Tree-based algorithms (Decision Trees, Random Forests, XGBoost) split features using threshold comparisons ($x > 50$) rather than distance metrics, so they generally do not require feature scaling. Neural networks, linear models, and SVMs strictly require it.

Encoding Categorical Data

Models require numbers, so text categories like "plan_tier": "pro" must be encoded:

• One-Hot Encoding: Creates binary 0/1 columns for nominal categories with no intrinsic order (e.g., color_red, color_blue, color_green).
• Ordinal Encoding: Maps ordered categories to sequential integers (e.g., "low": 0, "medium": 1, "high": 2).

---

4. Choosing the Right Dataset Format (CSV vs. JSON vs. JSONL)

Different stages of the AI pipeline favor different file formats:

Format	Best For	Key Characteristics
CSV	Classical tabular ML (scikit-learn, XGBoost, pandas)	Compact, flat rows and columns; poor support for nested arrays.
JSON	API payloads, configuration, small hierarchical datasets	Supports deep nesting; requires parsing the entire file into memory.
JSONL	LLM fine-tuning, prompt datasets, streaming big data	One self-contained JSON object per line; streamable line-by-line with minimal RAM.

When preparing conversational or instruction datasets for Large Language Models, JSONL (JSON Lines) is the industry standard. You can convert between standard JSON arrays and line-delimited JSONL files directly in your browser using our JSON ↔ JSONL Converter. And when a dataset grows too large to inspect or upload in a single chunk, use the File Splitter to break large CSV, JSON, or TXT files into smaller, manageable batches by line count or file size.

---

5. Splitting Datasets: Train, Validation, and Test Sets

The cardinal sin of machine learning is evaluating a model on the data it was trained on. A model with enough parameters can achieve 100% accuracy simply by memorizing the training rows—and then fail completely on tomorrow's unseen production data.

To measure true generalization, split your dataset into three mutually exclusive partitions:

1. Training Set (typically 70% – 80%): The data the model actively learns from by updating its weights via gradient descent.
2. Validation Set (typically 10% – 15%): Used during development to tune hyperparameters (learning rate, tree depth, regularization) and select the best model architecture.
3. Test Set (Holdout Set, typically 10% – 15%): Locked in a vault and touched only once at the very end of the project to report final, unbiased performance.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# 1. Split off 15% for the final Test Set
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42, stratify=y
)

# 2. Split the remaining 85% into Train (~70%) and Validation (~15%)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.1765, random_state=42, stratify=y_temp
)

# 3. CRITICAL: Fit the scaler ONLY on X_train to prevent data leakage!
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_val_scaled = scaler.transform(X_val)
X_test_scaled = scaler.transform(X_test)

---

6. Preventing Data Leakage (The Silent Model Killer)

Look closely at Step 3 in the Python snippet above: scaler.fit_transform(X_train) is called on the training set, while scaler.transform() is called on the validation and test sets.

Why?

If you compute the mean $\mu$ and standard deviation $\sigma$ across the entire dataset before splitting into train and test sets, information from the future test set bleeds into the training features. This mistake is called Data Leakage.

Common forms of data leakage to watch out for:

• Preprocessing Leakage: Fitting scalers, imputers, or vocabulary tokenizers on the full dataset prior to splitting.
• Target Leakage: Including a feature in $X$ that is only created after the target event occurs (e.g., including loan_late_fee_amount when trying to predict whether a new applicant will default on a loan).
• Time-Series Leakage: Randomly shuffling time-series data (like stock prices or server logs) so that Tuesday's data is used to predict Monday's outcome. Always use chronological splits for time-ordered data.
• Duplicate Leakage: Near-duplicate rows appearing in both the training set and the test set—especially common in LLM benchmark datasets.

---

Summary Checklist for AI Dataset Preparation

1. Audit Visually: Inspect distributions with histograms and summary statistics before modeling.
2. Clean Deliberately: Impute missing values and cap or remove corrupted outliers using statistical rules.
3. Format Appropriately: Use CSV/Parquet for tabular ML and JSONL for LLM instruction datasets.
4. Split First, Transform Second: Partition into Train, Validation, and Test sets before fitting any scalers or imputers.
5. Guard Against Leakage: Verify that every feature in your training set would genuinely be available at prediction time in production.