03 · Exploratory data analysis¶
CSE273 Unit III. Types of data, loading and first inspection, univariate and bivariate analysis, correlation and multicollinearity, skewness and kurtosis, patterns, anomalies and trends, and how EDA feeds the modelling work.
EDA is where most of the useful decisions get made. The question for this dataset: what actually drives irrigation need, and is anything in the data going to cause trouble for a model?
try:
import groundwork
except ImportError: # running on Kaggle
%pip install -q git+https://github.com/LuShadowX/Groundwork
import groundwork
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
from groundwork import data, plotting
plotting.setup()
PALETTE = plotting.CLASS_COLORS
1. Types of data¶
| Kind | Sub-type | Columns here |
|---|---|---|
| Quantitative | continuous | soil pH, moisture, organic carbon, conductivity, temperature, humidity, rainfall, sunlight, wind, field area, previous irrigation |
| discrete | (none; id is a counter, not a measurement) |
|
| Qualitative | nominal (no order) | soil type, crop type, season, irrigation type, water source, region, mulching |
| ordinal (ordered) | crop growth stage (Sowing → Vegetative → Flowering → Harvest), and the target Low < Medium < High |
The type decides the tools: means and histograms for quantitative columns, counts and bar charts for qualitative ones, and a different encoding for nominal vs ordinal in CSE274.
2. Loading data and initial inspection¶
df = data.load()
num, cat = data.NUMERIC, data.CATEGORICAL
print(df.shape, "| numeric:", len(num), "| categorical:", len(cat))
df.sample(5, random_state=0)
(630000, 21) | numeric: 11 | categorical: 8
| id | Soil_Type | Soil_pH | Soil_Moisture | Organic_Carbon | Electrical_Conductivity | Temperature_C | Humidity | Rainfall_mm | Sunlight_Hours | ... | Crop_Type | Crop_Growth_Stage | Season | Irrigation_Type | Water_Source | Field_Area_hectare | Mulching_Used | Previous_Irrigation_mm | Region | Irrigation_Need | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 243132 | 243132 | Clay | 7.63 | 13.52 | 0.92 | 0.41 | 15.71 | 74.11 | 2187.04 | 8.67 | ... | Sugarcane | Flowering | Kharif | Canal | River | 12.10 | Yes | 46.98 | West | Medium |
| 258232 | 258232 | Silt | 7.76 | 9.23 | 0.65 | 0.14 | 25.02 | 72.42 | 1257.11 | 7.69 | ... | Cotton | Vegetative | Kharif | Canal | Rainwater | 12.49 | Yes | 113.63 | West | Medium |
| 395989 | 395989 | Clay | 6.20 | 12.77 | 1.49 | 1.52 | 21.72 | 80.02 | 1163.75 | 5.45 | ... | Cotton | Harvest | Kharif | Drip | Groundwater | 3.43 | No | 92.94 | East | Medium |
| 376404 | 376404 | Silt | 7.30 | 43.90 | 0.48 | 1.78 | 30.05 | 36.07 | 972.57 | 8.17 | ... | Maize | Harvest | Zaid | Canal | River | 5.90 | No | 107.20 | North | Low |
| 452357 | 452357 | Sandy | 7.07 | 40.40 | 0.98 | 1.17 | 25.31 | 69.83 | 738.24 | 10.03 | ... | Rice | Vegetative | Rabi | Canal | Groundwater | 13.85 | Yes | 79.07 | Central | Low |
5 rows × 21 columns
pd.DataFrame({
"dtype": df.dtypes.astype(str),
"missing": df.isna().sum(),
"unique": df.nunique(),
"example": df.iloc[0],
}).loc[num + cat + [data.TARGET]]
| dtype | missing | unique | example | |
|---|---|---|---|---|
| Soil_pH | float64 | 0 | 341 | 4.92 |
| Soil_Moisture | float64 | 0 | 5223 | 32.58 |
| Organic_Carbon | float64 | 0 | 131 | 1.01 |
| Electrical_Conductivity | float64 | 0 | 341 | 3.05 |
| Temperature_C | float64 | 0 | 2934 | 15.01 |
| Humidity | float64 | 0 | 6475 | 50.61 |
| Rainfall_mm | float64 | 0 | 19308 | 725.99 |
| Sunlight_Hours | float64 | 0 | 701 | 5.9 |
| Wind_Speed_kmh | float64 | 0 | 1935 | 16.79 |
| Field_Area_hectare | float64 | 0 | 1466 | 0.82 |
| Previous_Irrigation_mm | float64 | 0 | 10110 | 112.16 |
| Soil_Type | str | 0 | 4 | Loamy |
| Crop_Type | str | 0 | 6 | Sugarcane |
| Crop_Growth_Stage | str | 0 | 4 | Sowing |
| Season | str | 0 | 3 | Zaid |
| Irrigation_Type | str | 0 | 4 | Drip |
| Water_Source | str | 0 | 4 | Rainwater |
| Mulching_Used | str | 0 | 2 | No |
| Region | str | 0 | 5 | East |
| Irrigation_Need | category | 0 | 3 | Low |
fig, axes = plt.subplots(3, 4, figsize=(13, 8))
for ax, col in zip(axes.flat, num):
ax.hist(df[col], bins=50, color=plotting.ACCENT, alpha=0.85)
ax.set_title(col.replace("_", " "), fontsize=10)
ax.set_yticks([])
axes.flat[-1].axis("off")
fig.suptitle("Distribution of every numeric feature", fontweight="bold")
fig.tight_layout()
plotting.save("g03-histograms")
plt.show()
Most features are close to uniform (flat between a minimum and a maximum), typical of synthetic Playground data. Rainfall is the only one with a visible hump in the middle and thinner ends.
B. Box plot (numerical)¶
s = data.sample(30_000, seed=3)
fig, axes = plt.subplots(1, 4, figsize=(13, 4))
for ax, col in zip(axes, ["Soil_Moisture", "Temperature_C", "Rainfall_mm", "Wind_Speed_kmh"]):
sns.boxplot(data=s, x=data.TARGET, y=col, hue=data.TARGET, palette=PALETTE, ax=ax, legend=False, fliersize=1)
ax.set_xlabel("")
fig.suptitle("The four features that move with irrigation need", fontweight="bold")
fig.tight_layout()
plotting.save("g03-boxplots")
plt.show()
C. Count plots (categorical)¶
fig, axes = plt.subplots(2, 4, figsize=(14, 6.5))
for ax, col in zip(axes.flat, cat):
order = df[col].value_counts().index
sns.countplot(data=s, x=col, hue=data.TARGET, palette=PALETTE, order=order, ax=ax, legend=False)
ax.set_title(col.replace("_", " "), fontsize=10)
ax.set_xlabel("")
ax.tick_params(axis="x", rotation=30, labelsize=8)
fig.tight_layout()
plotting.save("g03-countplots")
plt.show()
g = sns.jointplot(data=s.sample(8000, random_state=0), x="Soil_Moisture", y="Temperature_C", hue=data.TARGET,
palette=PALETTE, s=10, alpha=0.5, height=6)
g.figure.suptitle("Moisture against temperature", y=1.02, fontweight="bold")
plotting.save("g03-joint", g.figure)
plt.show()
B. Line plot (numerical vs sequential)¶
The data has no dates, but rainfall can be treated as an ordered axis: mean need score (Low = 0, Medium = 1, High = 2) across rainfall deciles, one line per growth stage.
s2 = df.assign(need=df[data.TARGET].cat.codes, rain_decile=pd.qcut(df["Rainfall_mm"], 10, labels=False) + 1)
line = s2.groupby(["rain_decile", "Crop_Growth_Stage"])["need"].mean().unstack()
ax = line.plot(marker="o")
ax.set(xlabel="rainfall decile (1 = driest)", ylabel="mean need score", title="Rainfall only matters for the driest 10% of fields")
plotting.save("g03-line")
plt.show()
Two things show up: the vegetative and flowering stages sit far above sowing and harvest at every rainfall level, and rainfall itself only matters at the dry end. Need jumps in the driest decile and is flat everywhere else, which is a threshold effect rather than a straight-line one (trees handle that; a linear model would miss it).
C. Violin plot (categorical vs numerical)¶
fig, ax = plt.subplots(figsize=(10, 4.5))
sns.violinplot(data=s, x="Crop_Growth_Stage", y="Soil_Moisture", hue=data.TARGET, palette=PALETTE,
order=["Sowing", "Vegetative", "Flowering", "Harvest"], cut=0, inner="quartile", ax=ax)
ax.set_title("Soil moisture by growth stage and need")
plotting.save("g03-violin")
plt.show()
corr_df = df[num].assign(Need=df[data.TARGET].cat.codes)
corr = corr_df.corr(method="pearson")
fig, ax = plt.subplots(figsize=(10, 8))
mask = np.triu(np.ones_like(corr, dtype=bool), k=1)
sns.heatmap(corr, mask=mask, annot=True, fmt=".2f", cmap="RdBu_r", center=0, vmin=-1, vmax=1,
annot_kws={"size": 7}, cbar_kws={"shrink": 0.7}, ax=ax)
ax.set_title("Pearson correlation (Need = 0 / 1 / 2)")
plotting.save("g03-heatmap")
plt.show()
corr["Need"].drop("Need").sort_values(key=abs, ascending=False).round(3).to_frame("r with need")
| r with need | |
|---|---|
| Soil_Moisture | -0.455 |
| Wind_Speed_kmh | 0.258 |
| Temperature_C | 0.253 |
| Rainfall_mm | -0.111 |
| Humidity | -0.022 |
| Previous_Irrigation_mm | 0.020 |
| Field_Area_hectare | 0.018 |
| Electrical_Conductivity | 0.011 |
| Organic_Carbon | 0.006 |
| Sunlight_Hours | -0.000 |
| Soil_pH | 0.000 |
Pearson's r only sees straight-line relations. Spearman's rank correlation catches any monotonic relation; for this data the ranking of features is the same:
pd.DataFrame({
"pearson": corr_df.corr()["Need"],
"spearman": corr_df.corr(method="spearman")["Need"],
}).drop("Need").round(3).sort_values("spearman", key=abs, ascending=False)
| pearson | spearman | |
|---|---|---|
| Soil_Moisture | -0.455 | -0.446 |
| Wind_Speed_kmh | 0.258 | 0.252 |
| Temperature_C | 0.253 | 0.242 |
| Rainfall_mm | -0.111 | -0.075 |
| Humidity | -0.022 | -0.021 |
| Field_Area_hectare | 0.018 | 0.019 |
| Previous_Irrigation_mm | 0.020 | 0.019 |
| Electrical_Conductivity | 0.011 | 0.015 |
| Organic_Carbon | 0.006 | 0.007 |
| Soil_pH | 0.000 | -0.005 |
| Sunlight_Hours | -0.000 | 0.000 |
B. Multicollinearity¶
Features that are strongly correlated with each other make linear-model coefficients unstable. The variance inflation factor, VIF = 1 / (1 − R²) where R² comes from regressing a feature on all the others, measures it. VIF above 5–10 is a warning.
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant
Xv = add_constant(s[num])
vif = pd.Series([variance_inflation_factor(Xv.values, i) for i in range(1, Xv.shape[1])], index=num, name="VIF")
print(vif.round(3).sort_values(ascending=False))
# a deliberately collinear feature, to see what VIF does when there is a problem
Xc = s[num].assign(Moisture_copy=s["Soil_Moisture"] * 0.9 + np.random.default_rng(0).normal(0, 1, len(s)))
Xc = add_constant(Xc)
print("\nwith a near-copy of soil moisture added:")
print(pd.Series([variance_inflation_factor(Xc.values, i) for i in range(1, Xc.shape[1])], index=Xc.columns[1:]).round(1).loc[["Soil_Moisture", "Moisture_copy"]])
Soil_Moisture 1.006 Rainfall_mm 1.005 Temperature_C 1.003 Wind_Speed_kmh 1.003 Organic_Carbon 1.002 Previous_Irrigation_mm 1.002 Electrical_Conductivity 1.001 Humidity 1.001 Sunlight_Hours 1.001 Field_Area_hectare 1.001 Soil_pH 1.000 Name: VIF, dtype: float64 with a near-copy of soil moisture added: Soil_Moisture 219.3 Moisture_copy 219.3 dtype: float64
All real features have VIF ≈ 1: they are independent of each other, so there is no multicollinearity to fix. The added near-copy shows what a problem looks like (VIF in the hundreds).
6. Distribution analysis¶
A. Skewness and B. kurtosis¶
Skewness measures asymmetry (0 for a symmetric distribution, negative = long left tail). Excess kurtosis measures tail weight relative to a normal distribution (0 = normal, negative = lighter tails / flatter, like a uniform's −1.2).
shape = pd.DataFrame({
"skewness": df[num].skew(),
"excess kurtosis": df[num].kurt(),
}).round(3)
shape["looks like"] = np.select(
[shape["excess kurtosis"] < -1.0, shape["skewness"].abs() > 0.5], ["uniform-ish (flat)", "skewed"], "roughly symmetric")
shape.sort_values("skewness")
| skewness | excess kurtosis | looks like | |
|---|---|---|---|
| Rainfall_mm | -0.118 | -0.940 | roughly symmetric |
| Humidity | -0.089 | -1.164 | uniform-ish (flat) |
| Soil_Moisture | -0.064 | -1.181 | uniform-ish (flat) |
| Sunlight_Hours | -0.035 | -1.206 | uniform-ish (flat) |
| Wind_Speed_kmh | -0.028 | -1.242 | uniform-ish (flat) |
| Previous_Irrigation_mm | -0.017 | -1.255 | uniform-ish (flat) |
| Temperature_C | -0.004 | -1.206 | uniform-ish (flat) |
| Electrical_Conductivity | 0.048 | -1.148 | uniform-ish (flat) |
| Field_Area_hectare | 0.052 | -1.176 | uniform-ish (flat) |
| Soil_pH | 0.070 | -1.145 | uniform-ish (flat) |
| Organic_Carbon | 0.105 | -1.166 | uniform-ish (flat) |
fig, axes = plt.subplots(1, 3, figsize=(13, 3.6))
for ax, col in zip(axes, ["Rainfall_mm", "Soil_Moisture", "Organic_Carbon"]):
x = df[col]
ax.hist(x, bins=60, density=True, color=plotting.ACCENT, alpha=0.7)
grid = np.linspace(x.min(), x.max(), 200)
ax.plot(grid, stats.norm.pdf(grid, x.mean(), x.std()), color="#e76f51", lw=2, label="normal with same mean, sd")
ax.set_title(f"{col}\nskew {x.skew():.2f} · kurtosis {x.kurt():.2f}", fontsize=10)
ax.set_yticks([])
axes[0].legend(fontsize=8)
fig.tight_layout()
plotting.save("g03-skew")
plt.show()
pat = (df.groupby(["Crop_Growth_Stage", "Mulching_Used"])[data.TARGET]
.apply(lambda t: (t == "High").mean()).unstack().loc[["Sowing", "Vegetative", "Flowering", "Harvest"]])
ax = pat.plot.bar(rot=0, color=["#e76f51", "#2a9d8f"])
ax.set(ylabel="share of High fields", title="Mulching matters in the vegetative and flowering stages")
plotting.save("g03-pattern")
plt.show()
pat.round(3)
| Mulching_Used | No | Yes |
|---|---|---|
| Crop_Growth_Stage | ||
| Sowing | 0.003 | 0.000 |
| Vegetative | 0.108 | 0.019 |
| Flowering | 0.106 | 0.014 |
| Harvest | 0.006 | 0.001 |
B. Anomalies (outliers)¶
Two standard rules: values more than 1.5 IQR outside the quartiles, and |z| > 3.
def outlier_table(frame):
q1, q3 = frame[num].quantile(0.25), frame[num].quantile(0.75)
iqr = q3 - q1
return pd.DataFrame({
"IQR outliers": ((frame[num] < q1 - 1.5 * iqr) | (frame[num] > q3 + 1.5 * iqr)).sum(),
"|z| > 3": ((frame[num] - frame[num].mean()) / frame[num].std()).abs().gt(3).sum(),
"min": frame[num].min(), "max": frame[num].max(),
}).round(2)
outlier_table(df)
| IQR outliers | |z| > 3 | min | max | |
|---|---|---|---|---|
| Soil_pH | 0 | 0 | 4.80 | 8.20 |
| Soil_Moisture | 0 | 0 | 8.00 | 64.99 |
| Organic_Carbon | 0 | 0 | 0.30 | 1.60 |
| Electrical_Conductivity | 0 | 0 | 0.10 | 3.50 |
| Temperature_C | 0 | 0 | 12.00 | 42.00 |
| Humidity | 0 | 0 | 25.00 | 94.99 |
| Rainfall_mm | 0 | 0 | 0.38 | 2499.69 |
| Sunlight_Hours | 0 | 0 | 4.00 | 11.00 |
| Wind_Speed_kmh | 0 | 0 | 0.50 | 20.00 |
| Field_Area_hectare | 0 | 0 | 0.30 | 15.00 |
| Previous_Irrigation_mm | 0 | 0 | 0.02 | 119.99 |
Neither rule flags a single value: every feature stays inside a sensible physical range (pH 4.8–8.2, moisture 8–65%, temperature 12–42 °C). The data is clean. That is worth knowing, but it doesn't show the rules working, so 40 typical sensor faults are injected into a copy (a stuck moisture probe reading 999, a temperature logged in Fahrenheit, a negative rainfall) to check that the rules catch them:
faulty = df.sample(50_000, random_state=0).reset_index(drop=True)
rng = np.random.default_rng(0)
bad = rng.choice(len(faulty), 40, replace=False)
faulty.loc[bad[:15], "Soil_Moisture"] = 999.0
faulty.loc[bad[15:30], "Temperature_C"] = faulty.loc[bad[15:30], "Temperature_C"] * 9 / 5 + 32
faulty.loc[bad[30:], "Rainfall_mm"] = -faulty.loc[bad[30:], "Rainfall_mm"]
table = outlier_table(faulty)
q1, q3 = faulty[num].quantile(0.25), faulty[num].quantile(0.75)
flag = ((faulty[num] < q1 - 1.5 * (q3 - q1)) | (faulty[num] > q3 + 1.5 * (q3 - q1))).any(axis=1)
print(f"rows flagged by the IQR rule: {flag.sum()} · injected faults caught: {flag[bad].sum()} of 40")
table.loc[["Soil_Moisture", "Temperature_C", "Rainfall_mm"]]
rows flagged by the IQR rule: 36 · injected faults caught: 36 of 40
| IQR outliers | |z| > 3 | min | max | |
|---|---|---|---|---|
| Soil_Moisture | 15 | 15 | 8.01 | 999.00 |
| Temperature_C | 13 | 15 | 12.00 | 105.24 |
| Rainfall_mm | 8 | 9 | -2479.97 | 2499.69 |
Both rules catch most of the faults: the z-score rule flags 39 of the 40 bad values and the IQR rule 36. For flat, uniform-like features the IQR fences end up further from the data than three standard deviations do (for a uniform feature, 1.5 IQR beyond the quartiles is wider than 3 sd from the mean), so the IQR rule is the more cautious of the two here. The misses are values that land close to the normal range, such as a cool morning logged in Fahrenheit (about 55 °F) or a small negative rainfall. Single-feature rules can't see those, which is why CSE274 uses multivariate detectors such as Isolation Forest.
C. Trends¶
A trend is a steady change along an ordered variable. Mean need climbs with temperature in a clean, almost linear way, and falls with soil moisture:
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
for ax, col in zip(axes, ["Temperature_C", "Soil_Moisture"]):
bins = pd.cut(df[col], 25)
m = s2.groupby(bins, observed=True)["need"].agg(["mean", "sem"])
x = [b.mid for b in m.index]
ax.errorbar(x, m["mean"], yerr=1.96 * m["sem"], fmt="o-", ms=4, color=plotting.ACCENT, capsize=2)
ax.set(xlabel=col.replace("_", " "), ylabel="mean need score")
axes[0].set_title("Rising with temperature")
axes[1].set_title("Falling with soil moisture")
fig.tight_layout()
plotting.save("g03-trends")
plt.show()
8. EDA workflow integration¶
What this notebook decided for the modelling work:
| Finding | Consequence |
|---|---|
| No missing values, no duplicates | no imputation needed |
| High is 3.3% of rows | stratified splits; optimise balanced accuracy; class weights or resampling |
| Moisture, temperature, rainfall, wind, growth stage and mulching carry the signal | candidates for engineered features (e.g. dryness = temperature / moisture) |
| Stage × mulching interaction | tree models capture it for free; linear models need an interaction feature |
| Features are independent (VIF ≈ 1) | linear-model coefficients are trustworthy |
| Distributions are bounded and not heavy-tailed | scaling is safe; outlier removal not needed |
The loop is: inspect → visualise one variable → visualise pairs → quantify (correlation, skew) → check for anomalies → write down decisions. The next notebooks use every one of these decisions.