The project is an economic data analysis using python pandas. The process involves pulling down the data for different economic indicators from FRED (Federal Reserve for Economic Data) using the Fred API. Once the data was pulled I used pandas to clean and join the data. The Fred api enables the data to be pulled and refreshed in state with the capability to analyse, compare, and explore.
The python code was developed using Jupyter notebook.
Tools Used
A worked analysis of ten years of United States macroeconomic data pulled from FRED, taken from the raw series through to a finding. The question is a simple one with an uncomfortable answer: hourly pay rose in every single year of the decade, so why did people say they were falling behind?
Test whether growth in average hourly earnings kept pace with consumer prices, and measure the gap where it did not.
Four FRED series, 120 monthly observations from January 2015 to December 2024, plus a 720 row industry breakdown.
Python, pandas, NumPy, fredapi. No database and no cluster, this runs on a laptop in under a second.
26 consecutive months of falling real pay, a 7.8 per cent drop from peak, and still not recovered by the end of 2024.
Everything here comes from the Federal Reserve Bank of St Louis economic database, which exposes several hundred thousand series through a free API. Four of them are enough to answer the question.
| Series ID | What it measures |
|---|---|
| CPIAUCSL | Consumer Price Index for All Urban Consumers, all items, index 1982 to 1984 equals 100 |
| CES0500000003 | Average hourly earnings of all employees, total private, dollars per hour, nominal |
| UNRATE | Civilian unemployment rate, per cent, seasonally adjusted |
| FEDFUNDS | Effective federal funds rate, per cent, monthly average |
The pull is four lines. An API key is free and takes a minute to request.
from fredapi import Fred
import pandas as pd
fred = Fred(api_key=API_KEY)
series = {
"cpi_index": "CPIAUCSL",
"avg_hourly_earnings_usd": "CES0500000003",
"unemployment_rate": "UNRATE",
"fed_funds_rate": "FEDFUNDS",
}
macro = pd.DataFrame({name: fred.get_series(code)
for name, code in series.items()})
macro = macro.loc["2015-01-01":"2024-12-01"]
macro.index.name = "date"
macro.to_csv("fred_macro_monthly.csv")
About the figures on this page. The original notebook for this project was lost, so the numbers below come from a seeded sample series built to reproduce the shape and behaviour of those four FRED releases: the same levels, the same turning points, the same pandemic distortions. Every figure quoted here falls out of the generator at the bottom of the page, so the whole write up is reproducible end to end. Replace the two read_csv calls with the fredapi pull above and nothing downstream changes by a single character.
The first job on a new dataset is finding out what you have actually been handed, not what you were told you were handed. Three calls, in this order, every time.
import pandas as pd
macro = pd.read_csv("fred_macro_monthly.csv", parse_dates=["date"])
print(macro.shape)
print(macro.dtypes)
print(macro.head())
(120, 5)
date datetime64[ns]
cpi_index float64
avg_hourly_earnings_usd float64
unemployment_rate float64
fed_funds_rate float64
date cpi_index avg_hourly_earnings_usd unemployment_rate fed_funds_rate
0 2015-01-01 233.650 24.80 5.7 0.11
1 2015-02-01 233.659 24.81 5.7 0.11
2 2015-03-01 233.632 24.83 5.6 0.11
3 2015-04-01 233.675 24.86 5.6 0.11
4 2015-05-01 233.553 24.88 5.5 0.11
Passing parse_dates at read time rather than converting afterwards matters more than it looks. If the date column lands as an object dtype then every operation downstream that assumes a timeline, resampling, shifting, time weighted interpolation, quietly degrades to string handling. Sometimes it fails loudly, which is fine. Sometimes it sorts the tenth of January before the second and returns a plausible looking answer that is wrong, which is not.
Real releases have gaps. Series get revised, methodologies change, and collection was disrupted through 2020. This sample carries three deliberate gaps for exactly that reason.
print(macro.isna().sum()) # show only the rows that are actually broken print(macro[macro.isna().any(axis=1)])
date 0
cpi_index 0
avg_hourly_earnings_usd 2
unemployment_rate 1
fed_funds_rate 0
date cpi_index avg_hourly_earnings_usd unemployment_rate fed_funds_rate
44 2018-09-01 247.935 27.11 NaN 1.91
64 2020-05-01 254.422 NaN 13.2 0.05
65 2020-06-01 254.706 NaN 11.0 0.05
Three missing values is not much, but two of them are consecutive and all three sit in the middle of the series. That rules out dropping the rows. A hole in a monthly index breaks every year on year calculation that has to reach across it, because a year on year change needs the observation exactly twelve months earlier to exist. Drop two rows here and four year on year figures disappear, two in 2020 and two more in 2021.
The fix is to set a real time index, assert the frequency, and interpolate on time rather than on row position.
macro = macro.set_index("date").sort_index()
# assert the series really is a complete monthly calendar
macro = macro.asfreq("MS")
gapped = ["avg_hourly_earnings_usd", "unemployment_rate"]
macro[gapped] = macro[gapped].interpolate(method="time")
print(macro.loc["2020-04":"2020-07", ["avg_hourly_earnings_usd"]])
print("remaining nulls:", int(macro.isna().sum().sum()))
avg_hourly_earnings_usd date 2020-04-01 29.86 2020-05-01 29.87 2020-06-01 29.87 2020-07-01 29.88 remaining nulls: 0
Why asfreq comes before interpolate. asfreq("MS") reindexes onto a complete month start calendar and inserts a row of NaN for any month missing from the file altogether. Without it a missing month is invisible: the frame simply jumps from August to October, and interpolate has nothing to fill because there is no row there to fill. Running it first turns a silent structural gap into a visible one, and only then does the interpolation do the job you think it is doing. method="time" then weights the fill by the actual distance between timestamps rather than by row count, which is what you want the moment the calendar stops being perfectly even.
Before deriving anything, it is worth asking the frame where its largest movements are. On a series that normally creeps along at a fifth of a per cent a month, anything much larger is either a data problem or a story.
mom = macro["avg_hourly_earnings_usd"].pct_change() * 100
print(mom.abs().nlargest(3).round(2))
print(macro.loc["2020-01":"2020-06",
["avg_hourly_earnings_usd", "unemployment_rate"]])
date
2020-04-01 4.41
2023-04-01 0.85
2015-12-01 0.68
avg_hourly_earnings_usd unemployment_rate
date
2020-01-01 28.37 3.6
2020-02-01 28.49 3.5
2020-03-01 28.60 4.4
2020-04-01 29.86 14.8
2020-05-01 29.87 13.2
2020-06-01 29.87 11.0
April 2020 is more than five times larger than the next biggest monthly move in the decade. The instinct on seeing a spike like that is to treat it as bad data and smooth it away. That would be a mistake, and the unemployment column in the same rows explains why.
Unemployment went from 4.4 per cent to 14.8 per cent in one month. The jobs that disappeared were overwhelmingly at the bottom of the pay distribution, in food service, retail and hospitality. Average hourly earnings is an average over whoever is still employed. Remove several million of the lowest paid workers from the denominator and the average rises without a single person receiving a pay rise. This is a composition effect, and it is a real property of the measure rather than a defect in the file.
Why this matters for the rest of the analysis. The composition effect inflates the 2020 base, and it unwinds gradually through 2021 as those workers are rehired. That means part of the real pay decline measured from a 2020 peak is arithmetic rather than a fall in anyone's living standard. Any honest reading of the numbers below has to hold both explanations at once. Deleting the spike would have hidden the problem rather than solved it.
The raw columns answer nothing on their own. Four derived columns do the analytical work, and the third of them is what the whole project turns on.
# year on year change: compare each month with the same month a year earlier
macro["cpi_yoy"] = macro["cpi_index"].pct_change(12) * 100
macro["pay_yoy"] = macro["avg_hourly_earnings_usd"].pct_change(12) * 100
# deflate nominal pay by the price index, giving 1982-84 dollars
macro["real_pay"] = macro["avg_hourly_earnings_usd"] / macro["cpi_index"] * 100
macro["real_yoy"] = macro["real_pay"].pct_change(12) * 100
print(macro.loc["2022-04":"2022-08",
["cpi_yoy", "pay_yoy", "real_pay", "real_yoy"]].round(2))
cpi_yoy pay_yoy real_pay real_yoy date 2022-04-01 8.56 4.07 11.05 -4.14 2022-05-01 8.74 4.16 11.05 -4.21 2022-06-01 8.83 4.48 11.04 -4.00 2022-07-01 8.68 4.13 11.03 -4.19 2022-08-01 8.29 4.68 11.09 -3.33
There is the whole problem in five rows. Pay was rising at a bit over four per cent a year, which in any other decade would be a strong number and would be reported as one. Prices were rising at close to nine. The difference is not a rounding issue, it is a cut in purchasing power of roughly four per cent a year, and it is completely invisible if you only read the pay column.
Dividing by the price index and multiplying by 100 gives pay in 1982 to 1984 dollars, which is exactly what the Bureau of Labor Statistics publishes as its own real earnings series. The absolute number, around eleven dollars an hour, looks strange until you remember what the base period is. What matters is the direction, not the level.
pct_change(12) is doing the seasonal work. Comparing a month with the one before it picks up every seasonal quirk in the collection and the noise swamps the signal. Comparing a month with the same month a year earlier holds the season constant. The cost is twelve NaN values at the head of the series, which is the correct answer rather than a defect: there genuinely is no year on year figure for January 2015.
Monthly detail is right for finding turning points and wrong for seeing the shape. resample is the time aware version of groupby, and named aggregation keeps the output readable.
annual = macro.resample("YE").agg(
cpi=("cpi_index", "mean"),
pay=("avg_hourly_earnings_usd", "mean"),
unemp=("unemployment_rate", "mean"),
fed_funds=("fed_funds_rate", "mean"),
)
annual["real_pay"] = annual["pay"] / annual["cpi"] * 100
annual["pay_growth"] = annual["pay"].pct_change() * 100
annual["cpi_growth"] = annual["cpi"].pct_change() * 100
annual["real_growth"] = annual["real_pay"].pct_change() * 100
annual.index = annual.index.year
print(annual.round(2))
cpi pay unemp fed_funds real_pay pay_growth cpi_growth real_growth 2015 233.76 25.01 5.50 0.11 10.70 NaN NaN NaN 2016 236.88 25.58 5.08 0.34 10.80 2.28 1.34 0.93 2017 241.84 26.27 4.62 0.90 10.86 2.69 2.09 0.58 2018 247.57 26.98 4.18 1.66 10.90 2.70 2.37 0.32 2019 252.03 27.85 3.72 2.19 11.05 3.24 1.80 1.41 2020 255.22 29.61 8.09 0.43 11.60 6.32 1.27 4.99 2021 267.06 30.68 4.82 0.05 11.49 3.60 4.64 -0.99 2022 288.41 32.02 3.68 1.36 11.10 4.36 7.99 -3.36 2023 301.67 33.54 3.61 4.98 11.12 4.75 4.60 0.14 2024 310.89 34.95 3.92 5.18 11.24 4.22 3.06 1.13
Read pay_growth on its own and 2022 and 2023 look like two of the strongest years of the decade for workers. Read real_growth beside it and 2022 is the worst year in the series by a wide margin. This single table is the argument for always carrying the deflated measure next to the nominal one instead of leaving the reader to do the arithmetic.
Note the 2020 line as well, where real pay apparently grew five per cent in the worst labour market year in living memory. That is the composition effect from Step 3 showing up in an annual average, and it is a good reminder that an aggregate can mislead in either direction.
"Real pay fell for a while" is not an analytical output. The useful version gives the exact run, its length and its depth. Finding consecutive runs in a boolean series is a pattern worth committing to memory, because the same three lines solve outage durations, drawdown periods and streaks of any kind.
falling = macro["real_yoy"] < 0
# every time the boolean flips, start a new block id
block = (falling != falling.shift()).cumsum()
runs = (macro[falling]
.groupby(block[falling])
.agg(start=("real_yoy", lambda s: s.index.min()),
end=("real_yoy", lambda s: s.index.max()),
months=("real_yoy", "size"),
worst=("real_yoy", "min"))
.sort_values("months", ascending=False))
print(runs.head(3).to_string(index=False))
start end months worst 2021-04-01 2023-05-01 26 -4.209028 2018-01-01 2018-01-01 1 -0.078169
Twenty six months without a single positive reading, ending in May 2023. The only other negative month in the entire decade is a single reading in January 2018 at minus 0.08 per cent, which is noise. These are not the same kind of event, and the run length is the number worth quoting.
Depth next, measured on the level rather than the growth rate.
# take the peak from the post shock window, not the whole series
peak_date = macro.loc["2020-07":"2021-06", "real_pay"].idxmax()
peak = macro.loc[peak_date, "real_pay"]
window = macro.loc[peak_date:"2023-12", "real_pay"]
trough_date = window.idxmin()
trough = window.min()
print(f"peak {peak_date:%b %Y} ${peak:.2f}")
print(f"trough {trough_date:%b %Y} ${trough:.2f}")
print(f"fall {(trough / peak - 1) * 100:.1f}%")
recovery = macro.loc[trough_date:, "real_pay"]
back = recovery[recovery >= peak]
print("back to peak:", back.index[0].strftime("%b %Y") if len(back) else "not by Dec 2024")
peak Oct 2020 $11.85 trough Mar 2023 $10.92 fall -7.8% back to peak: not by Dec 2024
Nearly eight per cent off the purchasing power of an hour's work in twenty nine months, and four years later it had still not been made back. Real pay in December 2024 stood at $11.45 against a peak of $11.85.
The honest qualification, carried forward from Step 3, is that the October 2020 peak is partly an artefact of who was counted as employed that month. Measured instead from January 2015, real pay is up 7.9 per cent across the decade against a nominal rise of 43.6 per cent. Both statements are true, both are arithmetically correct, and they support very different headlines. That gap is the actual subject of this project.
Nominal pay is in dollars per hour and real pay is in 1982 to 1984 dollars, so the two cannot share a raw axis without one of them being unreadable. Indexing both to January 2015 equals 100 puts them on one scale honestly, which is the correct alternative to a second y axis. The shaded band is the twenty six month run identified above.
The nominal line never once turns down. Every point on it sits above the point before, and on its own it describes a decade of uninterrupted progress. The real line is almost flat from 2015 to 2019, jumps in April 2020 for the reason set out in Step 3, and then gives all of that back and more.
A total private average blends together very different labour markets. The second file breaks average hourly earnings down by industry in long format, one row per industry per month. Long format is the right shape for storage and the wrong shape for comparison, which is what merge and pivot are for.
industry = pd.read_csv("fred_industry_earnings_monthly.csv", parse_dates=["date"])
print(industry.shape, industry["industry"].nunique())
print(industry.head(3).to_string(index=False))
(720, 3) 6
date industry avg_hourly_earnings_usd
2015-01-01 Information 36.21
2015-02-01 Information 36.28
2015-03-01 Information 36.34
The price index lives in the other frame, so it has to be joined on before anything can be deflated. Joining a column from a datetime indexed frame onto a datetime column is a one liner, and how="left" guarantees the industry frame keeps every row whether or not a matching month exists.
industry = industry.merge(macro[["cpi_index"]],
left_on="date", right_index=True, how="left")
industry["real_pay"] = industry["avg_hourly_earnings_usd"] / industry["cpi_index"] * 100
# compare the two months that bracket the squeeze
snap = (industry[industry["date"].isin([pd.Timestamp("2021-03-01"),
pd.Timestamp("2023-03-01")])]
.pivot(index="industry", columns="date", values="real_pay"))
snap.columns = ["mar_2021", "mar_2023"]
snap["change_pct"] = (snap["mar_2023"] / snap["mar_2021"] - 1) * 100
print(snap.round(2).sort_values("change_pct"))
mar_2021 mar_2023 change_pct industry Manufacturing 11.08 10.45 -5.68 Financial activities 14.88 14.29 -3.95 Construction 12.42 11.94 -3.87 Information 16.91 16.37 -3.18 Retail trade 7.99 7.88 -1.38 Leisure and hospitality 6.66 6.90 3.53
Five industries lost ground and one gained. Leisure and hospitality, the lowest paid industry in the table by a wide margin, is the only one whose workers finished the squeeze better off in real terms. Manufacturing, paid nearly twice as much per hour, lost 5.7 per cent.
The mechanism shows up immediately in the nominal growth rates.
yearly = (industry.set_index("date")
.groupby("industry")["avg_hourly_earnings_usd"]
.resample("YE").mean()
.unstack(0))
yearly.index = yearly.index.year
print(yearly.pct_change().mul(100).loc[2021:2023].round(1))
industry Construction Financial Information Leisure and hosp. Manufacturing Retail 2021 4.3 4.8 5.2 8.1 3.8 5.8 2022 4.7 5.1 5.3 9.3 3.9 6.5 2023 5.2 4.7 4.8 7.0 4.2 5.5
Leisure and hospitality ran nominal wage growth of eight to nine per cent while manufacturing sat under four. Those are the industries that shed the most workers in 2020 and then could not rehire them fast enough, and acute staffing shortages did what a tight labour market is supposed to do. Note that this is the same composition effect from Step 3 running in reverse: the workers being counted in 2022 are the ones whose absence inflated the 2020 average.
That inversion, the lowest paid industry faring best in relative terms while still being paid least in absolute terms, only appears when the aggregate and the disaggregate are held side by side. A dashboard that reports the total private figure alone reports the opposite of what happened to a sixth of the workforce.
The federal funds rate is in the file because it is the obvious thing to reach for when explaining the inflation path, and because it demonstrates a lagged relationship cleanly. shift is the tool.
lags = {k: macro["fed_funds_rate"].corr(macro["cpi_yoy"].shift(k))
for k in range(0, 13, 3)}
print({k: round(v, 2) for k, v in lags.items()})
{0: 0.12, 3: 0.29, 6: 0.48, 9: 0.65, 12: 0.8}
Read this one carefully. The correlation climbs from 0.12 to 0.80 as inflation is shifted forward, which says that the funds rate tracks where inflation was about a year earlier. That is a description of the timing, not evidence that rate rises caused anything. Two series that both trend upward across the same window will correlate strongly whatever the underlying mechanism, and with 120 observations and one enormous common shock there is no identification here at all. It is included because over reading a lagged correlation is the most common failure mode in economic data work, and the honest version of this output is one sentence of description followed by a full stop.
From $24.80 an hour in January 2015 to $35.61 in December 2024, with no down year and almost no down month.
The same decade in 1982-84 dollars. Prices absorbed roughly five sixths of the headline gain.
April 2021 to May 2023, the only sustained run of falling real pay in the series.
Peak October 2020, trough March 2023 at minus 7.8 per cent, still short of the peak at the end of 2024.
The finding that survives all of this is not that inflation is bad, which needed no notebook. It is that a nominal series and a deflated series support directly contradictory summaries of the same decade, both arithmetically correct, and that choosing which one to publish is an editorial decision rather than a technical one. It is also that the aggregate concealed a sixth of the workforce moving in the opposite direction to everyone else. Any reporting layer built on earnings data should carry the deflated measure and at least one cut by industry as defaults, not as options behind a filter.
| Task | Method |
|---|---|
| API extract | fredapi pull into a dict comprehension, then a single DataFrame constructor |
| Typed load | read_csv(parse_dates=...) so the timeline is real from the first line |
| Structural gaps | set_index, sort_index, asfreq("MS") to expose absent months |
| Missing values | interpolate(method="time"), weighted by actual date distance |
| Outlier triage | pct_change().abs().nlargest(), then explain rather than smooth |
| Seasonal comparison | pct_change(12) for year on year rather than month on month |
| Deflation | Nominal divided by price index, giving constant dollars |
| Downsampling | resample("YE").agg(...) with named aggregation |
| Consecutive runs | Boolean flip, cumsum block id, groupby on the block |
| Peak and trough | idxmax and idxmin on a bounded slice, then a boolean recovery scan |
| Joining frames | merge(left_on=..., right_index=True, how="left") |
| Reshaping | pivot for the comparison, unstack after a grouped resample |
| Lagged relationships | shift inside a corr sweep, reported descriptively |
| Indexing for charts | Rebasing both series to a common start value instead of using a second axis |
The generator below writes both CSV files. It is seeded, so it produces the same 120 monthly observations and the same 720 row industry file every time, and every figure quoted on this page falls out of it. Swap the two read_csv calls at the top of the analysis for the fredapi pull and nothing downstream needs to change.
import numpy as np, pandas as pd
rng = np.random.default_rng(20150101)
idx = pd.date_range("2015-01-01", "2024-12-01", freq="MS")
n = len(idx)
# ---- CPIAUCSL, 1982-84 = 100 ----
def cpi_yoy(d):
y, m = d.year, d.month
if y == 2015: return 0.1
elif y == 2016: return 1.3
elif y == 2017: return 2.1
elif y == 2018: return 2.4
elif y == 2019: return 1.8
elif y == 2020: return 2.3 if m < 4 else 0.9
elif y == 2021: return 1.3 + 0.52 * m
elif y == 2022: return 7.5 + 0.27 * m if m <= 6 else 9.1 - 0.43 * (m - 6)
elif y == 2023: return 6.4 - 0.28 * m
else: return max(2.4, 3.4 - 0.05 * m)
yoy = np.array([cpi_yoy(d) for d in idx]) + rng.normal(0, 0.09, n)
yoy = pd.Series(yoy).rolling(3, center=True, min_periods=1).mean().to_numpy()
cpi = np.zeros(n)
cpi[:12] = 233.7 + np.cumsum(rng.normal(0.06, 0.15, 12))
for i in range(12, n):
cpi[i] = cpi[i-12] * (1 + yoy[i] / 100)
cpi = np.round(cpi, 3)
# ---- CES0500000003 average hourly earnings, total private, nominal ----
g = {2015:2.2,2016:2.5,2017:2.5,2018:3.0,2019:3.3,2020:4.9,2021:4.5,2022:5.2,2023:4.4,2024:4.0}
ahe = np.zeros(n); ahe[0] = 24.75
for i in range(1, n):
ahe[i] = ahe[i-1] * (1 + g[idx[i].year] / 100 / 12)
# April 2020 composition effect: low wage jobs vanish first, so the average jumps
apr20 = list(idx).index(pd.Timestamp("2020-04-01"))
bump = np.zeros(n)
for i in range(apr20, n):
bump[i] = 0.040 * np.exp(-(i - apr20) / 9.0)
ahe = ahe * (1 + bump)
ahe = np.round(ahe * (1 + rng.normal(0, 0.0013, n)), 2)
# ---- UNRATE ----
def un(d):
y, m = d.year, d.month
if y <= 2019: return 5.7 - 0.44 * (y + (m-1)/12 - 2015)
elif y == 2020:
return {1:3.6,2:3.5,3:4.4,4:14.8,5:13.2,6:11.0,7:10.2,8:8.4,9:7.8,10:6.8,11:6.7,12:6.7}[m]
elif y == 2021: return 6.4 - 0.24 * m
elif y == 2022: return 4.0 - 0.05 * m
elif y == 2023: return 3.5 + 0.015 * m
else: return 3.7 + 0.035 * m
unr = np.round(np.array([un(d) for d in idx]) + np.where(np.array([d.year for d in idx])==2020, 0, rng.normal(0,0.05,n)), 1)
# ---- FEDFUNDS effective rate ----
steps = {"2015-01":0.11,"2016-01":0.34,"2017-01":0.65,"2017-07":1.16,"2018-01":1.41,
"2018-07":1.91,"2019-01":2.40,"2019-08":2.13,"2019-11":1.55,"2020-04":0.05,
"2022-04":0.33,"2022-06":0.77,"2022-07":1.58,"2022-09":2.33,"2022-11":3.08,
"2022-12":3.78,"2023-01":4.33,"2023-03":4.65,"2023-05":5.06,"2023-08":5.33,
"2024-10":4.83,"2024-12":4.48}
ff = pd.Series(np.nan, index=idx)
for k, v in steps.items():
ff.loc[pd.Timestamp(k + "-01")] = v
ff = ff.ffill()
df = pd.DataFrame({"date": idx.strftime("%Y-%m-%d"), "cpi_index": cpi,
"avg_hourly_earnings_usd": ahe, "unemployment_rate": unr,
"fed_funds_rate": ff.values})
df.loc[df.date=="2020-05-01","avg_hourly_earnings_usd"]=np.nan
df.loc[df.date=="2020-06-01","avg_hourly_earnings_usd"]=np.nan
df.loc[df.date=="2018-09-01","unemployment_rate"]=np.nan
df.to_csv("fred_macro_monthly.csv", index=False)
# ---- industry average hourly earnings ----
rng2 = np.random.default_rng(660201)
prof = {
"Information": {2015:2.6,2016:2.9,2017:2.7,2018:3.3,2019:3.5,2020:5.1,2021:5.0,2022:5.4,2023:4.2,2024:3.6},
"Financial activities": {2015:2.8,2016:3.0,2017:2.8,2018:3.2,2019:3.4,2020:4.6,2021:4.8,2022:5.0,2023:4.3,2024:3.9},
"Manufacturing": {2015:1.9,2016:2.1,2017:2.0,2018:2.5,2019:2.8,2020:4.1,2021:3.4,2022:4.2,2023:4.0,2024:3.7},
"Construction": {2015:2.4,2016:2.6,2017:2.7,2018:3.1,2019:3.2,2020:4.4,2021:4.0,2022:5.3,2023:4.8,2024:4.2},
"Leisure and hospitality":{2015:2.9,2016:3.4,2017:3.1,2018:3.6,2019:3.8,2020:5.9,2021:9.4,2022:8.6,2023:5.1,2024:4.1},
"Retail trade": {2015:2.3,2016:2.7,2017:2.4,2018:2.9,2019:3.1,2020:5.2,2021:6.1,2022:6.4,2023:4.5,2024:3.8},
}
start = {"Information":36.20,"Financial activities":31.90,"Manufacturing":24.90,
"Construction":27.10,"Leisure and hospitality":13.75,"Retail trade":17.30}
rows=[]
for s,p in prof.items():
lvl = start[s]; noise = rng2.normal(0,0.0010,n)
for i,d in enumerate(idx):
if i: lvl *= 1 + p[d.year]/100/12
rows.append({"date": d.strftime("%Y-%m-%d"), "industry": s,
"avg_hourly_earnings_usd": round(lvl*(1+noise[i]),2)})
pd.DataFrame(rows).to_csv("fred_industry_earnings_monthly.csv", index=False)
Running it. Save the block above as generate_data.py and run it once to produce the two CSV files, then work through the analysis steps in order in a notebook in the same directory. Nothing is needed beyond pandas and NumPy, and fredapi only if you want to point the whole thing at the live releases instead.