In 2019, 7.1 million households in the United States (5.4%) were unbanked and lacked a checking or savings account (FDIC 2020). Using three leading household surveys, this paper documents how the interaction between bank access and poverty has evolved over time. We present a historical time series of unbanked rates, showing high unbanked rates for those in poverty even with increases in financial access over time. In the 1980s, 49.6% of households in poverty were unbanked while 22.8% were unbanked in 2019. Unbanked rates were even higher for Black and Hispanic households that were in poverty. In the 1980s, these groups had unbanked rates of 73.6% and 66.5% which declined to 38.4% and 31.8% in 2019, respectively. To explain differences in banking rates by race, we use Kitagawa-Oaxaca-Blinder decompositions that account for the binary nature of banking status. Using a robust set of controls that include assets and poverty status, our results suggest that group characteristics explain slightly less than half the difference in unbanked rates for Blacks and around half for Hispanics. Additionally, our results suggest that minimum balance requirements are the most cited reason for households being unbanked with around 8% of Black households, 6% of Hispanic households, and 1% of White households being unbanked for this reason. Our findings suggest continued inequalities in access to the financial system that have persisted over time.