Does it take real talent to be a race car driver?

I have been looking at the requirements and backgrounds of race car drivers and it seems like it is for someone who has lots of money or connections. It doesn't look like alot of people even take up the activity as drivers school is expensive and there are few tracks. Of course it is daring but does is it really that physical and if so, does the physical skill only have to be somewhat better then the few that even consider taking up the sport? Are these drivers just marginal fish in a small pond?