I first started collecting anime 5 years ago, when I was only... 52 years of age! I am old enough to have watched the first anime broadcasts in the US as a kid way back in the mid 1960's, such as Speed Racer, Astroboy, Kimba the White Lion, and a few others, but I eventually lost interest in them and moved on to other things by around 1970, and didn't watch any anime after that. Then, in 2011, my two teenage daughters got me interested in anime once again when they introduced me to a couple Studio Ghibli films and The Girl Who Leapt Through Time, and those were intriguing enough that I wanted to see more. I started watching episodes of some of the more mature anime series on Adult Swim on television here in the US, such as Cowboy Bebop, Durarara!!, Samarai Champloo, and Casshern Sins. After I cut the cord on all TV at home back in 2012 and subscribed to Netflix so my daughters would have something to watch, I ended up viewing a good bit of anime on there. The first two anime series I watched from start to finish on Netflix were Blood+ and Last Exile, and those two series pretty much hooked me on anime and got me started on buying anime Blu-rays and building-up a collection.
Since I'm long divorced and have a decent-paying career in the IT field, I've had substantial finances available to build-up my collection fairly quickly, with well over 1000 titles, including a lot of Japanese, Korean, and UK imports and Aniplex of America releases. I also have a nice theater room setup in my basement with a 120" projection screen to watch my anime on, and I'm showing anime to a lot of my older, over-60 friends, who are surprised at how good some anime is from an adult perspective. So yeah, it's never too late to start collecting anime - actually it's probably best to be collecting as much as you can right now, if physical media is gradually going by the wayside.