I’ve never had a problem. If the collected rainwater has algae or other obvious particles in it, I might use a filter. (Or I might use that water just for irrigation, and see if the water in a different barrel is clearer.)
On that note, opaque and dark-colored barrels are most likely to keep the rainwater free of algae growth. A couple of my barrels are translucent or light-colored; those I cover keep covered with a sheet of canvas or other big cover (actually use a sauna cover that some neighbors were throwing away).
I always advise people that, if they have any doubt, to use the filter of their choice. A filter can be as simple as a piece of cheesecloth or a reusable coffee filter — or it could be a commercially available filter such as the Brita or Berkey, two very popular brands.
Or they could boil the water but it’s not usually necessary. Actually for water to be pasteurized it only needs to be 149° F for 20 minutes. (I just doublechecked and actually only 6 minutes at 149 F is needed!) Which is a lot easier than reaching the boiling point in some circumstances, such as situations with limited fuel.
Rainwater collected fresh off the roof and not right after a long dry spell (when dirt and other stuff can build up on a roof) is often very clear and fresh. If in doubt though, start small. Or start with just using it for bathing or for making hot tea/coffee.
If you have a downspout, there’s a special diverter attachment you can get that siphons off the first water off the dirty roof, so whatever follows is more likely to be clear and free of debris.
Fun story about water and impurities: 30 years ago when I lived in Tokyo, one of my fellow English conversation instructors was from Egypt. She told me that when she was growing up, her mother fed her a spoonful of Nile River water every day to help make her strong and resilient. For what it’s worth, she told me she was always very healthy.
That said, several billion people on the planet do not have access to safe drinking water, and do contract serious illnesses by drinking from rivers, lakes, and other groundwater that are their only available water source. One key aim of my rainwater research and advocacy is to help ensure safe drinking water for everyone on the planet (that people and communities are empowered to collect themselves, without corporate or government interference).