So, you have a problem with the means (data collection and distribution) because you believe they negatively affect the end (statistics).
I don't think anyone can argue with you on that reasoning.
I can only disagree that you can trust your eyes. You can't because people can cloak their physical abilities, mental abilities, and finances. Therefore, you can't accurately determine the unemployment rate of those people by observation. Also, using your criteria, you cannot physically observe who is "able-bodied" or if those people are employable.
You can only trust your observation if you first check what the Labor Department is measuring, who they are collecting this information from, and how they distribute it to determine the results. However, observing a sample population, such as who is working and who is being counted in your neighborhood, will never get you to a correct answer. And while it may bring you comfort, how does it bring a solution?