A decline of 2.3 percentage points doesn't seem like a lot, but when the eligible working population is over 100 million people it makes a big difference as the chart below shows (click to enlarge). (2015 data here, other data here)
The first line in the chart is the age 25 - 54 labor force participation rate. The second line is the 25 -54 civilian non-institutionalized population. This population number excludes people who are incarcerated or on active duty in the armed forces and is the one used by the Bureau of Labor statistics to calculate the labor force participation rate. The third row is the population in the labor force in that year e.g. in the 1992 column row 3 is the population in the labor force in 1992. The 4th row is the population in the labor force for all years using the 1992 rate e.g. in the 2012 column, if the labor force participation rate had been 83.6% instead of 81.4%, 103,960,780 people would have been in the labor force instead of 101,224,970 (2012 column, row 3). The 5th row does the same thing as the fourth only it uses the 2002 rate for all years and the last row takes the difference between the potential amount of workers at the 2002 rate and the actual amount of workers (5th row minus 3rd row for all columns, which is why it is 0 in the 2002 column).
What this chart shows is that if the labor force participation rate had been 83.3% in May, 2015 instead of 81% there would have been an additional 2,875,345 people in the labor force. Now perhaps not all of these people would have been employed since the labor force includes both employed and unemployed people who are looking for work. But if even 75% of them had a job that would be an additional 2,156,508 workers!
Those additional workers would be making cars, making airplanes, making hamburgers, serving food, driving trucks, cleaning houses, teaching kids, working at a bank or any number of other things. They would be actively engaged in the economy, producing stuff for others and earning a paycheck. So where are they?
Hopefully they are working in the underground economy. Perhaps they are being paid under the table to do yard work, freelance construction work, business consulting, or some other job that allows them to earn a living but keeps them out of the official statistics. This is still not good for the rest of us, since in the underground economy workers don't pay taxes, which means they don't contribute to the goods and services provided by the government, which means a higher tax burden for the rest of us.
In an even worse scenario those workers are not working at all. Instead they are relying on family members or friends to support them or they are receiving government aid in the form of food-stamps and other benefits. Either way, they are not supporting themselves but instead relying on other productive people to support them.
This makes us all worse off. The people who aren't working don't get the satisfaction of feeling useful and contributing to society while the rest of us not only have to support them, but we also miss out on the opportunity to purchase the goods and services that they could have produced. It's a lose-lose situation.
Now if some of these people left the labor force for private reasons e.g. they won the lottery and simply retired there is nothing wrong with that. But remember that this is the prime working age labor force participation rate, people age 25 - 54, so widespread early retirement should not be a big factor. If instead there are too many regulations and taxes and just general economic uncertainty, hindering entrepreneurs ability to create and produce their products which drags down job growth, then that is a serious problem that is hidden by low unemployment numbers. I think that this latter explanation is what is happening and I think it is a problem for the economy and will continue to be going forward, despite what the unemployment figure suggests.