This is a work in progress, but I want to get some thoughts down.
A traditional Christian view is that things in the world occur according to God’s providence.
Here's a theological definition: Providence is the means by which God directs all things — both animate and inanimate, seen and unseen, good and evil — toward a worthy purpose, which means His will must finally prevail. http://www.oneplace.com/ministries/thru-the-bible-with-j-vernon-mcgee/read/articles/providence-is-the-hand-of-god-11044.html
Often there’s an activist view of Providence.
Providence means that the hand of God is in the glove of human events. When God is not at the steering wheel, He is the backseat driver. He is the coach who calls the signals from the bench. Providence is the unseen rudder on the ship of state. God is the pilot at the wheel during the night watch. As someone has said, "He makes great doors swing on little hinges." God brought together a little baby's cry and a woman's heart down by the River Nile when Pharaoh's daughter went to bathe. The Lord pinched little Moses and he let out a yell. The cry reached the heart of the princess, and God used it to change the destiny of a people. That was providence. That was the hand of God. (same source)
So why do bad things happen to good people? Why is there evil in the world? God permits this to happen (permission) for various reasons, depending on the particulars of one’s theology. In particular, bad things may happen to us so that we may become better for the experience.
God can allow bad things to happen to good people in order to teach them lessons, to discipline them, to improve their character, to encourage them to depend on Him, etc. We know from the Scriptures that nothing occurs without God's permission (Ephesians 1:11). We also know that God is good, so we must conclude that He allows bad things to occur because they are according to His sovereign plan, and ultimately it will work out for good--especially for those who love Him (Romans 8:28). https://carm.org/why-do-bad-things-happen-good-people
So, we might deserve it due to our past sins, or it may be sent to teach us a lesson. But what lesson? Isn’t it hard enough to deal with the disaster that’s befallen us without trying to figure out how we might have deserved it, or trying to figure out what lesson we are to be learning?
Clearly in the Old Testament we have the idea that God’s providence leads to positive outcomes for those who live a good life, and negative outcomes for those who do not lead a good life.
10 The righteous shall rejoice when he seeth the vengeance: he shall wash his feet in the blood of the wicked.
In our time, this can lead to abominations, such as the prosperity gospel
Prosperity theology (sometimes referred to as the prosperity gospel, the health and wealth gospel, or the gospel of success)[A] is a religious belief among some Christians that financial blessing is the will of God for them, and that faith, positive speech, and donations (possibly to Christian ministries) will increase one's material wealth. Based on non-traditional interpretations of the Bible, often with emphasis on the Book of Malachi, the doctrine views the Bible as a contract between God and humans: if humans have faith in God, he will deliver his promises of security and prosperity. Confessing these promises to be true is perceived as an act of faith, which God will honor. https://en.wikipedia.org/wiki/Prosperity_theology
The flip side of the prosperity gospel is, of course, that if you are poor or sick, it’s your own fault.
The providence / permission theological view is similar to many systems of theology in that it is not falsifiable – there is no conceivable set of circumstances which the theory will not fit. It might not fit them well, but it will fit them. Not all such non-falsifiable theories are theological: similar criticisms have been made of Freudian psychology and string theory in physics.
In terms of science, if it can’t be falsified it can’t be supported, either. That’s why it’s called faith.
Probabilistic interpretation of reality
We can make an alternative interpretation in terms of probability theory. We can think of the universe as being set up in a certain way (e.g. by God) so that certain things happen probabilistically.
Let’s look at an example from my own life. In 1985, my wife and I had a child that lived only 28 days due to a rare birth defect, trisomy 18. Trisomy 18 occurs about 1 in every 6,000 births.
One can conceive of this as providence/permission, in which case God has chosen to have this happen to us. Why? To test us? To make us better people? There is no clear answer to this. If God was trying to make a point, it wasn’t clear at all.
One can also conceive of this as a random, probabilistic event. There is no particular meaning to this being our child, and our cross to bear, other than the fact that this happens 1/6000 of the time and unfortunately it was us. This is similar to buying a lottery ticket and winning – not because providence/permission intended you to win, but because of random chance.
In my case, thinking of our child as being a random event was liberating. I could use this as a learning experience, as a chance to grow, as an opportunity to be of service to my family, without trying to figure out some deep inner meaning that I might never find out.
These views of providence/permission and probability may seem inconsistent, but they are not necessarily so. In Herbert Weisberg’s book Willful Ignorance:The Mismeasure of Uncertainty he notes that making probability judgments involves putting events into classes. So we can make a class of births and a class of Trisomy 18 births and compare the size of the two classes in order to come to a probability. The willful ignorance part comes from the fact that in putting these events into classes, we ignore the differences between the individual events. We put all Trisomy 18s in a class, regardless of individual differences in the individual births, in order to measure the size of the class.
We might draw the classes up differently. We might look at female births and male births, and female Trisomy 18s and male Trisomy 18s and get somewhat different probabilities (because females with Trisomy 18 are more common), but we are still showing willful ignorance in putting these events into classes in order to determine a probability.
Note that by using such willful ignorance (and putting things into the right classes) we gain new knowledge about what will probably happen under some circumstances. But only probably happen – not certainly happen.
Not Necessarily Inconsistent
Here’s how we could resolve the apparent inconsistency: to us, things may appear to occur randomly, with a certain probability. This may be because they actually are random, or because they aren’t random, but we can’t see what’s really happening.
Weisberg uses the example of a roulette wheel, where each number on the wheel has a 1/38 chance of being the number. A roulette wheel is a random process, with the probabilities of occurrence clear.
Suppose, though, there was a genius who figured out a way to make the ball go into a specific slot, so he or his confederate could always win. But, in order not to make the casino suspicious, he picked his random winning numbers using a random number table, so he would decide “this time it is going to land in 7” and then “this time it is going to land on 25”. In this case, he would be perfectly in control, but the results would look random to both the casino and the other players. In this case, we have the genius controlling (providence/permission) but our best interpretation of this roulette universe would be that it is random.
That’s the key sentence there: our best interpretation of the world may be that it is random, within certain rules. One in 6000 births is a trisomy 18. 1 in X people will get lung cancer – a higher probability if you smoke, but still possible even if you don’t smoke. By now smoking, we are just improving our odds.
This finds an echo in the New Testament in Romans:. “For who has known the mind of the Lord, or who has been his counselor?” (Romans 11:34). Or, much earlier, Isaiah: “Who can fathom the Spirit of the LORD, or instruct the LORD as his counselor?” (Isaiah 40:13) But, of course, the probabilistic view does not require there to be a religious interpretation. Things could be random or from completely natural causes. A religious interpretation isn't falsifiable.
The probabilistic view at first seems cold and hard, but I don’t think it is. It seems better to me than to have good people try to figure out why God sent some bad thing their way, which seems an awful lot like blaming the victim, just as the Prosperity Gospel blames the poor for being poor.
A Clockwork Universe?
At this point, you might think this is similar to the clockwork universe of Laplace and Newton.
Laplace next stated his conception of a deterministic universe, arguing that blind chance, or fortune, is only “an illusion of the mind.” He subscribed to the principle of sufficient reason, according to which “a thing cannot occur without a cause which produces it.”
Weisberg, Herbert I. (2014-06-23). Willful Ignorance: The Mismeasure of Uncertainty (Kindle Locations 5158-5160). Wiley. Kindle Edition.
Laplace made many developments in probability theory, regarding probability as a way to approach our lack of complete understanding of causes.
But, after the development of quantum theory we now think Einstein was wrong when he contended God does not play dice with the world. At some subatomic level, things may indeed be probabilistic rather than deterministic.
Clockwork or not …
But this is a digression. In personally interpreting the world around us, it does not matter much if the randomness is due to our limited understanding, or if the randomness is really there at the heart of the matter.
Nor does it matter much if bad things are truly probabilistic, or if there is a God whose inscrutable ways we cannot understand if behind it. Bad things are best understood on a personal level as a probabilistic process, rather than God’s punishment or desire to teach us some lesson we can’t understand. Learn from the experience, but don't try to figure out why you were picked.