Ammunition penetration is a bit of a paradox. On the one hand, you want your ammunition to penetrate sufficiently into a human target. On the other hand, you are rightly concerned about ammunition that will either over penetrate the target or upon missing the intended mark, will travel an excessive distance through common household construction medium. Much good work has been done in the area of ballistic testing, a great deal of the research conducted using your tax dollars, by the FBI. A good use of my money as far as I am concerned. The FBI determined that for the average human target, during the average attack, the projectile should penetrate between a foot and a foot and a half into the target. This enables the bullet to reach vital organs and produce the desired effect. Keep in mind that this is penetration through clothing, skin, muscle, fat, and bones. For those of us that have shot into a variety of living things, we know that this type of penetration is not a difficult task for most modern firearms/ bullets.
The risk of over penetration is another issue altogether and one that is subject to a great many variables. A suspect wearing heavy clothing, leather jackets, carrying a large layer of fat or muscle or other obstacles may not suffer the same level of penetration that a slim, lightly clothed person might, despite being shot with identical weapons and ammunition.
Then there is the concern of a missed round, traveling through the home, into unintended targets. There are scenarios where one might want to shoot through walls inside a house. An attacker taking cover behind an interior wall, can be engaged through the wall with a reasonable chance of success. However, in most cases, you do not want the bullets from your firearm going through multiple walls, and then possibly into a neighboring residence. To some degree, tactics and shooting angles can mitigate some concerns, but the proper ammunition selection can give you the best chance of success on the target, while mitigating some of the risks.
For ammunition selection, I would suggest, if you have the means, contact your local Law Enforcement agency and ask what their employees use. There is a good chance the agency has done significant research into this matter. This approach also has the added benefit of court defensible ammunition. If it is suitable for Law Enforcement use, certainly it is okay for citizens to use for self-defense.
Barring that, I would suggest a well-constructed hollow point bullet for handgun ammunition. Winchester Ranger, Speer Gold Dot, and Federal HST all performed well in ballistic testing, reaching the prescribed penetration depth. Being hollow points, they expanded nicely while transferring energy, thus limiting the risk of over penetration. They have a lower chance of over penetration than ball ammunition, but like most bullets, you can expect a missed round to go through several layers of interior home construction.
For rifle rounds, almost any ball round will sufficiently penetrate a human target to reach vital organs. If you are concerned about over penetration, the Hornady TAP, Urban rounds are designed to fragment heavily and remain in the body. The downside is these rounds do not have the same depth of penetration that ball ammunition will have. However, energy transfer with these rounds is fantastic. The 60 grain bullet is a good compromise. Another good option is the Black Hills TSX offering. This is a fully copper bullet (that is also legal to hunt with in California) that offers good penetration with lower risk of over penetration. The Black Hills TSX round was designed specifically for Law Enforcement use.
Finally, we come to the shot gun. OO Buck shot has a solid reputation for good penetration into targets. Like everything else, if it will penetrate a flesh and bone target well, you can expect it will go through interior walls. If you want reduced risk of penetration, then a #4 birdshot will suffice. It does not have the same ballistic success of OO buck shot, but that is the compromise you make. As for ammunition manufactures, there are many. Federal, Hornady, and Winchester all make excellent ammunition specifically designed for anti-personal use.
As you can see, there is no magic bullet that will provide sure success on target, without risking penetration through interior walls. Your best solution is to test a variety of ammunition for function and accuracy in your firearms, then train with the firearm until you are proficient. Use tactical angles and shooting angles to your advantage and have a family plan. Come join us for our home defense course where these topics (and more) are addressed. We hope to see you at the range.