The optimum buy-in can be figured mathematically, but as I have said before, poker is more than math. There are some people who can buy-in for everything they own, because they have enough confidence to be able to build back up it they lose. Others choose the minimum allowed because of a self-imposed discipline or fear of loss.
I would suggest that you make an estimate of the total amount you will need to determine whether you can win. That amount should represent 1-to-2 percent of your total bankroll. Your buy-in should be anywhere from 1/3 to 1/5 of that amount. The percentage of your bankroll will permit you 50 to 100 opportunities which should reduce the tendency towards Gambler’s Ruin. The buy-in as a fraction of that number will be determined by the actual cost of the game. A tight game is 1/3 and a loose game could be up to 1/5.
Assume that you have calculated that a very loose 3/6 stud game is going to take about $300. That would leave you with five buy-ins at $60 or four buy-ins at $75. Because the game is very loose, you should opt for the $75 so as to maximize your best hands. Because stud starts are restrictive, you should adjust your strategy for the number of people in each hand and allow for enough money to cover your draws. You might even try three at $100.
There are several important factors to consider. The structure, the number of bets allowed and expected per round, and the types of hands you must play.
The structure is the blinds, fixed bets, antes and bring-in bets in relation to the maximum bets per round. The 10/20 hold’em in Vegas with a 5/10 blind is a more expensive game that the same 10/20 with a 2/5 blind, not just became of the higher coast of watching, but because that cost is high compared to the amount you can win. Stud has five rounds of betting and your opponents have four cards to draw out on you, so your starts must be of high quality. If the force-in betting per buy-in, you would be able to produce with three-to five buy-ins. Again, that total sum is 1-to-2 percent of bankroll I mentioned above. It is not necessary to lose the entire amount before you quit. If you feel you cannot win or risk, then leave and find a better game. Also these parameters assume that poker is your business and you are not renewing your bankroll from other sources.
Some games have more bets per hand than others. I played a 5/10 hold’em game at the Castaways (now sort of the Mirage parking lot) a few years ago that played like a 15/30. Some very good limit players were there and at least seven players had over $500 in front of them. One lady had assumed the role of designated opener and the isolation raised made identification very difficult. If a player tried to "rock" a $100 buy-in, he might look at three or four flops if they came quickly, but more likely he would get one and a half hands. This was a $300 minimum table with any chance of success with a $900 potential overall investment. The combination of players who were loosely aggressive and the table dynamics contributed to the cost factor of buy-in calculation.
It is hard to include personalities when you sit down, but be aware of what they can do to your calculations. One fixed raiser can really up the cost. If you realize you are in over your head, take a quick shot or get our. When your bankroll goes up, you will love these games, but wit for the right time. When I first started I tried to turn $20 into at least $100 everyday. A loose game cut down that strategy. It took more character to admit that I could not play loose games until later, than to accept a loss. I learned the hard way. (Is there any other way? That’s why I am writing so you can learn from my mistakes.)