The optimum buy-in can be figured mathematically, but poker is more than math. There are some people who can buy-in for everything they own, because they have enough confidence to be able to build back up if they lose. Others choose the minimum allowed because of a self-imposed discipline or fear of loss.
I would suggest that you make an estimate of the total mount you will need to determine whether you can win and the amount should represent one to two percent of your total bankroll. Your buy-in should be anywhere from 1/3 to 1/5 of that amount. The percentage of your bankroll will permit you 50 to 100 opportunities that should reduce the tendency towards Gambler’s Ruin. The buy-in as a fraction of that number will be determined by the actual cost of the game. A tight game is 1/3 and a loose game could be up to 1/5.
Assume that you have calculated that a very loose 3/6 stud game is going to take about $300. That would leave you with five buy-ins at $60 or four buy-ins at $75. Because the game is very loose, you should opt for the $75 so as to maximize you best hands. Because stud starts are restrictive, you should adjust your strategy for the number of people in each hand and allow for enough money to cover your draws. You might even try three at $100.
There are several important factors to consider. The structure, the number of bets allowed and expected per round, and the types of hands you must play.
The structure is the blinds, fixed bets, antes and bring-in bets in relation to the maximum bets per round. The 10/20 hold’em in Las Vegas with a 5/10 blind is a more expensive game than the same 10/20 with a 2/5 blind, not just because of the higher cost of watching, but because that cost is high compared to the amount you can win.
Stud has five rounds of betting and your opponents have four cards to draw out on you, so your starts must be of high-quality. If the force-in betting per buy-in, you would be able to produce with three-to five buy-ins. Again, that total is one to two percent of bankroll.
It is not necessary to lose the entire amount before you quit. If you feel you cannot win or risk, then leave and find a better game. Also these parameters assume that poker is your business and you are not renewing your bankroll from other sources.
Some games have more bets per hand than others. I played a 5/10 hold’em game at the Castaways (now sort of The Mirage parking lot) a few years ago that played like a 15/30. Some very good limit players were there and at least seven players had over $500 in front of them. One lady had assumed the role of designated opener and the isolation raised made identification very difficult. If a player tried to "rock" a $100 buy-in, he might look at three or four flops if they came quickly, but more likely he would get one and a half hands. This was a $300 minimum table with any chance of success with a $900 potential overall investment. The combination of players who were loosely aggressive and the table dynamics contributed to the cost factor of buy-in calculation.
It is hard to include personalities when you sit down, but be aware of what they can do to your calculations. One fixed-raiser can really up the cost. If you realize you are in over your head, take a quick shot or get out. When your bankroll goes up, you will love these games, but wait for the right time. When I first started I tried to turn $20 into at least $100 everyday. A loose game cut down that strategy. It took more character to admit that I could not play loose games until later, than to accept a loss. I learned the hard way. (Is there any other way?)
But, please, learn from my mistakes.