When you give someone a quitclaim deed for a piece of property, you're giving up whatever legal claim you have to that property. If you co-own a home with your brother, you could certainly give him a quitclaim deed. But if there's an outstanding mortgage on the property, the deed might not accomplish what you expect it to. You may not getting your finances untangled from your brother's that easily.
A quitclaim deed -- sometimes written "quit claim" or "quick-claim" -- is a document surrendering your right to assert ownership of a property. Unlike many other deeds, the quitclaim doesn't attempt to prove whether you actually have a legitimate claim to the property in question. It just says, "If I DID have some kind of legal claim, I'm giving it to you." Quitclaims are common in situations where several members of a family share ownership of a property.
The Mortgage Issue
The problem with using a quitclaim for a mortgaged home you own with your brother is that the quitclaim doesn't have any effect on the mortgage. Quitclaim deeds are used to deal with title questions -- establishing who the rightful owner of a property is. Your mortgage represents an obligation by you and your brother to repay the money you borrowed to buy the home. If the mortgage is still in both your names, then you're still on the hook for the money, quitclaim or not. If he says he'll make the payments, and he doesn't, the bank will come after you.
Look at Your Mortgage Terms
Depending on your home loan, a quitclaim might even violate the terms of your mortgage. Lenders commonly require borrowers to make certain guarantees about the homes they're planning to buy -- that they will use the home as their primary residence, that they'll keep it insured, that kind of thing. Trying to surrender your interest in the property may violate one of those terms. Be sure to check your mortgage agreement.
If you want to rid yourself of your mortgage obligations, you'll have to get your name taken off the loan. The easiest way to do that may be to "sell" the home to your brother so that he owns it free and clear. He would have to refinance the mortgage by taking out a new loan, one that's in his name only, and use the money to pay off the existing mortgage. During the refinance, you can have your name taken off the title, eliminating the need to deal with quitclaims at all.