This is one of those questions that has a hundred possible answers. All machines are going to create some heat. When passing electricity through a resistance (the wire in this case) the by product is heat. How much depends on the resistance of the wire. The wire naturally heats up, because the energy has to go somewhere, the energy is given off as heat.
One of the things that helps keep the heat down is the fact that the magnet is not constantly on, it's cycling and that helps the heat dissipate better. You want to avoid machines that heat up because they can run erratic and generally produce poor results. They can skip and sputter and once you move into guessing how your machine is going to react line to line, it's going to be to late for that tattoo.
Machines that are overheating can be a single problem, or a combination of problems. Always trouble shoot the whole electrical chain and all the components you are plugged into, but for now let's just focus on the machine.
In all my years of building and fixing tattoo machines, these are the most commons problems that result in a machine heating up, or over heating.
Dissimilar metals will cause a machine to heat up. In the case of brass and aluminum machine frames with a steel yoke, the smaller base path creates more resistance and then more heat. Brass is a denser material so it retains the heat, where as aluminum can dissipate it better, but neither creates as clean a path as a steel or iron machine frame. Also using a yoke that has been nickel plated or painted just creates more trouble on the travel path of the electrical current. The coatings will help insulate it as well (keep the heat in).
"If your machine is heating up you just need bigger coils." That is a myth. The idea that bigger coils are stronger isn't true. Coils with more wraps can develop a stronger magnet, but they take more voltage to power them and they can heat up quicker. I usually find machines with larger coil sets are set up with too much spring tension, and are troublesome to get to run smooth. There are many things to do to get a machine to run good and push larger groupings without larger coils.
One of the more popular problems I see is the old ground out. When a coil wire is not insulated properly and touching the frame it will result in the coils heating up and whole frame getting hot. Just a little contact won't ground the machine out so the a-bar just clicks down, but it will cause trouble. This is always the first stop on my check list. It's usually easy to spot, and easy to fix.
Again all parts of the coil wiring needs to be insulated, the wire, the capacitor wiring, the solder connections and the lugs. Make sure it's covered with insulator, shrink tubing, tape, glue, or shellac.
Now if you can't see a wire touching anywhere, and everything looks insulated you have to go next level and take the coils out of the frame. This is a sneaky ground out/ heat up problem I run into from time to time. You can see by the picture on the left that everything looks good and covered, but once you take the coils out just enough of the shrink has work away to cause a blind problem. I also see this happen with shrink tube or tape that has melted, especially when shrink has been shrunk with a lighter. At least use your girlfriends hair dryer to activate your shrink wrap. Be thorough.
Another deeper problem can be in the coils themselves. All magnet wire is coated to insulate it from itself, but if the coating becomes compromised it will cause the coils to heat up. The problem on the left is slight, but that scrape will cause a problem over time. The example on the right it a bit of an opposite problem, this wire is where the solder lug will connect and the coating hasn't been scraped well enough to make a good solder connection. Wire to be soldered needs to be scraped really well or the connection will be faulty. Also don't burn the coating off with a lighter. Most of the time the heat liquefies the coating, burns the dye off, and then dries fast and clear back over the wire. So the wire looks clean, but the temp coat is still there. Pretty sneaky sis...
Even deeper can be a problem at the coil core itself. If the core insulator is breached or scratched it will cause the wire to ground itself out on the core and heat up like nuts. To find this problem, you have to commit to scrap the coils. This is for those who are driven to have the answer. Yeah, I have found this problem more than once.
Another sneaky hidden problem can be the wire that comes out right by the coil core. It is tough to get the insulator to right up to the washer and still bend the wire without having it exposed to the core. Just another spot that can cause heat up trouble.
Some problems artists cause for themselves. Like too many elastics guy. That much needle tension will cause the machine to have to work hard, so ease off the elastics. Also make sure your bars are clear of you tubes and your needles aren't creating too much friction in the tip. I see that with some disposables. If the needle is riding in a misshapen tip will make a machine have to work harder and can heat up.
Painted and powder coated machine look great, but you have to make sure all points of contact are scraped clean of coatings. With out a good connection to the clip cord, the rear spring, the solder lugs and to the coil cores you are going to get resistance, heat and irregular running machines. Once a coated machine heats up those coating will also act as an insulator to keep the heat in.
Sometimes you just need to turn down the power. I will sometime trouble shoot a problem over the phone and run through 20 things and then I will ask the guy to run the machine for me so I can hear it. Once I hear the weed whacker going on the other end of the phone I try to get them to dial the power back.
My last favorite heat up story involves those folks who like to "break-in" a machine by running it non-stop for 24-48 hours. It will heat up, so don't do that.