Most new guitars arrive from the factory with the nut just barely playable. Older guitars may have the nut filed or worn down so much that fret buzz cannot be eliminated by neck or string height adjustment. If you have a new guitar, or you are replacing the nut with a new one, here is an alternative method to file and adjust the nut material to make your guitar play like the professionals guitars play.
Before adjusting anything, make sure your guitar is strung up correctly and that your neck is straight and not bowed or warped. If your neck is bowed you first need to adjust the truss rod. If your neck is warped it will require a more extensive repair. For the lowest possible action or to avoid fret buzz all across your finger board it may be necessary to have your frets leveled and crowned first.
You will need a set of nut files (available from Stewart MacDonald), and a good set of feeler gauges as well. Different grades of sandpaper are very useful too.
Fret each string individually, starting with the High E, between the second and third fret, use your feeler gauge to check the amount of space between the bottom of the string and the first fret. You should have approximately .005" of space between each one, with the string barely touching the second fret. If this measurement is close or dead on then move on to the next string right up to the Low E string. You may want to record the gap on a scrap piece of paper as you move across the fret board, to see the nut slot's height in relation to the fret board as you do so.
For most players a string height (also known in guitar slang as “action”) of 3/64" of an inch is considered normal. Some players choose a higher sting height such as 4/64" of an inch while players which tend to have a light touch and want the fastest action possible strive to lower the action as close as possible to 2/64" which in many case's is very hard to setup and maintain without fret buzzing somewhere on the finger board.
Of course, you can use the traditional method to set your string height in relation to the nut, by using multiple feeler gauges below the nut, and filing down to the factory depth and width. However, I have found this method to provide a better and more consistent feel while playing near the nut.