They basically define transparency on a layer without losing image information from that layer - i.e. if you have a photograph of a person with a beach background, but wanted to put them into a city background you could use a layer mask to hide the beach areas of the image with the person, and have only the person show on top of the city image. So it's kinda like turning parts of a layer off - you're not throwing any information away, you're just telling it what parts of the layer to show. Here's a tutorial on them: [link]
The way I had to do it was to use the pen tool, but select the "Exchange transparency" box near the color selectors, then you just use black to hide and white to reveal. Other than that I found you could use the selection tools is you've already got a solid line art going. lol, I'm not super well versed in SAI so there might be other ways to get it to work for you also.