One task I found challenging is to make the edge of a tree look "leafy" and realistic after it's been masked. Who has the time or patience to clip around all those little leaves? Here's a method I devised that works, and it will help you turn a masked tree from a green blob with a blurry round edge to a more realistic, leafy element that will blend more naturally into your new scene.

For this tutorial, the source image comes from a 2008 contest, called Train Bridge, on photoshoptalent.com (a great place to learn more about using Photoshop!) The tutorial was done in Adobe CS2, but the tools are the same in CS and later versions - only the layout may look different.

1. Make a copy. Open your Layers palette if it's not already open (Window -> Layers, or F7). Make a copy of your original image by dragging the background layer to the Create A New Layer icon, then letting go. (Most of these steps are guidelines; this one is a rule: ALWAYS work on a COPY of your original image.)

 

2. Make your selection as close to the final tree shape as possible (it will save you time later). I used the Pen tool, set to Paths.
When you've closed the loop of your selection, right click on the path and choose Make Selection from the pop-up menu.
Depending on the resolution of your image, choose Feather 0.5-3 pixels.
Use the keyboard shortcut Ctrl-C then Ctrl-V to copy and paste into a new layer.

 

3. Clone to fill in and repair patchy areas. For this tutorial, I deliberately selected part of the train that I actually want to hide. Use the Clone tool to do that. Use a medium sized brush (choose best size for your image resolution) at 35-45% hardness, 100% opacity.
The key to realistic patching of trees and bushes - actually, cloning of any object - is to use several source points. After one or two clicks, I change my source point to minimize what I call "clone stutter" - that unsightly repetitive straight line pattern that's the result of using the same clone source over and over.

 

4. Erase your way to a new leafy edge. Now, tackle that rounded blurry edge. Select the Eraser Tool. From the Default brushes, choose any Spatter brush (depending on tree size), set to 100% opacity. Start at size 100, but change the size as you go along, choosing larger or smaller sizes to add realistic and random edges. Shortcut keys are [ to make tool smaller and ] to make it larger.
Experiment with different Spatter brushes to add more variety and randomness. Once in a while, click several times in the same spot to get a very clean erasure.
As you go around the object, move your brush more to the edge or more into the tree, to help re-shape the roundness. DO NOT drag your brush. (Do it once and you'll know why I say that.)

 

5. Want some sky or other background to show through the tree? Take this next step. I loaded a simple sky layer behind the tree layer so you can see the effect.
Still using your Eraser tool, choose #39 Dry Brush from your Dry Media Brushes. Size it to about 90, at 100% opacity. Click in one place two, three, or more times, until you're satisfied with the look. Repeat in a couple of places on the tree.

FINISHED! Now your tree or shrub is ready to be placed into another scene!

 

For your convenience, here are the instructions without images:

1. Open your Layers palette (Window -> Layers, or F7). Make a copy of your original image by dragging the background layer to the Create A New Layer icon, then letting go.

2. Use the Pen Tool, set to Path, to make your selection as close as possible to the final desired tree shape.
When you've closed the loop of your selection, right click on the path and choose Make Selection from the pop-up menu.
Depending on the resolution of your image, choose Feather 0.5-3 pixels.
Use the keyboard shortcut Ctrl-C then Ctrl-V to copy and paste into a new layer.

3. Use the Clone Tool to fill in and repair patchy areas. Use a medium sized brush (choose best size for your image resolution) at 35-45% hardness, 100% opacity. After one or two clicks, change your source point to minimize "clone stutter" and add natural randomness.

4. Create the leafy edge with the Eraser Tool. From the Default brushes, choose any Spatter brush (depending on tree size), set to 100% opacity. Start at size 100, but change the size as you go along, choosing larger or smaller sizes to add realistic and random edges. Shortcut keys are [ to make tool smaller and ] to make it larger. Experiment with different Spatter brushes to add more variety and randomness. Once in a while, click several times in the same spot to get a very clean erasure. As you go around the object, move your brush more to the edge or more into the tree, to help re-shape the roundness. DO NOT drag your brush.

5. To have sky or other background show through the tree, choose #39 Dry Brush from your Dry Media Brushes for the Eraser tool. Size it to about 90, at 100% opacity. Click in one place two, three, or more times, until you're satisfied with the look. Repeat in a couple of places on the tree.

FINISHED! Now your tree or shrub is ready to be placed into another scene!