Mercury would be a challenging planet to terraform (though not as challenging as Venus.) It has many features that are positive for terraforming, including a low surface gravity (about the same as Mars' [0.38 G]), an internal dynamo that produces a magnetic field (which is especially important as a radiation shield so close to the sun), and considerable stores of water-ice and organic compounds in its polar craters.
Initial human habitats could be built mostly underground, to allow the establishment of a continuing human presence on the planet before terraforming. However, the above-ground parts of these colonies would need to be rigorously self-contained and the insides pressurized, like a space station. This is because Mercury's surface pressure is close to zero, a result of a combination of low gravity and proximity to the sun.
If it were determined that fully terraforming Mercury would be uneconomical or technologically impractical, the other option would be building enormous, transparent, geodesic domes across the planet's surface and terraforming the space contained inside them (as proposed by wikipedia ). This would mirror a method proposed to allow terraforming of the Earth's Moon.
Fully terrarforming Mercury would require first shielding the planet from excess solar radiation, before beginning the construction of an atmosphere on the planet. Mercury's native magnetic field is 1/100th the strength of Earth's. This is small; however, between the magnetosphere and the comparable gravity, Mercury in many ways should have an atmosphere comparable in thickness to Mars. However, Mercury's close proximity to the sun means that it experiences a much more intense pressure of solar wind, three times greater than that experienced by the Earth. The intense solar wind has prevented Mercury from accumulating an atmosphere thick enough to be relevant to humans.
Accordingly, construction of an atmosphere would only be possible once a shielding unit (such as solar shades) has directed some of the radiation pressure away from the planet's surface. There is a chance that gases from the solar wind would begin naturally accumulating to form a thin sort of atmosphere around Mercury; but human terraformers would undoubtedly want to thicken that atmosphere to make the surface of the planet more comfortable for colonists.
Unlike Mars, which contains frozen gases locked up in rocks across its surface, the gaseous components of Mercury's rocks have long been baked away. Gases to build an atmosphere would have to be brought to Mercury from elsewhere in the solar system. Since excess heat is an acute problem for a planet so close to the sun, greenhouse gases like methane or carbon dioxide would have to be kept to a bare minimum. Nitrogen for such an atmosphere could come from Titan. Icy comets could be redirected to Mercury to flood the low elevations with water, to create oceans. A supply of oxygen would be needed as well, most likely from belt asteroids.
Another problem with terraforming Mercury is that the length of its solar day is ~4223 hours, or the equivalent of 176 Earth days. Today, temperatures on Mercury are extremely different between the day side and the night side of the planet. Recent research has shown that an atmosphere thick enough to sustain humans might serve to even out the planet's overall temperatures; however, Earth plants would still have a very difficult time adapting to such an extreme day/night cycle.
As of right now, it is technologically unfeasible to alter the rotational speed of a planet, and this will continue to be impossible for the foreseeable future. One possible workaround to this would be to expand the terraformed world's solar shielding into a band of mirrors encircling the planet. This would enable the sun's light to be redirected to light one side of the planet's surface, simulating a shorter day/night cycle than would actually exist.
If terraformed, Mercury will face the problems of a Mountain Planet.