Although this can vary with design, aluminum flywheels are more likely to be damaged by heat than a standard weight iron or steel flywheel.
The lower weight/mass of an aluminum flywheel can cause the engine to idle rougher. And for the same reasons, the car will be more likely to stall like FocusBoy stated especially when starting from a stop.
If not made correctly, the starter ring gear on an aluminum flywheel is more likely to become loose.
The mounting bolts on some aluminum flywheels tend to loosen.(Torque the flywheel cold and use locktite)
Your car will accelerate noticeably faster(especially in the lower gears) with an aluminum flywheel.
If the car is everyday transportation and/or you find yourself in long traffic jams, I would not recommend an aluminum flywheel!