Customize Firefox Scrolling

You can use extensions like Yet Another Smooth Scrolling or Smoothwheel, but there is another way without installing any add on.

Just go to the about:config page in Firefox : type about:config in the Location Bar (address bar) and press Enter to display the list of preferences.

Uncheck Show this warning next time

You can search in the Search bar at the top of the page to filter the preferences that you want to inspect. The search bar is case-insensitive, unlike the actual configuration variables.
Examples to filter results : mousewheel and smoothscroll .
                                                  

If you want to try out the settings I use, do the following :
Quit Firefox, navigate to ~/.mozilla/firefox/xxxxxxxx.default and create a file user.js with the following content :

// === Mouse Wheel Scrolling ===================================================
user_pref("general.smoothScroll", true);                            // boolean    enable/disable smooth scrolling
user_pref("general.smoothScroll.mouseWheel", true);                 // boolean    enable/disable smooth scrolling with mouse wheel
user_pref("general.smoothScroll.mouseWheel.durationMaxMS", 600);    // integer    smooth out the start/end of scrolling operations in ms
user_pref("general.smoothScroll.mouseWheel.durationMinMS", 400);    // integer    smooth out the start/end of scrolling operations in ms
user_pref("mousewheel.acceleration.factor", 10);                    // integer    sets acceleration factor if mousewheel.acceleration.start > -1
user_pref("mousewheel.acceleration.start", 0);                      // integer    when to apply mousewheel.acceleration.factor (after how many scroll clicks of mouse wheel)
user_pref("mousewheel.default.delta_multiplier_y", 85);             // integer    vertical step size

http://kb.mozillazine.org/About:config_entries

Update : there is another setting in about:config you can play with :
mousewheel.min_line_scroll_amount  default setting = 5