The very term starboard means "side on which a vessel was steered," from steor- "rudder, steering paddle" + bord "ship's side." There were many small boats which were steered with a paddle (acting as a rudder) hung over the starboard side. When the rudder was shifted to the stern the actuator used to turn the rudder was attached to the rudder (a tiller) -- or the rudder was turned by lines run to a lever on the starboard side. The steering oar was located at the vessel's stern (rear), and most people then, as now, were right handed. For more information you can refer this link.