The impact depends on your hive power (HP), not on how long you are around. You need a bunch to make a noticeable difference, but the fewer votes someone already has the more difference it makes for them!