The left side of the elevation drawer always corresponds to the starting position of your trip. The right side of the elevation drawer is always your destination. So the pointer on the map may go left if your trip direction is in the westerly direction. It will go right on if your trip is heading east.
The elevation drawer itself does not have a directional concept- it just pushes the map pointer to and from your starting point and destination.
Here is a video that will help: