Think of the IP address as being like a street address, and the port being like an apartment number. If mail to residents of a large apartment building had no apartment numbers on the envelope, all the mail would arrive in one massive jumbled heap, and each resident would need to sort through the heap to find their own mail.
Likewise, without port numbers, the network interface on a system would not know which service to route each incoming packet to. Furthermore, it may not even be possible to look at an incoming packet and determine by inspection what service it "belongs" to, since the contents may be ambiguous.
When a service runs, it registers itself to "listen" on a particular port. In effect, this tells the network interface "when you receive a packet that is tagged with this port number, send it to me". Many services have standard port numbers which they normally use, e.g. web servers listen on ports 80 (HTTP) and 443 (HTTPS) by default; SSH (secure shell) listens on port 22; SMTP (e-mail transport) listens on port 25; and so on.
Non-standard ports (or port remapping) may be used for a variety of reasons. The most common one is that a system is behind a NAT router/firewall, and therefore shares a public IP address with many other systems. If multiple systems behind a NAT router all want to be able to receive unsolicited packets from the public Internet, they can't all use the same port number (otherwise the NAT router would not know which system to forward the packet to). So let's say you have multiple web servers running behind a NAT. They can't all listen on the standard port 80, so maybe you use port 80 for the first one, port 10080 for the second one, 10081 for the third, and so on. (In HTTP/HTTPS, URLs need to have the port number appended if a non-standard port is to be used, e.g. http://www.somedomain.com:10080
Another reason for non-standard ports is to cut down on the number of brute force password cracking attacks for protocols like SSH. If you run SSH on the standard port 22 the system logs will be filled with spewage from script kiddies trying to guess the password to the root account. Moving the service to a high-numbered non-standard port reduces that kind of nonsense by orders of magnitude.
If the world isn't making sense to you, you're either drinking too much or not drinking enough.