Yes, that can definitely be a problem if you are not redirecting all of your traffic to one or the other. The easiest way to solve this is to add a simple redirect to your site's .htaccess file. Here's an example of how to redirect all non-www traffic to www:

RewriteCond %{HTTP_HOST} ^domain.com
RewriteRule (.*) http://www.domain.com/$1 [R=301,L]

Editing your .htaccess file can be a little tricky, so be sure to do a little platform-specific research first before you do this as you could easily break your site!