The two little dots one occasionally sees over a, u, and o is called the umlaut, as we all know. It represents a vowel change also known as umlaut. This vowel change often has grammatical significance. Frequently, for example, plural nouns will have the umlaut. Also, some forms of some verbs will have it, and others will not. The obvious word pair distinguished by the umlaut is schon already, vs. schön pretty, fine.
As far as pronouncing o- and u-umlaut, my best advice is Don’t take lessons from Wayne Newton. The next best advice is listening, over and over, to a recording spoken by a native speaker (that is, if you can’t find a native speaker to help you). The third best advice is try what the textbooks say: form your lips to pronounce o or u (as the case may be) and say German e (long or short) with o, or i (long or short) with u. For me, ö is the hardest to pronounce naturally. A-umlaut is the easiest: just pronounce the German e sound (long or short), no need to pucker or do anything fancy with your lips.
The two dots originally represented a shortcut way of writing ‘e’ after the vowel. Some proper names are still written that way: Goethe and Bonhoeffer, come naturally to mind. In a pinch, you can always use that as a way of writing; if you can’t figure out the keyboard shortcuts, you can always write fuer for für, for example.
English preserves a few forms that have grammatically significant vowel change: man and men, for example; or even woman and women, which has an unwritten vowel change in the first syllable (i.e., o to i).