If it is just certain extensions, it's possible the preview handlers got unregistered. This can happen for a variety of reasons, usually because some other software grabbed the entry (either when it was installed or run), though it's possible MS does this automatically if one of the office previewer apps crashes under certain circumstances (if something's wrong with Office such that it is crashing while rendering a preview it may take down Explorer as well, so it's actually good practice to disable yourself if you detect that happening, lest you crash explorer every time it browses a directory containing those files). You can run regedit and see what's registered for, say .docx files in:
It should look something like
Under ShellEx there should be an entry with {84F66100-FF7C-4fb4-B0C0-02CD7FB668FE} which is the Class ID (GUID) for the Word Previewer.
If you don't see that, definitely don't go editing the registry directly. Your best action would be to do a repair install of Office (which is the TL;DR version of this comment).