I assume you are using a circuit something like this.
First, make sure you have the resistors in the correct place. The low, fixed value resistor (R1 in the schematic) is connected between Vout and ADJ. The variable resistor is connected between ADJ and GND (R2 in the schematic). Looking at a few schematics in Google images, several have R1 and R2 reversed in labeling and then reference the same formula you did. To get 3.3V out, you want R2 to be about 394 ohm.
The other item to consider is that the LM317 needs something around 3V difference between input and output, in order to stay within regulation. a 6.4V input is just barely within the margins for a 3.3V output. If the laptop power supply is slightly low, or your output voltage is closer to 3.4V, then the LM317 may be dropping out of regulation.