After testing something more, it turns out that after running about 1.5 miles, the gearbox works just fine.
Tried to just turn off the ignition and then start the engine right after then it works fine, but after stopping for 5 minutes to fill the tank then it has to run for 1.5 miles before it works right again.