Well, the glue from the velcro will probably force you to refinish the stock anyway. So you might as well 'play' with it...
Take off the recoil pad and look at the stock bolt hole - then find a piece of PVC pipe that is very close to the ID of the hole (just under), wrap the pipe in kitchen plastic wrap (acts as a 'release agent' for any glue) - you now have a 'backer' for whatever glue/filler you use on the hole.
I think I would try a mixture of glue mixed with sawdust as a filler (you can scrape/drill into the end of the stock that resides under the recoil pad). At first you want to just fill the bottom of the hole (use a strong epoxy here) - this will give you a base to build up from (this can be done with a glue/sawdust mix - of any origin because you will put a 'top coat' on it at the end (the top one needs matching sawdust and a clear drying glue). Several small coats will serve better than a single large one. I might even add an internal patch once the hole is sealed - think stronger glue - maybe an epoxy laid in with a mesh to add strength (window screen? Drywall seam tape?).
You will want to finish with the repair standing 'proud.' Next, as you refinish the stock, you can sand down the proud repair to match the contour - color/stain to match, and then refinish the stock.
That being said - if you are crazily careful in what you do, you might be able to just fill the hole, sand that area down, color it and then clear coat only the repair - will not be perfect, but saves the full refinish of the stock.