?
Get a gravel vacuum.
Quetzel
There are siphon tubes with large ends to siphon the debris out of the gravel without sucking the gravel out at the same time. They work really well.
?
You could try a few things, such as crimping the siphon hose to reduce the flow volume, use a weaker/smaller siphon, or shorten the siphon tube and place the bucket at a higher elevation to reduce the suction.