I have a sunny area which is full of gravel. Don't think I can get it removed. Lots of weeds doing very well, lots of worms in soil.
Can't recommend planting in the gravel, how deep is it? If you can rake the layer off you could amend the area with new soil and plant perennials. Just because weeds are happy growing there doesn't mean that parennials will do well. If the gravel is limestone it creates an acidic environment but if it is river stone it's more alkaline so if you plan to plant in it be sure the plants you choose like the enivironment. If the gravel is sparce you could try tough plants like monarda or sedums or some of the ornamental grasses.