There is this sense that you want to build from what works, so an immediate inference is to use and build off of verl. Although this is well reasoned, the flaw comes in the fact that verl doesn't just work, and many pushes to verl from open source contributors make glaring errors which you begin to pick up on when you step through the code on your own.
There are errors which have been popularly cited like those mentioning the complete mis-handling of off policy rollouts, the issue termed "token rerendering" whereby a model is made to do RL on tokens the original policy generated with a different tokenization, and presumably many others some of which related to the suboptimal handling of the inference engine's low precision mismatching with the training forward backward pass.
Now perhaps newer frameworks address these issues, such as prime RL, or SLIME, but my recommendation is to use whatever framework you choose as point of reference for developing your own implementation. Specifically develop your own code for training, and leverage their code as a known performance level your own should reach in terms of speed and training efficiency.
Larger models may be faster for experiments, or just make your ideas actually work. We found that with qwen 2.5 3b models there were things that these models just fell apart on, where larger models like qwen 2.5 7b would optimize on just fine. With only a handful of A100s available for optimization, this meant experiments would take a day, but they would work instead of just not working with instability and lackluster convergence. In conversations with others, they found a similar phenomena saying, they only got signal on their method after moving to a qwen3-8b long reasoning model over a qwen3-4b model.
Task interference for small models can distract from the findings of your method. We found that separating out the policy of belief generation and action generation resulted in a much better performing and more stable to train policies, although for time reasons this is not what we used for the policies trained in our paper.
vLLM vs SGLang: Early in experiments we found SGLang to be less stable for no apparent reason. Depending on verl, which at the time had a specific agent section that depended directly on SGLang, we only found this after moving frameworks days before a workshop submission. This was a very irritating instability to run into, and further informs my recommendation to depend on frameworks mostly as a point of reference for performance, as in the process of debugging another person's framework, you must understand how every small decision could be causing your issue. Creating the framework yourself makes the debugging process less frustrating.