I think it depends on the contract you sign
If it's non-exclusive, I don't see why not? You get paid, if it turns out horrible people might still come check out the original since folks who have read the original (hopefully you have a following) might rage/complain about it, therefore drawing readership, and you are not barred from signing another non-exclusive contract for someone to adapt it really well
Personally I think allowing folks the freedom to adapt your work is a mark of some of the great classics
Look at all the bad fanfiction the most popular fandoms have generated
Look at how many movie/show/comic/whatever spin-off adaptations the most popular classics of history have generated (Shakespeare, fairytales like Cinderella, asian classics like The Monkey King, the original Wuxia novels by Jinyong etc)... some adaptations are horrible, some are really good
Now if this is an EXCLUSIVE contract, and this is the ONLY adaptation of this medium that can be made of it, I would 100% think twice (and the answer is probably no)