Well, kind of. They could have your phone, but you have a strong passcode locking Signal, or you could have uninstalled Signal, as in the OP. In those cases, the full Signal conversations would be protected, while any notification history stored by the system would be recoverable.