The maximum key length for rc5 is 2040 bits so we should not attempt to
use keys longer than this.

Issue found by OSS-Fuzz and Guido Vranken.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8834)

(cherry picked from commit 792cb4ee8d82e4b063f707fc9f4992271ffd65ab)