RT2679: Fix error if keysize too short

In keygen, return KEY_SIZE_TOO_SMALL not INVALID_KEYBITS.

** I also increased the minimum from 256 to 512, which is now
documented in CHANGES file. **
Reviewed-by: NMatt Caswell <matt@openssl.org>

CHANGES

@@ -4,6 +4,10 @@
 
  Changes between 1.0.2 and 1.1.0  [xx XXX xxxx]
 
+  *) Increased the minimal RSA keysize from 256 to 512 bits [Rich Salz],
+     done while fixing the error code for the key-too-small case.
+     [Annie Yousar <a.yousar@informatik.hu-berlin.de>]
+
   *) Experimental support for a new, fast, unbiased prime candidate generator,
      bn_probable_prime_dh_coprime(). Not currently used by any prime generator.
      [Felix Laurie von Massenbach <felix@erbridge.co.uk>]

crypto/rsa/rsa_pmeth.c

@@ -504,9 +504,9 @@ static int pkey_rsa_ctrl(EVP_PKEY_CTX *ctx, int type, int p1, void *p2)
 		return 1;
 
 		case EVP_PKEY_CTRL_RSA_KEYGEN_BITS:
-		if (p1 < 256)
+		if (p1 < 512)
 			{
-			RSAerr(RSA_F_PKEY_RSA_CTRL, RSA_R_INVALID_KEYBITS);
+			RSAerr(RSA_F_PKEY_RSA_CTRL, RSA_R_KEY_SIZE_TOO_SMALL);
 			return -2;
 			}
 		rctx->nbits = p1;