VMS open() doesn't take O_BINARY, but takes a context description

Tell open() O_BINARY on VMS doesn't make sense, as it's possible to
use more precise file attributes.  However, if we're still going to
fdopen() it in binary mode, we must set the fd in binary context.

Reviewed-by: Rich Salz <rsalz@openssl.org>

1 files changed, 11 insertions, 1 deletions

diff --git a/apps/apps.c b/apps/apps.c
index 43a51f63ae..bb47039ce0 100644
--- a/apps/apps.c
+++ b/apps/apps.c
@@ -2817,7 +2817,17 @@ BIO *bio_open_owner(const char *filename, int format, int private)
 #endif
     }
 
-    fd = open(filename, mode, 0600);
+#ifdef OPENSSL_SYS_VMS
+    /* VMS doesn't have O_BINARY, it just doesn't make sense.  But,
+     * it still needs to know that we're going binary, or fdopen()
+     * will fail with "invalid argument"...  so we tell VMS what the
+     * context is.
+     */
+    if (!textmode)
+        fd = open(filename, mode, 0600, "ctx=bin");
+    else
+#endif
+        fd = open(filename, mode, 0600);
     if (fd < 0)
         goto err;
     fp = fdopen(fd, modestr('w', format));