VMS: use selective search when linking with shareable images

VMS linking complains a lot about multiply defined symbols unless told
otherwise, especially when shareable images are involved.  For example, this
involves the legacy provider, where there are overriding implementations of
certain ERR functions.

To quiet the linker down, we need to say that symbols should be searched
selectively in shareable images.

However, that's not quite enough.  The order in which the VMS linker
processes files isn't necessarily top to bottom as given on the command line
or the option file(s), which may result in some symbols appearing undefined,
even though they are.  To remedy that, it's necessary to explicitly include
all object files and object libraries into a cluster, thus ensuring that
they will be processed first.  This allows the search for remaining symbol
references to be done in the as desired in the shareable images that follow.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19327)

1 files changed, 42 insertions, 31 deletions

diff --git a/Configurations/descrip.mms.tmpl b/Configurations/descrip.mms.tmpl
index b73db70c7d..2de2d6d633 100644
--- a/Configurations/descrip.mms.tmpl
+++ b/Configurations/descrip.mms.tmpl
@@ -1270,16 +1270,28 @@ EOF
       # previous line's file spec as default, so if no directory spec
       # is present in the current line and the previous line has one that
       # doesn't apply, you're in for a surprise.
+      # Furthermore, we collect all object files and static libraries in
+      # an explicit cluster, to make it clear to the linker that these files
+      # shall be processed before shareable images.
+      # The shareable images are used with /SELECTIVE, to avoid warnings of
+      # multiply defined symbols when the module object files override some
+      # symbols that are present in the shareable image.
       my $write_opt1 =
-          join(",-\"\n\t", map { my $x = $_ =~ /\[/ ? $_ : "[]".$_;
-                                 "WRITE OPT_FILE \"$x" } @objs).
-          "\"";
+          join(",-\"\n\t",
+               "\@ WRITE OPT_FILE \"CLUSTER=_,,",
+               (map { my $x = $_ =~ /\[/ ? $_ : "[]".$_;
+                      "\@ WRITE OPT_FILE \"$x" } @objs),
+               (map { my $x = ($_->{lib} =~ /\[/) ? $_->{lib} : "[]".$_->{lib};
+                      "\@ WRITE OPT_FILE \"$x/LIB" }
+                grep { $_->{lib} =~ m|\.OLB$| }
+                @deps))
+          ."\"";
       my $write_opt2 =
-          join("\n\t", map { my $x = $_->{lib} =~ /\[/
-                                 ? $_->{lib} : "[]".$_->{lib};
-                             $x =~ s|(\.EXE)|$1/SHARE|;
-                             $x =~ s|(\.OLB)|$1/LIB|;
-                             "WRITE OPT_FILE \"$x\"" } @deps)
+          join("\n\t",
+               (map { my $x = ($_->{lib} =~ /\[/) ? $_->{lib} : "[]".$_->{lib};
+                      "\@ WRITE OPT_FILE \"$x/SHARE/SELECTIVE\"" }
+                grep { $_->{lib} =~ m|\.EXE$| }
+                @deps))
           || "\@ !";
       return <<"EOF"
 $dso : $deps
@@ -1334,30 +1346,29 @@ EOF
       # is present in the current line and the previous line has one that
       # doesn't apply, you're in for a surprise.
       my $write_opt1 =
-          join(",-\"\n\t", map { my $x = $_ =~ /\[/ ? $_ : "[]".$_;
-                                 "\@ WRITE OPT_FILE \"$x" } @objs).
-          "\"";
+          "\@ WRITE OPT_FILE \"CASE_SENSITIVE=YES\"\n\t"
+          .join(",-\"\n\t",
+                "\@ WRITE OPT_FILE \"CLUSTER=_,,",
+                (map { my $x = $_ =~ /\[/ ? $_ : "[]".$_;
+                       "\@ WRITE OPT_FILE \"$x" } @objs),
+                (map { my $x = ($_->{lib} =~ /\[/) ? $_->{lib} : "[]".$_->{lib};
+                       # Special hack to include the MAIN object
+                       # module explicitly.  This will only be done
+                       # if there isn't a 'main' in the program's
+                       # object modules already.
+                       my $main = $_->{attrs}->{has_main}
+                           ? '/INCLUDE=main' : '';
+                       ( "\@ IF nomain THEN WRITE OPT_FILE \"$x/LIB$main",
+                         "\@ IF .NOT. nomain THEN WRITE OPT_FILE \"$x/LIB" ) }
+                 grep { $_->{lib} =~ m|\.OLB$| }
+                 @deps))
+          ."\"";
       my $write_opt2 =
-          join("\n\t", "WRITE OPT_FILE \"CASE_SENSITIVE=YES\"",
-                       map { my @lines = ();
-                             use Data::Dumper;
-                             my $x = $_->{lib} =~ /\[/
-                                 ? $_->{lib} : "[]".$_->{lib};
-                             if ($x =~ m|\.EXE$|) {
-                                 push @lines, "\@ WRITE OPT_FILE \"$x/SHARE\"";
-                             } elsif ($x =~ m|\.OLB$|) {
-                                 # Special hack to include the MAIN object
-                                 # module explicitly.  This will only be done
-                                 # if there isn't a 'main' in the program's
-                                 # object modules already.
-                                 my $main = $_->{attrs}->{has_main}
-                                     ? '/INCLUDE=main' : '';
-                                 push @lines,
-                                     "\@ IF nomain THEN WRITE OPT_FILE \"$x/LIB$main\"",
-                                     "\@ IF .NOT. nomain THEN WRITE OPT_FILE \"$x/LIB\""
-                             }
-                             @lines
-                           } @deps)
+          join("\n\t",
+               (map { my $x = $_->{lib} =~ /\[/ ? $_->{lib} : "[]".$_->{lib};
+                      "\@ WRITE OPT_FILE \"$x/SHARE/SELECTIVE\"" }
+                grep { $_->{lib} =~ m|\.EXE$| }
+                @deps))
           || "\@ !";
       # The linking commands looks a bit complex, but it's for good reason.
       # When you link, say, foo.obj, bar.obj and libsomething.exe/share, and