This post covers the workings of a couple of scripts that allow for the secure transmission of files between two computers. The usual approach to accomplish this is by using scp with pre-shared keys. However, there are some problems with this approach, such as:

It opens a large attack surface on the side that trusts the keys of the other party, making it hard to limit.
It assumes that both parties can communicate with each other directly, which is not always true.

In this guide, I will not cover how files are transferred from one computer to another. Instead, it focuses solely on securing communications.

General Approach

The goal is to send a file securely from system A to system B.

Both sides generate private/public RSA keys and exchange them. The exact method for exchanging public keys without them being substituted in transit is left to the reader.
The sending side, A, generates one-time-use transport encryption keys for a symmetrical encryption algorithm.
Side A encrypts these keys with the public key of side B so that only side B can decrypt them.
Side A encrypts the payload file with a symmetrical cipher using the transport encryption keys.
Side A combines the encrypted keys and the encrypted payload into a single file and sends it to B.
Side A signs the combined (encrypted payload + encrypted keys) file with its private key and sends it to B.

Via some mechanism, the encrypted payload and signature file get transferred to system B. B processes it as follows.

Verify the sizes of both the signature file and the file with encrypted payload + keys. The size of the signature file is fixed, while the size of the payload file should be no less than the size of the encrypted transport keys.
Verify that the received signature matches using the public key of A.
Extract and decrypt the one-time-use transport encryption keys using the private key of B.
Use the decrypted transport keys to decrypt the encrypted payload.
Dance and celebrate.

Below is the shell script implementing the described logic for both sides. It made some assumption about base system, but there is nothing to specific in it. It should work with any decent Unix system, but your mileage may vary (YMMV).

Preparation

Side A ( sending )

 openssl genpkey -algorithm RSA -out A_private_key.pem -pkeyopt rsa_keygen_bits:4096
 openssl rsa -pubout -in A_private_key.pem -out A_public_key.pem

Securely tranfer A_public_key.pem to computer B. Public keys are not confidential but you need to ensure that file did not get substituted on transit.

Side B ( reciving )

Same deal

  openssl genpkey -algorithm RSA -out B_private_key.pem -pkeyopt rsa_keygen_bits:4096
  openssl rsa -pubout -in B_private_key.pem -out B_public_key.pem

Securely tranfer B_public_key.pem to computer A.

Side A (sending) scripts

side A is operated in trusted enviroment, so there is need for error handling.

TARGET_DIR="/tmp/out"  # must exist
SOURCE_DIR="/tmp/in"
BASE_FILENAME="top_secret.bin"
REMOTE_PUBLIC_KEY="/tmp/certs/B_public_key.pem"
LOCAL_PRIVATE_KEY="/tmp/certs/A_private_key.pem"

OPENSSL="/usr/bin/openssl"

# Function to handle errors and exit the script
error_exit() {
    echo "Error: $1" >&2
    exit 1
}

# Check if required files exist
[ -f "$REMOTE_PUBLIC_KEY" ] || error_exit "Remote system public key not found: $REMOTE_PUBLIC_KEY"
[ -f "$LOCAL_PRIVATE_KEY" ] || error_exit "Local private key not found: $LOCAL_PRIVATE_KEY"
[ -f "$SOURCE_DIR/$BASE_FILENAME" ] || error_exit "Source file not found: $SOURCE_DIR/$BASE_FILENAME"

# Step 2: Create transport secret key and IV.
TRANSPORT_KEY=$($OPENSSL rand -hex 32)
TRANSPORT_IV=$($OPENSSL rand -hex 16)

# Step 3: Encrypting transport key/IV
echo "$TRANSPORT_KEY$TRANSPORT_IV" | $OPENSSL rsautl -encrypt -inkey "$REMOTE_PUBLIC_KEY" -pubin -out "$TARGET_DIR/$BASE_FILENAME.enc"
if [ "$?" -ne 0 ]; then
    error_exit "Failed to encrypt transport key and IV"
fi

# Step 4 and 5: Encrypting the source file on the secret key and adding it the encrypted file
$OPENSSL enc -aes-256-cbc -in "$SOURCE_DIR/$BASE_FILENAME" -K "$TRANSPORT_KEY" -iv "$TRANSPORT_IV" >> "$TARGET_DIR/$BASE_FILENAME.enc"
if [ "$?" -ne 0 ]; then
    error_exit "Failed to encrypt the archive"
fi

# Step 6: Building and signing hash of encrypted file
$OPENSSL dgst -sha3-512 -binary "$TARGET_DIR/$BASE_FILENAME.enc" | $OPENSSL pkeyutl -sign -inkey "$LOCAL_PRIVATE_KEY" -out "$TARGET_DIR/$BASE_FILENAME.sig"
if [ "$?" -ne 0 ]; then
    error_exit "Failed to sign the hash"
fi

echo "Done. Please transfer $TARGET_DIR/$BASE_FILENAME.enc and $TARGET_DIR/$BASE_FILENAME.sig to other system"

Side B (reciving) scripts

#!/bin/sh

WORK_DIR="/tmp/certs"  # must exist
BASE_FILENAME="top_secret.bin"
REMOTE_PUBLIC_KEY="/tmp/certs/A_public_key.pem"
LOCAL_PRIVATE_KEY="/tmp/certs/B_private_key.pem"

OPENSSL="/usr/bin/openssl"

# Function to handle errors and exit the script
error_exit() {
    echo "Error: $1" >&2
    exit 1
}

# Expecting top_secret.bin.enc and top_secret.bin.sig to be in $WORK_DIR

# Check file sizes

if [ $(stat -f%z $WORK_DIR/$BASE_FILENAME.sig ) -ne 512 ]; then
    error_exit "$BASE_FILENAME.sig should be exactly 512 bytes long"
fi

if [ $(stat -f%z $WORK_DIR/$BASE_FILENAME.enc) -lt 512 ]; then
    error_exit "$BASE_FILENAME.enc should be at least 512 bytes long"
fi

# Verify signature
# first line calculate digest second file verify it

$OPENSSL dgst -sha3-512 -binary $WORK_DIR/$BASE_FILENAME.enc | \
$OPENSSL pkeyutl -verify -pubin -inkey $REMOTE_PUBLIC_KEY -sigfile $WORK_DIR/$BASE_FILENAME.sig
if [ "$?" -ne 0 ]; then
    error_exit "Signature verification failed"
fi

# Decrypt TRANSPORT_KEY and TRANSPORT_IV
# first 512 bytes of .enc file contains encrypted transport key. it get extracted, got piped via openssl to decrypted and then first 64 is very KEY is
TRANSPORT_KEY=$(head -c 512 $WORK_DIR/$BASE_FILENAME.enc | $OPENSSL pkeyutl -decrypt -inkey $LOCAL_PRIVATE_KEY | head -c 64)
if [ "$?" -ne 0 ]; then
    error_exit "Failed to decrypt TRANSPORT_KEY"
fi
# same as abone but last 32 byte store IV
TRANSPORT_IV=$(head -c 512 $WORK_DIR/$BASE_FILENAME.enc | $OPENSSL pkeyutl -decrypt -inkey $LOCAL_PRIVATE_KEY | tail -c 33)
if [ "$?" -ne 0 ]; then
    error_exit "Failed to decrypt TRANSPORT_IV"
fi

# Decrypt the rest of the file
# need to drop encrypted keys from start of the file
# !!! this command will make encryption keys be visible in output for PS command. If it is unaceptable risk dump keys to file first ( and do not forget to remove them)
tail -c +513 $WORK_DIR/$BASE_FILENAME.enc | $OPENSSL enc -d -aes-256-cbc -out $WORK_DIR/$BASE_FILENAME -K "$TRANSPORT_KEY" -iv "$TRANSPORT_IV"
if [ "$?" -ne 0 ]; then
    error_exit "Failed to decrypt $WORK_DIR/$BASE_FILENAME.enc"
fi

echo "Decryption successful!"

Using openssl to Secure File Transfer Between Computers