SshBuildTestExecution

Executing long-running build and test scripts on a remote machine via ssh can be error-prone if not done carefully. For example, connection timeouts may cause the job to be killed when the controlling terminal dies, and some programs don't run properly in the background without job control disabled.

nohup command

Here is an approach which is known to work well with Eigenbase projects. It shows user jvs starting from machine apoptosis, ssh'ing to machine tikki01-red, launching a job via the nohup command, logging out, and then later logging back in to check on the progress:

jvs@apoptosis:~$ ssh tikki01-red
[jvs@tikki01-red ~]$ cd open/luciddb
[jvs@tikki01-red luciddb]$ rm -f nohup.out 
[jvs@tikki01-red luciddb]$ set +m
[jvs@tikki01-red luciddb]$ nohup ./initBuild.sh --with-tests &
[1] 17316
[jvs@tikki01-red luciddb]$ nohup: appending output to `nohup.out'
[jvs@tikki01-red luciddb]$ exit
logout
Connection to tikki01-red closed.
...
jvs@apoptosis:~$ ssh tikki01-red
[jvs@tikki01-red ~]$ cd open/luciddb
[jvs@tikki01-red luciddb]$ tail -f nohup.out 
...
# Configure Fennel
cd "$SAVE_PWD"
rm -rf autom4te.cache
autoreconf --force --install
...

Important points:

• Forgetting to execute set +m will cause hangs.
• Forgetting to specify the ampersand on the end will cause trouble because after set +m, you no longer have job control to move the job into the background.
• Forgetting to use nohup will cause the job to die or hang when you exit the shell (or when your ssh connection gets timed out, or your network connection drops, or...)
• After launching a job, never run tail -f immediately in the same shell. This is known to cause hangs in some circumstances. Instead, use the approach above (log out first, then log back in and run tail -f).

screen command

The Linux screen command is a much nicer alternative; it allows you to create a named interactive session, detach from it, and re-attach to it later by name.

basic starting, detaching, re-attaching, exiting

To start a new screen session on a remote machine:

ogothberg@tawnyfur:~$ ssh tikki01-red
[ogothberg@tikki01-red ~]$ screen

Or if you want to make sure the screen session is a login shell (i.e. with .bash_profile executed):

ogothberg@tawnyfur:~$ ssh tikki01-red
[ogothberg@tikki01-red ~]$ screen bash --login

This gives you a fresh virtual terminal for whatever you want to execute. First you might want to log activities in this terminal, press:

   control-a H     (case sensitive)

The log file is "screenlog.n", where n is the "window number". Usually this is 0.

Now you can just execute a command as you would in any other terminal:

(inside screen session)
[ogothberg@tikki01-red ~]$ cd open/luciddb/
[ogothberg@tikki01-red luciddb]$ ./initBuild.sh --with-tests
...building...

To detach the screen session while your command is running, and have it run in the background on the remote server, press:

control-A, control-D

...which takes you back to your original remote SSH session:

[detached]
[ogothberg@tikki01-red ~]$ 

Now you can monitor the progress of the screened command via

   tail -f screenlog.0

It is now safe to log out:

[ogothberg@tikki01-red ~]$ exit
Connection to tikki01-red closed.
ogothberg@tawnyfur:~$

Now the screen session is running untouched on your remote server. To check up on it, log back into the server again and either check the log file "screenlog.n", or re-attach it by executing "screen -r -d":

.
ogothberg@tawnyfur:~$ ssh tikki01-red
[ogothberg@tikki01-red ~]$ screen -r -d

...which puts you back into your screen session:

(inside screen session)
...
BUILD SUCCESSFUL

[ogothberg@tikki01-red luciddb]$ 

If you want to scroll up in the screen buffer do the following command. This will allow you to use the arrow keys to scroll up and down the buffer or vi/emacs-like commands to search through the buffer. To get out of this mode just press any key, such as "q"

Control-a ESC 

To send Control-a enter

Control-a a

To kill the screen session, just type "exit" from within the session:

(inside screen session)
[ogothberg@tikki01-red ~]$ exit

Which terminates the screen session and brings you back to your SSH session again. You can also use Control-d as usual.

Afterwards, the command trace in your terminal may look something like this:

ogothberg@tawnyfur:~$ ssh tikki01-red
[ogothberg@tikki01-red ~]$ screen
[detached]
[ogothberg@tikki01-red ~]$ exit
Connection to tikki01-red closed.
ogothberg@tawnyfur:~$ ssh tikki01-red
[ogothberg@tikki01-red ~]$ screen -r -d
[screen is terminating]
[ogothberg@tikki01-red ~]$

basic configuration

You can control the command character (defaults to Control-a) from the command line:

screen -e ^Jj

or by creating a file called .screenrc in your home directory containing:

escape ^Jj

In either case ^Jj means use Control-j as the command character and generate a literal Control-j when Control-j j is entered. The literal command character need not be related to the command character. In particular these are all valid

screen -e ^Ja    # "Control-j a" generates Control-j

screen -e ^A^A   # "Control-a Control-a" generates Control-a, but masks the window toggle command

screen -e xx     # makes typing an x kind of hard