November 08, 2024
Expect is a command-line tool that can help you automate interactions with other programs.
One of the main uses, and what I use it for, is to spawn a program, wait for one or more prompts, and react to them.
It may initially be a bit hard to understand, also because expect scripts are written in Tcl, which has some unusual features by today’s standards.
In this post, I’ll write a script to automate logging into a remote server, while answering any prompts (e.g. “change your password”, “accept this fingerprint”, etc.) automatically.
This will hopefully serve as a useful overview of writing expect scripts.
First off, a disclaimer: I’m doing this with a server and network I trust. In many cases, it may not be a good idea to automate some of these things, such as accepting a fingerprint.
When I connect to a host vps via SSH, I might be asked a series of questions (but possibly not all of them).
I want to answer all of them automatically, log into the server, and elevate my privileges to root level with sudo.
An interactive session with all prompts might look something like this (some lines aren’t included in the excerpt for brevity):
$ ssh vps
...
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
...
WARNING: Your password has expired.
You must change your password now and login again!
New password: password
Retype new password: password
passwd: password updated successfully
Connection to 10.89.97.83 closed.
$ ssh vps
me@vps$ sudo su
[sudo] password for me: password
root@vps:/home/me#
In the excerpt, I typed ‘yes’ and ‘password’ myself, and I’d like expect to do it for me, so that I only have to run the script, enter my password, and I’m automatically logged in as root.
autoexpectA program called autoexpect may be included with your installation of expect.
It lets you run a program (e.g. autoexpect ssh vps), it watches your interaction, and then generates an expect script based on that to automate your interaction.
autoexpect can be a good way to create an initial script, which you can then adjust.
Let’s create a file sshw, meaning ‘ssh wrapper’, and put this at the top (you might need to replace /usr/bin/expect with the path to your expect executable):
#!/usr/bin/expect -f
In the rest of this section, I’ll be appending lines to this file, unless I explicitly say otherwise.
We’d like to call the script from the command line with sshw vps, so the first step is extracting the hostname.
Command-line arguments get passed to the script in variables: argv0 for the name of the file containing the program, and the argv list for other arguments, with argc containing the number of arguments.
Here’s how we’d save the first argument to the variable hostname (an explanation follows):
if { $argc < 1 } {
send_user "No hostname provided\n"
exit 1
}
set hostname [lindex $argv 0]
# And this is the command we will run:
set command "ssh $hostname"
if works like in any other imperative language.
The condition is surrounded by { }, which is not a block, because there are no blocks in Tcl.
All values in Tcl are strings, and braces are a quoting mechanism.
A Tcl script contains lines (delimited by newline or semicolon) made up of word (delimited by a space), and quotes group words together to be treated as a single word.
The first word on a line is the routine to run, and other words on the line are arguments.
The if part could be rewritten like this, with the same effect: if "$argc < 1" "send_user \"No hostname provided\n\"; exit 1".
We’re running the if routine, and passing it arguments.
This is a key concept, so I’ll reiterate: everything is a string in Tcl.
$argc < 1 checks if there no arguments to the script – the dollar sign is variable interpolation.
If the condition succeeds (there are no arguments), we print a message with send_user (it doesn’t automatically add a newline, so we add it in the string), and exit the script with an error status of 1.
If the condition fails, we set the variable hostname to the first argument.
lindex list index gets the value at index (0-based) in list, so lindex $argv 0 gets the first argument.
The brackets [ ] embed a script to be evaluated, the result of which is then embedded at the location of the brackets, like backticks or $() in shell (so in our case, [lindex $argv 0] is replaced with the first argument).
set variable value is how you set a variable to value in Tcl (as a side note, set also returns the value).
And # is how you start a comment.
Next, we want to read a password from the user, but we don’t want to show it in the terminal. Append this to the file:
stty -echo
send_user "password: "
expect_user -re "(.*)\n"
stty echo
set pass "$expect_out(1,string)\r"
send_user "\n"
stty changes terminal modes, like the stty command.
stty -echo disables echoing (printing) of text you type to the terminal, and stty echo re-enables it; see the stty manpage for other options.
We saw send_user already – it’s a print command.
expect_user reads user input, and the -re flag enables regular expressions
The line expect_user -re "(.*)\n" can be read as: expect the user to type text matching the regular expression “(.*)\n” (any number of characters followed by a newline).
Text captured in an expect_user statement gets automatically stored in the expect_out variable.
expect_out(buffer) contains the whole buffer, and expect_out(N,string) contain the text matching a regular expression capture group N.
In our case, expect_out(1,string) contains the text matched by (.*) in the expect_user statement; i.e., the password typed by the user.
So, we save it in the variable pass (with a carriage return appended), and because we captured the user’s enter keypress, we print out a newline to move the cursor down.
Cool, we have a server and a password, let’s log in to the server (detailed breakdown of this follows):
proc login {pass command} {
global spawn_id
eval spawn $command
expect {
# Change password as needed
-ex "New password: " {
send -- "$pass"
expect -exact "Retype new password: "
send -- "$pass"
expect eof
login $pass $command
}
# Accept fingerprint as needed
-ex "Are you sure you want to continue connecting (yes/no/\[fingerprint\])? " {
send -- "yes\r"
exp_continue
}
# On a shell prompt, stop expecting
" $" {}
# Catch all - we got something we didn't expect
default {
send_user "Got some unexpected output\n"
exit 1
}
}
}
login $pass $command
With proc, we define a procedure login that takes two arguments: pass and command.
Don’t let it fool you, these are all just strings, it could be rewritten as proc login "pass command" "...".
At the bottom, we call the login procedure with the password and the command to run.
Let’s break the rest of it down part-by-part.
These are the first two lines:
global spawn_id
eval spawn $command
I’ll explain the second one first.
Remember that we set the variable command to the string "ssh $hostname".
To evaluate a value as a script, we have to use eval.
The spawn routine runs a process, setting spawn_id to a descriptor referring to that process (making it the current process).
This is what runs our ssh command.
spawn_id is what determines the “current process”, and it can be modified to control which process expect is speaking to.
Since we’re inside a procedure definition, things we set will not be accessible outside of the procedure.
However, since we want to spawn an SSH connection and be able to control it, even outside of the procedure, we have to make spawn_id global, with the keyword global.
As soon as SSH connects, we might get some prompts to answer, or just a shell prompt. This is where we have to expect and deal with different types of messages (explanation follows):
expect {
# Change password as needed
-ex "New password: " {
send -- "$pass"
expect -exact "Retype new password: "
send -- "$pass"
expect eof
login $pass $command
}
# Accept fingerprint as needed
-ex "Are you sure you want to continue connecting (yes/no/\[fingerprint\])? " {
send -- "yes\r"
exp_continue
}
# On a shell prompt, stop expecting
" $" {}
# Catch all - we got something we didn't expect
default {
send_user "Got some unexpected output\n"
exit 1
}
}
This is a big block, so let’s break it down.
expect routine worksexpect has the form expect opts1 pattern1 body1 opts2 pattern2 body2....
It waits until the output of a spawned process matches one of the patterns, then executes its corresponding body, and finishes (returning the result of the body and moving to the next line after the closing brace of expect).
Patterns are unanchored by default, so they can match in the middle of a string.
Without an option, they’re compared with Tcl’s string match, where you can use * to match any sequence of characters and ? to match a single character.
There are a few options you can pass to an expect pattern, I use -ex and -re the most.
-ex matches an exact string, so literally, without interpreting any special characters (but still unanchored).
-re matches a regular expression defined by Tcl’s regexp.
Here’s the first part:
# Change password as needed
-ex "New password: " {
send -- "$pass"
expect -exact "Retype new password: "
send -- "$pass"
expect eof
login $pass $command
}
When the SSH process asks for a new password, e.g. after a password expiry, this part gets executed.
The send command sends characters to the spawned command, and we use a double dash to make sure that if there’s a leading dash in $pass, it doesn’t get interpreted as an option.
send sends characters to the spawned process just like send_user sends characters to the screen of the user, and expect reads characters from the process just like expect_user reads characters from the user.
So with send -- "$pass", we type the password into the SSH session.
The next expect statement has no body, so it just waits until the SSH process asks for the password again.
We then send the password again.
Because SSH closes the connection after the password has been updated, we expect eof (end-of-file).
Finally, we call login again (recursively), with the same password and command.
Here’s the second part:
# Accept fingerprint as needed
-ex "Are you sure you want to continue connecting (yes/no/\[fingerprint\])? " {
send -- "yes\r"
exp_continue
}
This one’s simpler, and mostly what we’ve already seen before.
If we get asked to accept the fingerprint, we send “yes” and a carriage return to the SSH process.
Finally, we use exp_continue to re-execute the same big surrounding expect routine (because otherwise, the script would continue at the line after the closing brace of expect).
Remember, brackets ([ ]) are like backticks, they interpolate the result of a script, so we have to escape them to match them literally.
On a shell prompt, we’re done with the login process, so we can leave the expect statement:
# On a shell prompt, stop expecting
" $" {}
This is a simple pattern matching the start of my shell prompt (might be different for whoever reads this), and an empty body.
It just says that, if the output of the spawned command matches the shell prompt, it should continue the script after the expect routine.
Since it’s a simple pattern, I don’t use any options.
We could also put this as the last pattern in the expect routine, and leave out the braces:
expect {
# ...other patterns...
" $"
}
The final pattern is the default:
# Catch all - we got something we didn't expect
default {
send_user "Got some unexpected output\n"
exit 1
}
The body of default runs after a timeout, or at end-of-file.
We just send an error message to the user, and exit the script with an error status of 1.
OK, at this point we’re logged in, but we still want to become root by way of sudo.
We do that with:
send -- "sudo su\n"
expect -re "password for \[^ \]+: "
send -- "$pass"
At this point, there’s nothing new here: send the command sudo su to the process, expect it to ask for a password, and send the password.
Finally, we want to give control of the session to the user:
interact
The interact routine can let you do quite a few things (see man 1 expect), but here we only use it to get interactive control of the SSH session.
If we didn’t include that, expect would stop after a timeout and the SSH connection would be closed (since it was spawned by expect).
And there it is – we’ve automated the login process and learned a bit of expect (and Tcl).
I’m including the script in full below for those who want to copy-paste.
Here are some other useful resources:
#!/usr/bin/expect -f
if { $argc < 1 } {
send_user "No hostname provided\n"
exit 1
}
set hostname [lindex $argv 0]
set command "ssh $hostname"
stty -echo
send_user "password: "
expect_user -re "(.*)\n"
stty echo
set pass "$expect_out(1,string)\r"
send_user "\n"
proc login {pass command} {
global spawn_id
eval spawn $command
expect {
# Change password as needed
-ex "New password: " {
send -- "$pass"
expect -exact "Retype new password: "
send -- "$pass"
expect eof
login $pass $command
}
# Accept fingerprint as needed
-ex "Are you sure you want to continue connecting (yes/no/\[fingerprint\])? " {
send -- "yes\r"
exp_continue
}
# On a shell prompt, stop expecting
" $" {}
# Catch all - we got something we didn't expect
default {
send_user "Got some unexpected output\n"
exit 1
}
}
}
login $pass $command
send -- "sudo su\n"
expect -re "password for \[^ \]+: "
send -- "$pass"
interact