Questions and answers from HPC On-boarding May 2022, Day 0 (May 3)
Icebreaker question
What is the most important thing for you to get clarified about the UNIX shell (terminal) today?
(Add answers below with a leading
-
)?
Possibilities to add scripts from git?
How to find programs/scripts (module avail?)
how to log in to the server :)
how to track which processes are running and how to start/end processes gracefully
Using clusters via terminal and being a fast user of Linux
how SAGA works and where to find things
logging and running codes on sigma2 resources
running bioinformatics on saga
How to log in specific login node or switch to another login node?
How to run programs in ‘background’, meaning being able to work on other stuff while programs are running in background.
I use ubuntu for windows. How do I navigate to files saved from this terminal from the windows file explorer?
Intro to command line
What are the methods that you know and use to interact with the computer and open a file?
double clicking icons
…
…
Graphical file/folder navigator
Via terminal command line
Microphone and assistant
Punch cards
Are you able to open a terminal on your computer?
(vote by adding an “o” at end of each line)
“I found my terminal” : ooooo ooooooooooooo
“I did not find my terminal”:
Yes windows command prompt o
Not looking for it, only want to watch the course:
Search Git Bash and you get one
Is the use of Mobaxterm okay as an terminal?
Should be okay
On windows ssh on CMD did not work, but using putty.exe works.
SSH is available on my computer
yes: oooooooooooo
no: o
If you have a problem, please describe the error here:
…
Questions and answers
What is the best FTP client for mac, and do we need this when we work on saga?
We will go through all the necessary tools that you will need for working on Saga
I am currently taking the HPC onboarding course, I applied for my HPC account yesterday, I haven’t received my account information yet, am I able to log in to the HPC cluster?
Try
ssh saga.sigma2.no
and use your current credentials. If you manage to log in, then it is ok. Did you apply for SAGA cluster?yes I applied for SAGA, I did not have a chance to set a password yet
Then you can simply try to follow the course on your local terminal. We’ll try to fix this for you tomorrow.
Should I email someone with my username?
Let me check
Thank you
It is been taken care of now
I get permission denied..?
Which command returned this error?
Permission denied, please try again. [is the command returned]
Could you paste the command you ran in your terminal which returned the above error?
I MADE IT!!! forgot to add my username first :)
Good to hear!
I use ubuntu for windows. How do I navigate to files saved from this terminal from the windows file explorer?
I don’t use WSL, but I found this article which I hope can help
I cannot find my username in the email I got from sigma that my account is created
This is the username you used when making the application
I found it :)
Good !!
After login via SSH the command line sometimes freezes what justifies that?
This is most likely caused by your internet connection. SSH is dependent on a constant connection which sometimes might lead to these freezes. One alternative to SSH is to use
mosh
How do we reset password to saga in terminal
checking edit mode on question form-ok got it
I am getting access denied using putty
We can fix this in the break, before this, ensure that you use the correct username that you got when applying for access
works now
need to find the commands on the page
You can find the current page that we are presenting above the questions
After the wget command I get the following output:
The command "wget" is either misspelled or could not be found.
Did you do this on Saga or on your local machine?
You´re right sorry, I was not logged in now:) Thanks!
Happy to help!
I need to find the page where you are to copy-I tried this yesterday and it failed? I got it now-it is going a bit fast to try and follow both the terminal, zoom presentation and the course pages
Do you mean the target page or the source page?
You can always find the current course page right above the questions and answers on this hackmd
https://training.pages.sigma2.no/tutorials/unix-for-hpc/episodes/moving-around.html
How can I copy paste commands from windows to the terminal? It does not work for me with Ctr+shift+C or Ctr+shift+V neither without shift
CTRL+c and CTRL+V
Try the mouse, if you have one, right click + copy + paste
right click with your mouse :)
With the mouse works great, thanks!
Good to hear!
can you explain what the list command actually lists? I understand but it is organized in a weird way- can you point where the fields are, file name, size etc-looks a bit confusing if you are not used to this-ok thanks:)
The
ls
command lists the content in the current folder, in other words it shows what your terminal can “see” in the current placeCould you specify what you mean with
fields
?We will go into detail later today
“Shortly” it is this - per column: (but these notions will be explained later, I guess)
When the long listing format is used, you can see the following file information: The file type. The file permissions. Number of hard links to the file. File owner. File group. File size. Date and Time. File name.
How do I exit the VI command…I am now reading poem…but don’t know how to get out
ESC then
:wq
Could also just do
:q!
if you don’t care about saving the current file
This will write the buffer and quit.
May I suggest using the far better option emacs ?
all editors are equally good but a matter of personal preference
if you are new to terminal editors, we recommend to use
nano
and we will show this one later as this one is the easiest to get started with
please explain what is a terminal editor if it is different from the terminal itself?
It’s a text based editor, it’s written for a 80x24 text terminal. Today terminal windows can have any size, but still character based.
An editor is used for changing/editing text files
Ok so you use it “off line” like a word pad, then paste it into the terminal, right?
You run it on the remote machine directly, no cut/paste.
There are multiple ways to work with remote machines, one way is to write on your local machine and then copy to the remote machine, but one could also use the terminal editor to directly do all the work on the remote machine
does tab-complete only work for subfolders?
it works for commands in the PATH to binaries and many other things (e.g. try to type
py
+ Tab),it completes the subcommands too
where does autocomplete look for info to what to compete from-these are file names I never wrote so who knows what it can come up with!
Hitting
<TAB>
is always an option.It is a cascade: right after the prompt, it looks in the list of executable files (see quest. 14 above), then it guesses the next info with regard to the already typed command, e.g. if you type
cat
(display a file), the tab aftercat
will show you the available files in the directory you are in.
Why would you use
ls -l
it is so much easier to just dols
?Using
-l
gives you a very nice list if the folder contains many filesIt also give you additional information that is not shown without
-l
What does exactly
cp poem1.txt poem2.txt
does? Copy the files where?Next to each other in the same directory
You mean like merge in the same file or located next to each other
The latter : two copies of the same file but under two different names
Ah perfect! This makes sense, thank you
Yes, you could very well do
cp poem1.txt novel1.txt
- still the same content
What is the difference between
more
andless
?They are more or less equal :)
more
was first and is used for scrolling through a file, jumping forward through itless
was developed later, with support for scrolling back, and got the name as a joke (less is more).I usually just always use
less
less
is usually what you want since it can scroll
How do you find files in the whole computer? And to tell that does not look into subdirectories?
first question: if I want to search entire file system (don’t try this on saga as you will get errors due to permissions since you cannot search through other people’s directories and also it might take forever):
find /
you can give
find
a “depth” on how far down it should go (checkman find
and search there for “depth”; you can search in man pages by typing “/” followed by what you search for)Thank you very much, this was helpful
How is the speed so far?
(vote by adding an “o”)
too slow: ooo
too fast: oo
just right: oooo
too confusing: o
Finding things
I used grep command - but cannot exit
Your command will be
grep -R "computation failed" .
Dot at the end!![st12156@login-3.SAGA ~/exercise-moving-around/results]$ grep -R "computation failed" . grep: .: Is a directory (this is what I got - running this?)
Sorry, add
-R
as shown
I think I copied the wget part-run on saga to download the instruction files as instructed but my computer returns “command not found”
and you ran the command(s) on saga?
where are you standing when executing the command? (what is the output of
pwd
?) my home dir, I used cd .. to get back from the poem parttry these commands in this order:
hostname
cd
wget https://gitlab.sigma2.no/training/tutorials/unix-for-hpc/-/raw/master/content/downloads/finding.tar.gz
tar xzvf finding.tar.gz
does it still fail?
wget
should be available on sagathe output of
wget --version
is also command not found?one more hint! Make sure you don’t have other characters before
wget
we unfortunately never explained that the “$” is not part of the command but means “prompt” = ready for new command
it works now, I got that with $. problem with copy paste is that the enter strokes also seem to get copied if you do the whole thing at once, so it executes at once. thanks for the help
excellent. sorry for the troubles. we should make it so that copy paste does not copy the “$”
not sure i understand why this “udon” pops up? ok cool!
this was the line right after the grepped line. with
-C 1
we got some “context” around the searched lines andgrep
printed the line before and the line after the found lines.
What is the role of quotations in ‘*.jpg’ or can we skip it?
I think in this case they need to be there to delimit how far the argument goes.
In some commands you don’t need them. Example:
ls -l *.jpg
Often you need quotes when there are spaces. Example:
grep "text with spaces" somefile
Wildcards are expanded by the shell, try
echo "*"
andecho *
Creating/modifying files
which terminal are you using, because you could open another teminal in the next tab?
It is the default terminal on ubuntu which allows to open several tabs
Can you suggest one for windows? I am using WSL, Vs code , but cant open two tabs together
Look here : https://askubuntu.com/questions/1074283/multiple-terminal-windows-in-windows-ubuntu
Shift-click means to hold the Shift key while clicking on the Ubuntu icon.
I am a bit confused-is this nano version a program or a file? we opened it by opening as a file?
It is a program, like Word, for example. You start it by typing its name in the terminal. If the name is followed by a file name, this file will be opened in the editor called nano.
A program is also a file, it’s a binary file that the system can execute.
so “learn-nano” is a file created in nano?
Exactly! In Unix files may or may not have extensions (e.g.
.doc
. or similar). This is why the files may simply bear their names without any extensions. If you prefer, you may give them extensions to improve readability when dispalying directory contents.Ok-where is Nano running, did we download it somewhere along the way or is it simply already there on saga?
Yes,
nano
is already installed on Saga (and all the other clusters that NRIS support)Unix (Linux) distributions always contain a number of text editors which get installed by default when the Operating System itself is installed
Ok great-see also my question 13, I tended to regard the terminal window already as a text editor in its own right-is that wrong? After all, it translate text into instructions-ok thanks for the explanation :)
The terminal is the place where you communicate with the computer, as a dialogue board. You type smth and then “Enter” it and wait for the machine to respond. It works in a dialogue regime. The editor is a text editor, kind of static, meant to create files, which might be programs, text files etc.
I tried emacs and I cant now get out of it…??
Control X Control C
doesnt work :(
Try Control Z
doesnt work
Try control G a couple of times, then Control X control C
I hate emacs!!
It’s a huge program that can do a lot of things, it can be intimmidating the first time.
Still not out of this editor
Can you type anything there ?
yes I can type control G and then it comes up “quit” - but then when i press enter - northing happens
This is right, C-G is a quit command stopping any ongoing things, then C-X C-C coule work.
when I couple C-G with any of the following - nothing happens
Then try Control-X and Control C
I figured it our it was C-G and then C-Z
Control Z suspend emacs, it’s still running. You can start it up again with
fg
or kill it withkill %1
Super - thank you!
Open emacs again and try ESCAPE M and then type hanoi for some fun.
Substitute hanoi with doctor for a session with psychotherapist inside emacs.
15 years of UNIX experience, but I have never used
cat
with its “intended” purpose, for concatenating files :) Never too old to learn new tricks from an intro coursecool! I also learned new things. It’s great to watch somebody else type because only then I discover new ways which I never tried.
Composing commands with pipes
the dash in the file name is simply to avoid a space, right? Thanks I am hapy to not use spaces :)
Yes. We can include space in names on the terminal, but it is a bit cumbersome (
touch file\ with\ space.txt
will create a filefile with space.txt
) and easy to make mistakes (touch file with space.txt
will create three filesfile
,with
andspace.txt
)Underscore, dash and dot are commonly used to avoid space.
but I guess dot is dangerous as it also separates extension in other applications
The shell does not care much about the dot, it mostly used for us humans. Executables are often postfixed with .x , but a.out is in most cases an executable. To check the type of file try
file /bin/ls
.
find ..
takes find this up one level?Yes
What is gs in xar gs wc -l?
the string is
xargs
, no spaces in itAh okay, got it! thanks
Also don’t forget the pipe
|
, so| xargs COMMAND
they look like italics confusingly but on the rendered page they are vertical lines
For clarity perhaps better to not use italics on the pipe character-just a note (ok thanks my inexperience ;)
If you look on the rendered page it is not actually italized, that is simply HackMD trying to intelligently show that the pipe is within the backticks
it confused me too so thanks for double checking and asking
Scripting
Exercise:
use
#!/bin/bash
instead of#!/bin bash
(the space is a typo we forgot to fix)fixed on the page now
the extension “.sh” appears to be essential, I may have missed it but where does it come from? if I enter bash list it does not run the script as indicated, but if I do “bash list.sh” it does. OK it’s the integral file name I get it; this line about the “.sh file” is a bit confusing: can now run it by using our .sh file as an input for bash like this:
The file name of the script is
list.sh
, if you just writelist
it can’t be foundIt’s not really essential but best practise to inform yourself and others that it is a (bash) script and not some other text file
you could also give it extension .bash or .script or .x or no extension and it would still work but it helps communicating to humans what the content is and how it is intended to be used
note this line from the tutorial: References: This counts the number of references (hard links) to the item (file, folder, symbolic link or “shortcut”). Following the link to “hard link”, shortcuts are explained as being “soft links”. So are they included in this number?
No they are not counted in the reference counter
we learnt that “#” prevents a line from being executed. Yet, the shebang #!/bin/bash starts with “#”. Is the exclamation mark the key here, or what causes this line to actually do something?
By the way I did not clearly understood the difference between Node, cores, and CPU
A node corresponds best to what we think of a traditional computer: something that has one or more CPUs, has memory, and network capabilities.
A node contains often several CPUs. A CPU looks like we would imagine a chip.
CPUs contain typically several cores.
Feedback for the day
One thing you really liked:
fast responses on (at times my ignorant) questions (+1)
presentation format with 2 presenters (one “interviewing” the other) works well for me (+1)
great content to learn to use terminal
very good tutorials!
useful tips and commands and good pace of the course
One thing you want us to improve:
not enough time for exercises (allocate more time or less content)
I have problems keeping up… high speed, new jargon, several windows to follow
too many pages/windows or too few breaks and too high speed?
good point about the jargon
having problems to find special character on key board - and then things ran a little fast
I noticed that some of the exercises contained requirements which come up in the next session, e.g. the exercise about : find the string “computation failed” appears before the
grep
command :)we need clear warning that Copy/Paste from the training materials is risky unless we use the copy icon in the upper corner
so far, everything is fine!