Table of Contents
Best (my) way to use CRC clusters
Software needed
- PuTTY https://www.putty.org/
- Notepad++ https://notepad-plus-plus.org/downloads/
If you are using a Mac, use the following similar software
- Terminal (already installed) to replace PuTTY.
- TextWrangler to replace Notepad++
- FileZilla can be equally used on a Mac.
Upload files using FileZilla
See the screen below:
- Host: crcfe01.crc.nd.edu
- Username: your netid
- Password: your password
- Port: 22
- Click Quickconnect
- On the remote site (right panel), browse to the Private folder. Typically create folders and files under the Private folder.
- On the Local site (left panel), choose the files/folders to upload and drag them to the remote site folder.
Use PuTTY
Steps
- Host name: crcfe01.crc.nd.edu
- Port: 22
- Connection type: ssh
- You might save the information by clicking “Save” button.
- Click on Open button
- In the new window
- log in as: Your netid
- password: your password (note that you will not say anything when typing. just finish typing your password).
- Then you should login.
Most useful commands
- pwd : print the current directory
- ls : list the file and folders
- ls -alh: list all the information
- cd dir : change to a new directory (dir)
- cd .. : go to last level of directory
- cd : go the the initial directory you log in
- mkdir dir : create a new directory called dir
- rm file1 : delete a file
- rm -r dir : delete a directory
- nano : a text editor.
- chmod g+s dirname: file created belongs to the group.
- setfacl -d -m g::rwx mathfeud : set the permission for the group
Use Notepad++
- Use the ftp of Notepad++ to edit files directly.
- Show the ftp windows by the menu: Plugins – nppFTP – show NppFTP window (see the screen below)
- On the ftp window, click the settings button and then Profile settings (see the screen below)
- Then in the new window
- Click “Add new” on the bottom left corner
- Give a name, e.g., CRC
- Then input the information required
- Hostname: crcfe01.crc.nd.edu
- Connection type: SFTP
- Port: 22
- Username: your netid
- Password: leave empty and check “Ask for password”
- Initial remote directory: the output of pwd commend in putty.
- close the window
- Connect to ftp. Click the connection button and choose CRC as shown below.
- Input your password when asked.
- Your files will be listed.
- To edit a file, click on it and it will open in Notepad++.
- You can also right click to create/delete a directory, create/delete a file.
- After editing, save the file directly using Notepad++.
Submit a job (SGE)
For more information, go to http://wiki.crc.nd.edu/wiki/index.php/Submitting_Batch/SGE_jobs
- Log in using Putty
- Create a new folder -test- within Putty using command
mkdir test
or do that within Notepad++ using the FTP plugin
- Go the new folder test.
- Create a test R file with the following content. You can use Notepad++. If using Putty, you can use the
nano test.R
The content of the R file is
rnorm(100)
- When using nano, Ctrl+O saves the file and Ctrl+X quits the edits.
- Create another file (a submission file) called test.sh. In the file, include the following information (change the email address to yours)
#!/bin/csh #$ -M xxxx@nd.edu #$ -m abe #$ -r y module load R R CMD BATCH test.R
- To submit a job, in Putty (has to use Putty or other SSH client), type the command
qsub test.sh
Your job should be submitted now.
- To check the status of your job, use
qstat -u netid
Your job should already finished by now, so you won't see anything for this simply example.
- You can list all the files using command ls. You will see a file called test.Rout. Any error can also be found in it.
- There will also be a file called test.sh.oxxxx (xxxx is a number). If no error, the size of it should be 0 typically.
- If there is error in your job, you will see a file like test.sh.exxxx.
- To use SAS, change module load R to module load sas. All available module can be found using module avail
- If the test job works fine, you can start your regular jobs!
Submit an array job (SGE)
Sometimes, it might be convenient to split a big jobs to smaller ones. Preferably, each small job runs within its own folder. For example, one might have a sas job in the following folders
- sasjob1
- sasjob2
- sasjob3
- sasjob4
Within each folder, there is a sas file called runsas.sas. To submit such jobs, use the batch file with contents below
#!/bin/csh #$ -M zzhang4@nd.edu #$ -m abe #$ -r y #$ -t 1-4 module load sas cd sasjob$SGE_TASK_ID sas runsas.sas
If one prefers to put all small jobs in one folder like
- sasjob1.sas
- sasjob2.sas
- sasjob3.sas
- sasjob4.sas
Then the following script file can be used
#!/bin/csh #$ -M zzhang4@nd.edu #$ -m abe #$ -r y #$ -t 1-4 module load sas sas sasjob$SGE_TASK_ID.sas
Debug
To reset the CRC environment, use
/opt/crc/usr/local/bin/crc_setup