====== Best (my) way to use CRC clusters ======

===== Software needed =====

  * PuTTY https://www.putty.org/
  * Notepad++ https://notepad-plus-plus.org/downloads/
  * FileZilla https://filezilla-project.org/download.php?type=client

If you are using a Mac, use the following similar software
  * Terminal (already installed) to replace PuTTY.
  * TextWrangler to replace Notepad++
  * FileZilla can be equally used on a Mac.
===== Upload files using FileZilla =====
See the screen below:

{{:blog:ftp.jpg}}

  * Host: crcfe01.crc.nd.edu
  * Username: your netid
  * Password: your password
  * Port: **22**
  * Click Quickconnect
  * On the remote site (right panel), browse to the Private folder. Typically create folders and files under the Private folder.
  * On the Local site (left panel), choose the files/folders to upload and drag them to the remote site folder. 

===== Use PuTTY =====
See the screen below:

{{:blog:putty.jpg}}

=== Steps ===
  * Host name: crcfe01.crc.nd.edu
  * Port: 22
  * Connection type: ssh
  * You might save the information by clicking "Save" button.
  * Click on Open button
  * In the new window
    * log in as: Your netid
    * password: your password (note that you will not say anything when typing. just finish typing your password).
    * Then you should login.

=== Most useful commands ===
  * **pwd** : print the current directory
  * **ls** : list the file and folders
    * **ls -alh**: list all the information
  * **cd dir**  : change to a new directory (dir)
    * **cd ..** : go to last level of directory
    * **cd** : go the the initial directory you log in
  * **mkdir dir** : create a new directory called dir
  * **rm file1** : delete a file
  * **rm -r dir** : delete a directory
  * **nano** : a text editor.
  * chmod g+s dirname: file created belongs to the group.
  * setfacl -d -m g::rwx mathfeud  : set the permission for the group
===== Use Notepad++ =====

  * Use the ftp of Notepad++ to edit files directly.
  * Show the ftp windows by the menu: Plugins -- nppFTP -- show NppFTP window (see the screen below)

{{:blog:notepadpp.jpg}}

  * On the ftp window, click the settings button and then Profile settings (see the screen below)

{{:blog:notepadpp3.jpg}}

  * Then in the new window
    * Click "Add new" on the bottom left corner
    * Give a name, e.g., CRC
    * Then input the information required
      * Hostname: crcfe01.crc.nd.edu
      * Connection type: SFTP
      * Port: 22
      * Username: your netid
      * Password: leave empty and check "Ask for password"
      * Initial remote directory: the output of pwd commend in putty.
      * close the window

{{:blog:notepadpp2.jpg}}

  * Connect to ftp. Click the connection button and choose CRC as shown below.
    * Input your password when asked.
    * Your files will be listed.
    * To edit a file, click on it and it will open in Notepad++.
    * You can also right click to create/delete a directory, create/delete a file.
    * After editing, save the file directly using Notepad++.

{{:blog:notepadpp4.jpg}}

{{:blog:notepadpp5.jpg}}

===== Submit a job (SGE) =====
For more information, go to http://wiki.crc.nd.edu/wiki/index.php/Submitting_Batch/SGE_jobs

  * Log in using Putty
  * Create a new folder -test- within Putty using command <code>mkdir test</code> or do that within Notepad++ using the FTP plugin
  * Go the new folder test.
  * Create a test R file with the following content. You can use Notepad++. If using Putty, you can use the <code>nano test.R</code> The content of the R file is <code>rnorm(100)</code>
    * When using **nano**, Ctrl+O saves the file and Ctrl+X quits the edits.
  * Create another file (a submission file) called **test.sh**. In the file, include the following information (change the email address to yours) <code>
#!/bin/csh
#$ -M xxxx@nd.edu
#$ -m abe
#$ -r y

module load R

R CMD BATCH test.R

</code>
  * To submit a job, in Putty (has to use Putty or other SSH client), type the command <code>qsub test.sh</code> Your job should be submitted now.
  * To check the status of your job, use <code>qstat -u netid</code> Your job should already finished by now, so you won't see anything for this simply example.
  * You can list all the files using command **ls**. You will see a file called test.Rout. Any error can also be found in it.
  * There will also be a file called test.sh.oxxxx (xxxx is a number). If no error, the size of it should be 0 typically.
  * If there is error in your job, you will see a file like test.sh.exxxx.
  * To use SAS, change **module load R** to **module load sas**. All available module can be found using **module avail**
  * **If the test job works fine, you can start your regular jobs!**

===== Submit an array job (SGE) =====
Sometimes, it might be convenient to split a big jobs to smaller ones. Preferably, each small job runs within its own folder. For example, one might have a sas job in the following folders

  * sasjob1
  * sasjob2
  * sasjob3
  * sasjob4

Within each folder, there is a sas file called runsas.sas. To submit such jobs, use the batch file with contents below 
<code>
#!/bin/csh
#$ -M zzhang4@nd.edu
#$ -m abe
#$ -r y
#$ -t 1-4

module load sas
cd sasjob$SGE_TASK_ID

sas runsas.sas
</code>

If one prefers to put all small jobs in one folder like
  * sasjob1.sas
  * sasjob2.sas
  * sasjob3.sas
  * sasjob4.sas

Then the following script file can be used
<code>
#!/bin/csh
#$ -M zzhang4@nd.edu
#$ -m abe
#$ -r y
#$ -t 1-4

module load sas

sas sasjob$SGE_TASK_ID.sas
</code>

==== Debug ====
To reset the CRC environment, use

<code>/opt/crc/usr/local/bin/crc_setup</code>

{{tag>}}

~~LINKBACK~~
~~DISCUSSION~~