{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Linux command line introduction" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Super fast introduction to UNIX/Linux...*command line*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## UNIX/Linux/command line" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The exclamation mark (!) or percentage (%) at the beginning of the lines is specific for Jupyter notebook. Don't use it in a normal command line!!!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## On purpose, we will use some commands we will not comment on. It's up to you to search and figure out what they do." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example of system commands you can try\n", "Some commands you can try to see what happens in the command line and how it looks like." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "guest03\r\n" ] } ], "source": [ "!# name of the user\n", "!whoami" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wolf03\r\n" ] } ], "source": [ "!# name of the computer\n", "!hostname" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "HOSTNAME(1) Linux Programmer's Manual HOSTNAME(1)\r\n", "\r\n", "N\bNA\bAM\bME\bE\r\n", " hostname - show or set the system's host name\r\n", " domainname - show or set the system's NIS/YP domain name\r\n", " ypdomainname - show or set the system's NIS/YP domain name\r\n", " nisdomainname - show or set the system's NIS/YP domain name\r\n", " dnsdomainname - show the system's DNS domain name\r\n", "\r\n", "S\bSY\bYN\bNO\bOP\bPS\bSI\bIS\bS\r\n", " h\bho\bos\bst\btn\bna\bam\bme\be [-\b-a\ba|\b|-\b--\b-a\bal\bli\bia\bas\bs] [-\b-d\bd|\b|-\b--\b-d\bdo\bom\bma\bai\bin\bn] [-\b-f\bf|\b|-\b--\b-f\bfq\bqd\bdn\bn|\b|-\b--\b-l\blo\bon\bng\bg] [-\b-A\bA|\b|-\b--\b-a\bal\bll\bl-\b-f\bfq\bqd\bdn\bns\bs]\r\n", " [-\b-i\bi|\b|-\b--\b-i\bip\bp-\b-a\bad\bdd\bdr\bre\bes\bss\bs] [-\b-I\bI|\b|-\b--\b-a\bal\bll\bl-\b-i\bip\bp-\b-a\bad\bdd\bdr\bre\bes\bss\bse\bes\bs] [-\b-s\bs|\b|-\b--\b-s\bsh\bho\bor\brt\bt] [-\b-y\by|\b|-\b--\b-y\byp\bp|\b|-\b--\b-n\bni\bis\bs]\r\n", " h\bho\bos\bst\btn\bna\bam\bme\be [-\b-b\bb|\b|-\b--\b-b\bbo\boo\bot\bt] [-\b-F\bF|\b|-\b--\b-f\bfi\bil\ble\be f\bfi\bil\ble\ben\bna\bam\bme\be] [h\bho\bos\bst\btn\bna\bam\bme\be]\r\n", " h\bho\bos\bst\btn\bna\bam\bme\be [-\b-h\bh|\b|-\b--\b-h\bhe\bel\blp\bp] [-\b-V\bV|\b|-\b--\b-v\bve\ber\brs\bsi\bio\bon\bn]\r\n", "\r\n", " d\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be [n\bni\bis\bsd\bdo\bom\bma\bai\bin\bn] [-\b-F\bF f\bfi\bil\ble\be]\r\n", " y\byp\bpd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be [n\bni\bis\bsd\bdo\bom\bma\bai\bin\bn] [-\b-F\bF f\bfi\bil\ble\be]\r\n", " n\bni\bis\bsd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be [n\bni\bis\bsd\bdo\bom\bma\bai\bin\bn] [-\b-F\bF f\bfi\bil\ble\be]\r\n", "\r\n", " d\bdn\bns\bsd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be\r\n", "\r\n", "D\bDE\bES\bSC\bCR\bRI\bIP\bPT\bTI\bIO\bON\bN\r\n", " H\bHo\bos\bst\btn\bna\bam\bme\be is used to display the system's DNS name, and to display or\r\n", " set its hostname or NIS domain name.\r\n", "\r\n", " G\bGE\bET\bT N\bNA\bAM\bME\bE\r\n", " When called without any arguments, the program displays the current\r\n", " names:\r\n", "\r\n", " h\bho\bos\bst\btn\bna\bam\bme\be will print the name of the system as returned by the g\bge\bet\bth\bho\bos\bst\bt‐\b‐\r\n", " n\bna\bam\bme\be(2) function.\r\n", "\r\n", " d\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be will print the NIS domainname of the system. d\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be\r\n", " uses the g\bge\bet\bth\bho\bos\bst\btn\bna\bam\bme\be(2) function, while y\byp\bpd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be and n\bni\bis\bsd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be\r\n", " use the y\byp\bp_\b_g\bge\bet\bt_\b_d\bde\bef\bfa\bau\bul\blt\bt_\b_d\bdo\bom\bma\bai\bin\bn(3).\r\n", "\r\n", " d\bdn\bns\bsd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be will print the domain part of the FQDN (Fully Qualified\r\n", " Domain Name). The complete FQDN of the system is returned with h\bho\bos\bst\btn\bna\bam\bme\be\r\n", " -\b--\b-f\bfq\bqd\bdn\bn (but see the warnings in section T\bTH\bHE\bE F\bFQ\bQD\bDN\bN below).\r\n", "\r\n", " S\bSE\bET\bT N\bNA\bAM\bME\bE\r\n", " When called with one argument or with the -\b--\b-f\bfi\bil\ble\be option, the commands\r\n", " set the host name or the NIS/YP domain name. h\bho\bos\bst\btn\bna\bam\bme\be uses the\r\n", " s\bse\bet\bth\bho\bos\bst\btn\bna\bam\bme\be(2) function, while all of the three d\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be,\b, y\byp\bpd\bdo\bom\bma\bai\bin\bn‐\b‐\r\n", " n\bna\bam\bme\be and n\bni\bis\bsd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be use s\bse\bet\btd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be(2). Note, that this is effec‐\r\n", " tive only until the next reboot. Edit /etc/hostname for permanent\r\n", " change.\r\n", "\r\n", " Note, that only the super-user can change the names.\r\n", "\r\n", " It is not possible to set the FQDN or the DNS domain name with the d\bdn\bns\bs‐\b‐\r\n", " d\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be command (see T\bTH\bHE\bE F\bFQ\bQD\bDN\bN below).\r\n", "\r\n", " The host name is usually set once at system startup in\r\n", " _\b/_\be_\bt_\bc_\b/_\bi_\bn_\bi_\bt_\b._\bd_\b/_\bh_\bo_\bs_\bt_\bn_\ba_\bm_\be_\b._\bs_\bh (normally by reading the contents of a file\r\n", " which contains the host name, e.g. _\b/_\be_\bt_\bc_\b/_\bh_\bo_\bs_\bt_\bn_\ba_\bm_\be).\r\n", "\r\n", " T\bTH\bHE\bE F\bFQ\bQD\bDN\bN\r\n", " The FQDN (Fully Qualified Domain Name) of the system is the name that\r\n", " the r\bre\bes\bso\bol\blv\bve\ber\br(3) returns for the host name, such as, _\bu_\br_\bs_\bu_\bl_\ba_\b._\be_\bx_\ba_\bm_\bp_\bl_\be_\b._\bc_\bo_\bm.\r\n", " It is usually the hostname followed by the DNS domain name (the part\r\n", " after the first dot). You can check the FQDN using h\bho\bos\bst\btn\bna\bam\bme\be -\b--\b-f\bfq\bqd\bdn\bn or\r\n", " the domain name using d\bdn\bns\bsd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be.\r\n", "\r\n", " You cannot change the FQDN with h\bho\bos\bst\btn\bna\bam\bme\be or d\bdn\bns\bsd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be.\r\n", "\r\n", " The recommended method of setting the FQDN is to make the hostname be\r\n", " an alias for the fully qualified name using _\b/_\be_\bt_\bc_\b/_\bh_\bo_\bs_\bt_\bs, DNS, or NIS.\r\n", " For example, if the hostname was \"ursula\", one might have a line in\r\n", " _\b/_\be_\bt_\bc_\b/_\bh_\bo_\bs_\bt_\bs which reads\r\n", "\r\n", " 127.0.1.1 ursula.example.com ursula\r\n", "\r\n", " Technically: The FQDN is the name g\bge\bet\bta\bad\bdd\bdr\bri\bin\bnf\bfo\bo(3) returns for the host\r\n", " name returned by g\bge\bet\bth\bho\bos\bst\btn\bna\bam\bme\be(2). The DNS domain name is the part after\r\n", " the first dot.\r\n", "\r\n", " Therefore it depends on the configuration of the resolver (usually in\r\n", " _\b/_\be_\bt_\bc_\b/_\bh_\bo_\bs_\bt_\b._\bc_\bo_\bn_\bf) how you can change it. Usually the hosts file is parsed\r\n", " before DNS or NIS, so it is most common to change the FQDN in\r\n", " _\b/_\be_\bt_\bc_\b/_\bh_\bo_\bs_\bt_\bs.\r\n", "\r\n", " If a machine has multiple network interfaces/addresses or is used in a\r\n", " mobile environment, then it may either have multiple FQDNs/domain names\r\n", " or none at all. Therefore avoid using h\bho\bos\bst\btn\bna\bam\bme\be -\b--\b-f\bfq\bqd\bdn\bn, h\bho\bos\bst\btn\bna\bam\bme\be\r\n", " -\b--\b-d\bdo\bom\bma\bai\bin\bn and d\bdn\bns\bsd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be. h\bho\bos\bst\btn\bna\bam\bme\be -\b--\b-i\bip\bp-\b-a\bad\bdd\bdr\bre\bes\bss\bs is subject to the\r\n", " same limitations so it should be avoided as well.\r\n", "\r\n", "O\bOP\bPT\bTI\bIO\bON\bNS\bS\r\n", " _\b-_\ba_\b, _\b-_\b-_\ba_\bl_\bi_\ba_\bs\r\n", " Display the alias name of the host (if used). This option is\r\n", " deprecated and should not be used anymore.\r\n", "\r\n", " _\b-_\bA_\b, _\b-_\b-_\ba_\bl_\bl_\b-_\bf_\bq_\bd_\bn_\bs\r\n", " Displays all FQDNs of the machine. This option enumerates all\r\n", " configured network addresses on all configured network inter‐\r\n", " faces, and translates them to DNS domain names. Addresses that\r\n", " cannot be translated (i.e. because they do not have an appropri‐\r\n", " ate reverse IP entry) are skipped. Note that different addresses\r\n", " may resolve to the same name, therefore the output may contain\r\n", " duplicate entries. Do not make any assumptions about the order\r\n", " of the output.\r\n", "\r\n", " _\b-_\bb_\b, _\b-_\b-_\bb_\bo_\bo_\bt\r\n", " Always set a hostname; this allows the file specified by _\b-_\bF to\r\n", " be non-existant or empty, in which case the default hostname\r\n", " _\bl_\bo_\bc_\ba_\bl_\bh_\bo_\bs_\bt will be used if none is yet set.\r\n", "\r\n", " _\b-_\bd_\b, _\b-_\b-_\bd_\bo_\bm_\ba_\bi_\bn\r\n", " Display the name of the DNS domain. Don't use the command\r\n", " d\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be to get the DNS domain name because it will show the\r\n", " NIS domain name and not the DNS domain name. Use d\bdn\bns\bsd\bdo\bom\bma\bai\bin\bnn\bna\bam\bme\be\r\n", " instead. See the warnings in section T\bTH\bHE\bE F\bFQ\bQD\bDN\bN above, and avoid\r\n", " using this option.\r\n", "\r\n", " _\b-_\bf_\b, _\b-_\b-_\bf_\bq_\bd_\bn_\b, _\b-_\b-_\bl_\bo_\bn_\bg\r\n", " Display the FQDN (Fully Qualified Domain Name). A FQDN consists\r\n", " of a short host name and the DNS domain name. Unless you are\r\n", " using bind or NIS for host lookups you can change the FQDN and\r\n", " the DNS domain name (which is part of the FQDN) in the\r\n", " _\b/_\be_\bt_\bc_\b/_\bh_\bo_\bs_\bt_\bs file. See the warnings in section T\bTH\bHE\bE F\bFQ\bQD\bDN\bN above und\r\n", " use h\bho\bos\bst\btn\bna\bam\bme\be -\b--\b-a\bal\bll\bl-\b-f\bfq\bqd\bdn\bns\bs instead wherever possible.\r\n", "\r\n", " _\b-_\bF_\b, _\b-_\b-_\bf_\bi_\bl_\be _\bf_\bi_\bl_\be_\bn_\ba_\bm_\be\r\n", " Read the host name from the specified file. Comments (lines\r\n", " starting with a `#') are ignored.\r\n", "\r\n", " _\b-_\bi_\b, _\b-_\b-_\bi_\bp_\b-_\ba_\bd_\bd_\br_\be_\bs_\bs\r\n", " Display the network address(es) of the host name. Note that this\r\n", " works only if the host name can be resolved. Avoid using this\r\n", " option; use h\bho\bos\bst\btn\bna\bam\bme\be -\b--\b-a\bal\bll\bl-\b-i\bip\bp-\b-a\bad\bdd\bdr\bre\bes\bss\bse\bes\bs instead.\r\n", "\r\n", " _\b-_\bI_\b, _\b-_\b-_\ba_\bl_\bl_\b-_\bi_\bp_\b-_\ba_\bd_\bd_\br_\be_\bs_\bs_\be_\bs\r\n", " Display all network addresses of the host. This option enumer‐\r\n", " ates all configured addresses on all network interfaces. The\r\n", " loopback interface and IPv6 link-local addresses are omitted.\r\n", " Contrary to option _\b-_\bi, this option does not depend on name reso‐\r\n", " lution. Do not make any assumptions about the order of the out‐\r\n", " put.\r\n", "\r\n", " _\b-_\bs_\b, _\b-_\b-_\bs_\bh_\bo_\br_\bt\r\n", " Display the short host name. This is the host name cut at the\r\n", " first dot.\r\n", "\r\n", " _\b-_\bV_\b, _\b-_\b-_\bv_\be_\br_\bs_\bi_\bo_\bn\r\n", " Print version information on standard output and exit success‐\r\n", " fully.\r\n", "\r\n", " _\b-_\by_\b, _\b-_\b-_\by_\bp_\b, _\b-_\b-_\bn_\bi_\bs\r\n", " Display the NIS domain name. If a parameter is given (or -\b--\b-f\bfi\bil\ble\be\r\n", " n\bna\bam\bme\be ) then root can also set a new NIS domain.\r\n", "\r\n", " _\b-_\bh_\b, _\b-_\b-_\bh_\be_\bl_\bp\r\n", " Print a usage message and exit.\r\n", "\r\n", "N\bNO\bOT\bTE\bES\bS\r\n", " The address families h\bho\bos\bst\btn\bna\bam\bme\be tries when looking up the FQDN, aliases\r\n", " and network addresses of the host are determined by the configuration\r\n", " of your resolver. For instance, on GNU Libc systems, the resolver can\r\n", " be instructed to try IPv6 lookups first by using the i\bin\bne\bet\bt6\b6 option in\r\n", " /\b/e\bet\btc\bc/\b/r\bre\bes\bso\bol\blv\bv.\b.c\bco\bon\bnf\bf.\r\n", "\r\n", "F\bFI\bIL\bLE\bES\bS\r\n", " /\b/e\bet\btc\bc/\b/h\bho\bos\bst\btn\bna\bam\bme\be Historically this file was supposed to only contain the\r\n", " hostname and not the full canonical FQDN. Nowadays most software is\r\n", " able to cope with a full FQDN here. This file is read at boot time by\r\n", " the system initialization scripts to set the hostname.\r\n", "\r\n", " /\b/e\bet\btc\bc/\b/h\bho\bos\bst\bts\bs Usually, this is where one sets the domain name by aliasing\r\n", " the host name to the FQDN.\r\n", "\r\n", "A\bAU\bUT\bTH\bHO\bOR\bRS\bS\r\n", " Peter Tobias, \r\n", " Bernd Eckenfels, (NIS and manpage).\r\n", " Michael Meskes, \r\n", "\r\n", "net-tools 2009-09-16 HOSTNAME(1)\r\n" ] } ], "source": [ "!# manual for the command; press q to quit\n", "!man hostname" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Paths and directories\n", "Everytime you work in the command line you will have to work with paths and directories (folders in Windows). You can go from a directory to a directory without opening the graphical browser - much faster! You can also copy files and directories." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/guest03/Desktop\r\n" ] } ], "source": [ "!# what is the path to the directory where I am now; Path of Working Directory\n", "!pwd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Be careful, commands are case-sensitive!" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/bin/sh: 1: PWD: not found\r\n" ] } ], "source": [ "!# doesn't do anything and possibly throws an error that it doesn't know the command (simply it doesn't exists unless there is a different command with very similar name - might happen)\n", "!PWD" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can try using the **absolute path** and **relative path**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We would like to go one directory up from our current 'location'." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/guest03/Desktop\n", "linux_intro.ipynb\n" ] } ], "source": [ "!# where we are now\n", "!pwd\n", "!# what is the content of the current directory\n", "!ls" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/guest03/Desktop\n", "/home/guest03/Desktop\n", "linux_intro.ipynb\n" ] } ], "source": [ "!# go one directory up using the absolute path - it goes from the root of the computer - from the very 'starts'\n", "%cd /home/guest03/Desktop\n", "!pwd\n", "!ls" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/guest03\n", "/home/guest03\n", "Desktop Documents Downloads Music Pictures\tPublic\tTemplates Videos\n" ] } ], "source": [ "# we can also use a relative path - it goes relative to the current directory\n", "%cd ../\n", "!pwd\n", "!ls" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also list a directory without even going there." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "linux_intro.ipynb\r\n" ] } ], "source": [ "!ls /home/guest03/Desktop" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is also a special sign `~` (tilde) that marks your **home** directory. Home is basicaly your default directory." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/guest03\n", "/home/guest03\r\n" ] } ], "source": [ "!# using a special sign to go to your home directory\n", "%cd ~\n", "!pwd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Operation with directories and files - create, copy, move, etc.\n", "Of course, using the command line you can copy files from one directory to another, create new ones, delete, rename, and everything else you can imagine. " ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Desktop Downloads newdir\tPublic\t Videos\r\n", "Documents Music Pictures\tTemplates\r\n" ] } ], "source": [ "!# first, create an empty, new directory\n", "!mkdir newdir\n", "!# do we really have it there?\n", "!ls" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/guest03/newdir\n", "newfile\n" ] } ], "source": [ "!# go to the new directory (relative path) and create a new empty file there\n", "%cd newdir/\n", "!touch newfile\n", "!ls" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we can move the file using again both relative and absolute paths. " ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Bi5444_Analysis_of_sequencing_data_lesson_02_MMraz.pdf\tnewfile\r\n" ] } ], "source": [ "!# move the file to a different directory using the absolute path\n", "!mv newfile /home/guest03/Downloads\n", "!ls /home/guest03/Downloads\n", "!ls" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "newfile\n", "Bi5444_Analysis_of_sequencing_data_lesson_02_MMraz.pdf\n" ] } ], "source": [ "!# and now we can move it back in here using the relative path\n", "!mv /home/guest03/Downloads/newfile .\n", "!ls\n", "!ls /home/guest03/Downloads" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also copy and rename the file." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "newfile\n", "anotherfile newfile\n" ] } ], "source": [ "!# copy (not move) the file\n", "!cp /home/guest03/newdir/newfile ../Pictures/\n", "!ls \n", "!ls ../Pictures/" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "anotherfile newfile\r\n" ] } ], "source": [ "!# copy the file and rename it\n", "!cp ../newdir/newfile ../Pictures/anotherfile\n", "!ls ../Pictures/" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "anotherfile movedfile\tnewfile\r\n" ] } ], "source": [ "!# in a simillar manner we can move the file\n", "!mv newfile ../Pictures/movedfile\n", "!ls\n", "!ls ../Pictures/" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/guest03/Pictures\n", "anotherfile movedfile\tnewfile\n", "movedfile newfile renamedfile\n" ] } ], "source": [ "!# renaming a file in the command line is the same as moving the file\n", "%cd ../Pictures/\n", "!ls\n", "!mv anotherfile renamedfile\n", "!ls" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "movedfile newfile\r\n" ] } ], "source": [ "!# and we can remove a file\n", "!rm renamedfile\n", "!ls" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can do the same things with directories as with files but we need some minor changes. " ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "movedfile newdir newfile\r\n" ] } ], "source": [ "!# make new directory\n", "!mkdir newdir\n", "!ls" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cp: omitting directory 'newdir'\n", "Desktop Downloads newdir\tPublic\t Videos\n", "Documents Music Pictures\tTemplates\n" ] } ], "source": [ "!# copy directory - have to use RECURSIVE copying\n", "!cp newdir ../\n", "!cp -r newdir ../\n", "!ls ../" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "movedfile newfile\n", "Bi5444_Analysis_of_sequencing_data_lesson_02_MMraz.pdf\tnewdir\n" ] } ], "source": [ "!# move the directory\n", "!mv newdir /home/guest03/Downloads/\n", "!ls\n", "!ls /home/guest03/Downloads/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Please note the \"/\" character at the end of the command. If you want to move something **IN** in directory, it better to use the \"/\" at the end. If you don't use it the command line, in some cases, might think you want to move and **RENAME** the moved directory and replace the target directory." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Bi5444_Analysis_of_sequencing_data_lesson_02_MMraz.pdf\r\n" ] } ], "source": [ "!# we can remove an empty directory\n", "!rmdir /home/guest03/Downloads/newdir\n", "!ls /home/guest03/Downloads" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But if the directory is not empy, simple rmdir won't work. We have to use **RECURSIVE** delete. But be careful, command line doesn't ask \"Are you sure?\" but does it right away. " ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "bagr movedfile newfile\n", "rmdir: failed to remove 'bagr': Directory not empty\n", "movedfile newfile\n" ] } ], "source": [ "!# remove non-empty directory\n", "!mkdir bagr\n", "!touch bagr/file\n", "!ls\n", "!rmdir bagr\n", "!rm -r bagr\n", "!ls" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Special making directories - more than one \"level\". We can also create a long \"list\" of directories which are in a sequence one after the other. For this, we have to create all the **parent** directories of the last listed directory = all directories which are \"above\" the last one." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "mkdir: cannot create directory ‘/home/guest03/Desktop/testdir1/testdir2’: No such file or directory\n", "testdir2\n" ] } ], "source": [ "!# make all the directories even if some of them doesn't exist\n", "!mkdir /home/guest03/Desktop/testdir1/testdir2\n", "!mkdir -p /home/guest03/Desktop/testdir1/testdir2\n", "!ls /home/guest03/Desktop/testdir1" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### Reading and writing to a file\n", "Now, we can see how to download a file, how to read it and how to copy it to a different file (differently than the previous time)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, we have to download a file. There are numerous ways how to do it but this one is one of the most easiest. " ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2018-10-02 11:06:05-- https://www.dropbox.com/s/7ja4d2kifo3cbqx/textfile.txt?dl=0\n", "Resolving www.dropbox.com (www.dropbox.com)... 162.125.66.1, 2620:100:6022:1::a27d:4201\n", "Connecting to www.dropbox.com (www.dropbox.com)|162.125.66.1|:443... connected.\n", "HTTP request sent, awaiting response... 301 Moved Permanently\n", "Location: /s/raw/7ja4d2kifo3cbqx/textfile.txt [following]\n", "--2018-10-02 11:06:05-- https://www.dropbox.com/s/raw/7ja4d2kifo3cbqx/textfile.txt\n", "Reusing existing connection to www.dropbox.com:443.\n", "HTTP request sent, awaiting response... 302 Found\n", "Location: https://uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com/cd/0/inline/AR8LJ2h3BWAupBGIBvHXaN7jK8hXQyeLjDKjiLGbCy-6zahar3bD90qVb2axdHYlCIrnmKU7o4Q_T74OeHuB3Mz0Bgk9Fy94rt2luJH-jrDXwLQKeriidDZ0kqgjVF1E5Hd2MquPSKUVBuCTzxJwoO6kxBWN75MNJet2p2t5gp-QbyMvFY4WRXKq52jTV_WqapenTz2_lcyib9kSMy-qcbSy/file [following]\n", "--2018-10-02 11:06:05-- https://uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com/cd/0/inline/AR8LJ2h3BWAupBGIBvHXaN7jK8hXQyeLjDKjiLGbCy-6zahar3bD90qVb2axdHYlCIrnmKU7o4Q_T74OeHuB3Mz0Bgk9Fy94rt2luJH-jrDXwLQKeriidDZ0kqgjVF1E5Hd2MquPSKUVBuCTzxJwoO6kxBWN75MNJet2p2t5gp-QbyMvFY4WRXKq52jTV_WqapenTz2_lcyib9kSMy-qcbSy/file\n", "Resolving uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com (uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com)... 162.125.66.6, 2620:100:6022:6::a27d:4206\n", "Connecting to uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com (uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com)|162.125.66.6|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 1640 (1.6K) [text/plain]\n", "Saving to: ‘textfile.txt?dl=0’\n", "\n", "textfile.txt?dl=0 100%[===================>] 1.60K --.-KB/s in 0s \n", "\n", "2018-10-02 11:06:05 (137 MB/s) - ‘textfile.txt?dl=0’ saved [1640/1640]\n", "\n" ] } ], "source": [ "!# download a file from the command line\n", "!wget https://www.dropbox.com/s/7ja4d2kifo3cbqx/textfile.txt?dl=0" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "movedfile newfile textfile.txt textfile.txt?dl=0\r\n" ] } ], "source": [ "!# see if we have it\n", "!ls" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1. Introduction to NGS technologies: a brief introduction to biology, sequencing, history, NGS technologies and their applications, sample extraction, library preparation, basic glossary.\r\n", "2. The basic scheme of data analysis: how the data look like, definition of general steps in NGS data analysis, differences in dependence on the application (eg. variant calling vs RNA-Seq …), projects introduction.\r\n", "3. Student project assignment and introduction to software for data analysis: a brief introduction to work with Linux, Bash and R, data formats and the differences between them, on-line courses, discussion about projects.\r\n", "4. Quality control, data processing, specifications and start of work on projects: tools for quality control, Phred score, data pre-processing, examples on sample data.\r\n", "5. Alignment and post-processing: reference genome databases, annotations, the differences between them and application, explanations of alignment algorithms, differences between spliced/non-spliced ​​tools and their application, alignment quality control, alignment visualization.\r\n", "6. Theory to specifics parts of the analysis of the projects 1. (based on the student selections)\r\n", "7. Theory to specifics parts of the analysis of the projects 2. (based on the student selections)\r\n", "8. Theory to specifics parts of the analysis of the projects 3. (based on the student selections)\r\n", "9. Theory to specifics parts of the analysis of the projects 4. (based on the student selections)\r\n", "10. Projects processing/analysis, consultations.\r\n", "11. Projects processing/analysis and projects finalization, consultations.\r\n", "12. Presentation of the project results.\r\n" ] } ], "source": [ "!# see the content of the file\n", "!cat textfile.txt\\?dl\\=0" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2018-10-02 11:07:43-- https://www.dropbox.com/s/7ja4d2kifo3cbqx/textfile.txt?dl=0\n", "Resolving www.dropbox.com (www.dropbox.com)... 162.125.66.1, 2620:100:6022:1::a27d:4201\n", "Connecting to www.dropbox.com (www.dropbox.com)|162.125.66.1|:443... connected.\n", "HTTP request sent, awaiting response... 301 Moved Permanently\n", "Location: /s/raw/7ja4d2kifo3cbqx/textfile.txt [following]\n", "--2018-10-02 11:07:43-- https://www.dropbox.com/s/raw/7ja4d2kifo3cbqx/textfile.txt\n", "Reusing existing connection to www.dropbox.com:443.\n", "HTTP request sent, awaiting response... 302 Found\n", "Location: https://uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com/cd/0/inline/AR8LJ2h3BWAupBGIBvHXaN7jK8hXQyeLjDKjiLGbCy-6zahar3bD90qVb2axdHYlCIrnmKU7o4Q_T74OeHuB3Mz0Bgk9Fy94rt2luJH-jrDXwLQKeriidDZ0kqgjVF1E5Hd2MquPSKUVBuCTzxJwoO6kxBWN75MNJet2p2t5gp-QbyMvFY4WRXKq52jTV_WqapenTz2_lcyib9kSMy-qcbSy/file [following]\n", "--2018-10-02 11:07:43-- https://uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com/cd/0/inline/AR8LJ2h3BWAupBGIBvHXaN7jK8hXQyeLjDKjiLGbCy-6zahar3bD90qVb2axdHYlCIrnmKU7o4Q_T74OeHuB3Mz0Bgk9Fy94rt2luJH-jrDXwLQKeriidDZ0kqgjVF1E5Hd2MquPSKUVBuCTzxJwoO6kxBWN75MNJet2p2t5gp-QbyMvFY4WRXKq52jTV_WqapenTz2_lcyib9kSMy-qcbSy/file\n", "Resolving uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com (uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com)... 162.125.66.6, 2620:100:6022:6::a27d:4206\n", "Connecting to uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com (uc1016a6073b86c8ad789bb93f74.dl.dropboxusercontent.com)|162.125.66.6|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 1640 (1.6K) [text/plain]\n", "Saving to: ‘textfile.txt’\n", "\n", "textfile.txt 100%[===================>] 1.60K --.-KB/s in 0s \n", "\n", "2018-10-02 11:07:44 (150 MB/s) - ‘textfile.txt’ saved [1640/1640]\n", "\n" ] } ], "source": [ "!# fix the ugly name\n", "!wget https://www.dropbox.com/s/7ja4d2kifo3cbqx/textfile.txt?dl=0 -O textfile.txt" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "movedfile newfile textfile.txt textfile.txt?dl=0\r\n" ] } ], "source": [ "!# see if it's there and delete the old one\n", "!ls\n", "!rm textfile.txt\\?dl\\=0" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "12 textfile.txt\r\n" ] } ], "source": [ "!# how many lines do we have\n", "!wc -l textfile.txt" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Top 10 lines.\n", "1. Introduction to NGS technologies: a brief introduction to biology, sequencing, history, NGS technologies and their applications, sample extraction, library preparation, basic glossary.\n", "2. The basic scheme of data analysis: how the data look like, definition of general steps in NGS data analysis, differences in dependence on the application (eg. variant calling vs RNA-Seq …), projects introduction.\n", "3. Student project assignment and introduction to software for data analysis: a brief introduction to work with Linux, Bash and R, data formats and the differences between them, on-line courses, discussion about projects.\n", "4. Quality control, data processing, specifications and start of work on projects: tools for quality control, Phred score, data pre-processing, examples on sample data.\n", "5. Alignment and post-processing: reference genome databases, annotations, the differences between them and application, explanations of alignment algorithms, differences between spliced/non-spliced ​​tools and their application, alignment quality control, alignment visualization.\n", "6. Theory to specifics parts of the analysis of the projects 1. (based on the student selections)\n", "7. Theory to specifics parts of the analysis of the projects 2. (based on the student selections)\n", "8. Theory to specifics parts of the analysis of the projects 3. (based on the student selections)\n", "9. Theory to specifics parts of the analysis of the projects 4. (based on the student selections)\n", "10. Projects processing/analysis, consultations.\n", "Last 10 lines\n", "3. Student project assignment and introduction to software for data analysis: a brief introduction to work with Linux, Bash and R, data formats and the differences between them, on-line courses, discussion about projects.\n", "4. Quality control, data processing, specifications and start of work on projects: tools for quality control, Phred score, data pre-processing, examples on sample data.\n", "5. Alignment and post-processing: reference genome databases, annotations, the differences between them and application, explanations of alignment algorithms, differences between spliced/non-spliced ​​tools and their application, alignment quality control, alignment visualization.\n", "6. Theory to specifics parts of the analysis of the projects 1. (based on the student selections)\n", "7. Theory to specifics parts of the analysis of the projects 2. (based on the student selections)\n", "8. Theory to specifics parts of the analysis of the projects 3. (based on the student selections)\n", "9. Theory to specifics parts of the analysis of the projects 4. (based on the student selections)\n", "10. Projects processing/analysis, consultations.\n", "11. Projects processing/analysis and projects finalization, consultations.\n", "12. Presentation of the project results.\n" ] } ], "source": [ "!# what is the beginning of the file (top 10 lines)\n", "!echo \"Top 10 lines.\"\n", "!head textfile.txt\n", "!# what is the end of the file (last 10 lines)\n", "!echo \"Last 10 lines\"\n", "!tail textfile.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Editing a text file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also edit the files right from the command line. There is a number of tools for this purpose but one of the most easiest one is `nano` or `pico`. Unfortunately, this editing doesn't work in Jupyter and we have to open a command line for this. Please, do this now." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Edit our text file using `nano`.\n", "`nano textfile.txt`\n", "Now you can do whatever you want, change, delete, etc. You can also see a 'help' bar at the bottom. \n", "\n", "Once you are done you can save the file by pressing `control+o`, typing a new name of the file or saving to a same file.\n", "\n", "`nano` can do a lot of other stuff. Just see the manual. \n", "\n", "It might happen that the computer won't have `nano` nor `pico` installed. In that case you have to use the 'hardcore' tools such as `vi` or `vim` but this I will leave to you." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Loop commands\n", "A very useful (=you will use it a lot) are so called **loops**. There is a lot of them and this is one of very nice tutorials which you can go through are [here](https://ryanstutorials.net/bash-scripting-tutorial/bash-loops.php)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will mainly use the **for loop**." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "!# this loops goes thourgh all files that have a \".txt\" suffix and prints their name and first 10 lines\n", "for file in *.txt\n", "do\n", " echo $file\n", " head $file\n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note the **\\*** character. This is one of the **regular expressions**. The **\\*** stands for **any number of any characters**. This means there could be any name of any length of the file but it just has to finish with \".txt\"." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.1" } }, "nbformat": 4, "nbformat_minor": 2 }