Day 1: Linux Basics – Mastering Linux Command Line: A Comprehensive Guide to File Management, Shell Operations, and Scripting Techniques

Linux Common Commands

Linux commands can be categorized based on their functionality. Here are some main categories along with some common commands in each category:

1. File and Directory Operations

• ls – List directory contents.
• cd – Change the current directory.
• pwd – Print the current working directory.
• mkdir – Create a new directory.
• rmdir – Remove an empty directory.
• touch – Create an empty file or update file timestamps.
• cp – Copy files or directories.
• mv – Move or rename files and directories.
• rm – Remove files or directories.

2. Viewing and Editing File Contents

• cat – Concatenate and print files to standard output.
• less – View file contents page by page.
• more – Display file contents screen by screen.
• head – View the beginning part of a file.
• tail – View the end part of a file, with options for following updates.
• nano / vim / emacs – Text editors.

3. File Permissions and Ownership

• chmod – Change access permissions for files or directories.
• chown – Change the owner of files or directories.
• chgrp – Change the group of files or directories.

4. File Searching

• find – Search for files in a directory tree.
• grep – Search for text patterns within files.
• locate – Quickly search for files in the file system using a database.

5. System Monitoring and Management

• top – Monitor processes and resource usage in real time.
• htop – Interactive process viewer.
• ps – Display current system process status.
• free – Show memory usage information.
• df – Report file system disk space usage.
• du – Check disk usage of files or directories.

6. Network Operations

• ping – Test network connectivity to a host.
• netstat – Display network connections, routing tables, etc.
• ifconfig / ip addr – Configure or display network interface parameters.
• ssh – Secure remote login and file transfer.
• scp – Securely copy files to a remote machine.

7. System Boot and Service Management

• systemctl – Control the systemd system and service manager.
• service – Manage system services (on older systems).
• crontab – Schedule cron jobs.

8. File Compression and Archiving

• tar – Archive files.
• gzip – Compress files.
• bzip2 – Compress files.
• zip – Create ZIP compressed files.
• unzip – Decompress ZIP files.

9. Disk and Filesystem Management

• fdisk – Disk partitioning.
• mkfs – Create a filesystem.
• mount – Mount a filesystem.
• umount – Unmount a filesystem.

10. User and Group Management

• useradd – Add a new user.
• usermod – Modify user information.
• userdel – Delete a user.
• groupadd – Add a new group.
• groupmod – Modify group information.
• groupdel – Delete a group.

11. Package Management

• apt-get / apt – Package management for Debian and Debian-based systems (e.g., Ubuntu).
• yum – Package management for Red Hat and Red Hat-based systems.
• dnf – Package management for Fedora.
• pacman – Package management for Arch Linux.

12. Process Management

• kill – Send signals to processes.
• killall – Kill processes by name.
• top – Display processes and resource usage in real time.
• pstree – Show process trees.

13. Text Processing and Regular Expressions

• sed – Stream editor for text processing.
• awk – Powerful text processing tool.
• grep – Search for text patterns using regular expressions.

14. Time and Date

• date – Display or set the date and time.
• cal – Display a calendar.

15. System Information

• uname – Display system information.
• hostname – Display or set the system’s hostname.
• lscpu – Display CPU information.
• lsblk – List block devices.

Linux Commands and Parameters

Here are some common Linux commands along with their parameter combinations used to perform specific tasks:

1. ls
   • ls -l: List files and directories in long format.
   • ls -a: Include hidden files (those starting with a dot).
   • ls -lrt: Sort by modification time in reverse order, list in long format.
2. grep
   • grep -r “pattern” /path: Recursively search for “pattern” in all files within the /path directory.
   • grep -i “pattern” file.txt: Search for “pattern” without case sensitivity.
3. find
   • find /path -type f -name “*.txt”: Find all files with a .txt extension in the /path directory.
   • find /path -exec rm {} \;: Delete all files found in the /path directory.
4. chmod
   • chmod 755 file: Set permissions to read/write/execute for the owner, and read/execute for the group and others.
   • chmod -R 700 /path: Recursively set read/write/execute permissions for the owner on the /path directory and its contents.
5. chown
   • chown user:group file: Change the owner of the file to user and the group to group.
   • chown -R user:group /path: Recursively change the owner and group of the /path directory and its contents.
6. tar
   • tar -cvf archive.tar /path: Create a tar archive containing the contents of /path.
   • tar -xvf archive.tar -C /path: Extract the contents of archive.tar into the /path directory.
7. scp
   • scp localfile user@remote:/path: Copy a local file to a specified path on a remote host.
   • scp user@remote:/path/remotefile .: Copy a file from a remote host to the local machine.
8. ping
   • ping -c 4 host: Send 4 ECHO requests to the host.
   • ping -t host: Continuously ping the host until interrupted.
9. ssh
   • ssh -p port user@host: Connect to a remote host through a specified port.
   • ssh -X user@host: Enable X11 forwarding when connecting to a remote host.
10. df
    • df -h: Display disk space usage in a human-readable format (e.g., KB, MB, GB).
    • df -i: Show inode usage.
11. du
    • du -sh /path: Show the total disk usage of the /path directory.
    • du -h –max-depth=1 /path: Show the sizes of the /path directory and its immediate subdirectories.
12. top
    • top -b -n1: Run in batch mode, displaying once and then exiting.
    • top -p PID: Display information related only to the specified process.
13. ps
    • ps aux: Show detailed information about all processes.
    • ps -ef | grep “process”: Search for and display information related to processes with “process” in their name.
14. kill
    • kill -9 PID: Forcefully terminate the process with the specified PID.
15. chmod
    • chmod u+x file: Add execute permission for the owner of the file.
    • chmod g-w file: Remove write permission for the group.
16. ln
    • ln -s /source /link
    • Create a symbolic link /link pointing to /source.

17. wget
    • wget -c <http://example.com/file: Download a file with the c (continue) option.
18. chmod
    • chmod -R g+rwx /path: Recursively add read/write/execute permissions for the group on the /path directory and its contents.
19. awk
    • awk ‘{print $1, $3}’ file.txt: Print the first and third fields of each line in the file.
20. sed
    • sed -i ‘s/search/replace/g’ file.txt: Globally replace “search” with “replace” in the file.

These command and parameter combinations cover various aspects such as file operations, text processing, network communication, and performance monitoring, and are commonly used tools for Linux users and system administrators.

Linux Filesystem Hierarchy

The Linux filesystem hierarchy adheres to a standard known as the Filesystem Hierarchy Standard (FHS). This standard defines how files and directories should be organized and what types of files each directory should contain. Here are some of the most important directories in the Linux filesystem hierarchy and their purposes:

1. / (root) – This is the top-level directory of the filesystem hierarchy. In Linux, all files and directories start from the root.
2. /bin – Contains the most important commands that the system administrator and users need to use in single-user mode. These commands are essential.
3. /boot – Contains the files required to boot the Linux system, such as the kernel and boot loader (e.g., GRUB).
4. /dev – Contains device files that act as interfaces to the hardware devices on the system.
5. /etc – Contains system and installed software configuration files. These files typically are not the programs themselves but rather configuration or status information.
6. /home – The personal home directories for users. Each user has their own subdirectory.
7. /lib – Contains shared library files required for the system to boot and run.
8. /media – The mount point for automatically mounted removable media (e.g., USB drives, CDs, DVDs, etc.).
9. /mnt – A directory for temporarily mounting filesystems, usually mounted manually by the system administrator or user.
10. /opt – The optional software package installation directory, used for third-party software.
11. /proc – A virtual filesystem that contains runtime information about the system kernel and processes.
12. /root – The home directory of the system administrator (root user).
13. /sbin – Contains commands used by system administrators, typically not for individual command binaries.
14. /srv – Contains data for services provided by the system, such as Web, FTP, etc.
15. /sys – Similar to /proc, a virtual filesystem that provides a mechanism for accessing and interacting with the kernel.
16. /tmp – A temporary file storage directory, typically used for temporary files of the system and users.
17. /usr – A place for user programs and data. It is divided into /usr/bin, /usr/lib, /usr/share, etc.
18. /var – Variable state files, such as log files, databases, documents for Web servers, etc.
19. /run – Contains system runtime system information, such as log files and temporary device files.
20. /lost+found – When a filesystem is damaged, the fsck command places recovered files here.

Understanding the purpose of these directories is crucial for Linux system administrators and users, as they need to know where to look for files, configure the system, or manage data. This hierarchy helps maintain organization and consistency in the system.

Linux Path Shortcuts

In Linux, a path is a string that specifies the location of a file or directory. Mastering some path-related shortcuts can greatly improve your efficiency in the command line. Here are some commonly used path shortcuts:

1. / (root directory)
   • Represents the top-level directory of the filesystem hierarchy.
2. ./ (current directory)
   • Refers to the current working directory. For example, ./file.txt refers to the file.txt in the current directory.
3. ../ (parent directory)
   • Refers to the directory one level above the current directory. For example, ../files/file.txt refers to file.txt in the files directory of the parent directory.
4. ~/ (user’s home directory)
   • Refers to the home directory of the current user, typically located at /home/username.
5. Environment variable $HOME
   • Usually points to the home directory of the current user, the same as ~/.
6. Relative path
   • A path relative to the current working directory. For example, file.txt refers to file.txt in the current directory.
7. Absolute path
   • A complete path starting from the root directory /. For example, /usr/bin/python.
8. – (previous directory)
   • When using certain commands (such as cd), – can represent the previous working directory.
9. //
   • In a path, extra slashes are generally ignored, so // is the same as /.
10. Special character quoting
    • If a path name contains special characters (such as spaces), you can use quotes ” ” or escape characters \\ to ensure it is parsed correctly.
11. Path expansion
    • The shell provides some path expansion shortcuts, such as ~/Documents which automatically fills in to /home/username/Documents.
12. $(pwd)
    • Using command substitution, $(pwd) expands to the full path of the current working directory.
13. $(dirname path)
    • The dirname command can get the directory part of a path.

Using these shortcuts allows you to quickly navigate the file system, write more concise commands, and manage files and directories more effectively.

Linux File Types

The Linux filesystem supports various types of files:

1. Regular File: The most common type of file used for storing data.
2. Directory: A special type of file that contains other files and directories.
3. Link Files:
   • Hard Link: A direct reference to another file located in the same filesystem.
   • Symbolic Link (Symlink): Similar to a shortcut in Windows, it can point to another file or directory across filesystems.
4. Character Device: Device files representing character data streams.
5. Block Device: Device files that store data in blocks.
6. Socket: An interface for inter-process communication.
7. FIFO (Named Pipe): A method for inter-process communication.

Common commands:

1. ln – Create links (hard or symbolic). ln -s /source /linkname: Create a symbolic link /linkname pointing to /source.
2. ls -l: List files and directories in long format.
3. tree – Display the directory structure in a tree-like format.
4. stat – Show statistical information about a file or filesystem.

Linux Permission Levels

In Linux systems, file permissions are a core mechanism for controlling access to files or directories. Here are the basic concepts of file permissions:

Linux file permissions are divided into three levels:

1. Owner: The owner of the file or directory.
2. Group: The user group to which the file or directory belongs.
3. Others: Users who are neither the owner nor part of the group.

Permission Types

Each file or directory has the following types of permissions:

1. Read (r):
   • For files, allows reading the content.
   • For directories, allows listing the contents.
2. Write (w):
   • For files, allows modifying the content.
   • For directories, allows adding, deleting, or renaming files within.
3. Execute (x):
   • For files, allows executing the file as a program.
   • For directories, allows entering the directory.

Viewing and Modifying Permissions

Use the ls -l command to view the permissions of a file or directory:

-rwxr-xr-- 1 user group 4096 Dec 20 12:00 example.txt

Here:

• The first character indicates the file type (- for regular file, d for directory).
• The next three groups of characters represent the permissions for the owner, group, and others, respectively.
• The final number indicates the number of hard links to the file or directory.

Use the chmod command to modify the permissions of a file or directory:

chmod u+x example.txt  # Add execute permission for the owner
chmod g-w groupfile   # Remove write permission from the group
chmod o=rx otherfile   # Set read and execute permissions for others

In Linux, the chmod command can set file or directory permissions using numbers. This numeric representation is based on binary, with each number representing different permission levels.

Numeric Permission Representation

• 4 – Read (r)
• 2 – Write (w)
• 1 – Execute (x)

For files:

• Read permission allows you to view the file’s content.
• Write permission allows you to modify the file’s content.
• Execute permission is generally meaningless for files unless it is an executable program.

For directories:

• Read permission allows you to list the directory’s contents.
• Write permission allows you to create or delete files within the directory.
• Execute permission allows you to enter the directory (i.e., “access” or “traverse” the directory).

Numeric Permission Combinations

• 7 – Read + Write + Execute (4+2+1)
• 5 – Read + Execute (4+1)
• 3 – Write + Execute (2+1)

Modifying Permissions with Numeric Methods

When modifying permissions using the numeric method, you need to specify permissions for the file’s owner (user), group (group), and others (others). These permissions are specified separately and represented by numbers.

For example, if you want to set a file’s permissions so that the owner has read and write permissions and the group and others only have read permissions, you can use the following command:

chmod 644 filename

Here, 644 is interpreted as follows:

• The first digit 6 represents the owner’s permissions, 6 is 4+2 (read and write).
• The second digit 4 represents the group’s permissions, with only read permission.

If you want to set a directory’s permissions so that the owner has all permissions and the group and others only have the permissions to enter and list the directory’s contents, you can use:

chmod 755 directoryname

Here, 755 is interpreted as follows:

• The first digit 7 represents the owner’s permissions, 7 is 4+2+1 (read, write, and execute).
• The second and third digits 5 represent the group and others’ permissions, 5 is 4+1 (read and execute).

Notes on Using chmod

• Make sure you understand the permissions represented by each numeric combination to avoid accidentally granting too many or too few permissions.
• Only the file’s owner or the superuser (root) can change the file’s permissions.
• Be careful when using chmod, especially when modifying the permissions of system files or directories. Incorrect permission settings can affect the system’s security and stability.

Special Permissions

• Sticky Bit (t):
  • For directories, it prevents users from deleting or moving files owned by other users.
  • Setting method: chmod +t directory.
• SetGID (g) and SetUID (u):
  • Files or directories inherit the parent directory’s group or owner.

A correct understanding and management of file permissions are crucial for system security and data protection. System administrators need to carefully plan permissions to ensure both system security and user convenience.

Shell as an Interface for User-Operating System Interaction

The Shell is a command-line interpreter that serves as an interface between the user and the operating system, allowing users to execute commands, launch programs, and manage files. The Shell reads commands input by the user, translates them into system calls that the operating system can recognize, then waits for the system call to complete and return results.

Different Types of Shells

There are various types of Shells, each with its specific syntax and features. Here are some common Shell types:

• Bash (Bourne Again Shell): The default Shell for most Linux distributions, compatible with the Bourne Shell (sh), and includes many enhanced features.
• Zsh (Z Shell): A Shell that extends the functionality of Bash, offering a rich set of interactive features and scripting capabilities.
• Csh (C Shell): Designed with a syntax similar to the C language, user-friendly for programmers, but less popular than Bash and Zsh.
• Tcsh (TENEX C Shell): An enhanced version of Csh, providing better interactivity and improved scripting features.
• Fish (Friendly Interactive Shell): A user-friendly interactive Shell with intelligent auto-completion and color syntax highlighting.

Basic Shell Commands and Operations

Here are some basic Shell commands and operations:

• Command History (history): Displays a list of recently executed commands, allowing you to quickly repeat operations using history commands.
• Command Aliases (alias): Allows users to create shortcuts for long commands or frequently used commands.
• Pipes (|): Connects the output of one command as the input to another, used to link multiple commands together.

Redirection (>, >>):

• Use > to redirect the output of a command to a file, overwriting if the file exists.
  echo “Hello, World!” > output.txt
• Use >> to append the output of a command to the end of a file, rather than overwriting.
  echo “Another line” >> output.txt

In Linux, pipes are a powerful feature that allows you to connect multiple commands, enabling the output of one command to be directly used as the input for another. Pipes are implemented using the vertical bar symbol |. Here are some common command combinations using pipes:

1. Combining grep with pipes
   • Search for specific text in a file and then filter the results further using grep.

   grep "pattern1" file.txt | grep "pattern2"
   

2. Combining ls with grep
   • List all files in the current directory that end with .txt.

   ls -l | grep "\.txt"
   

3. Combining ps with grep
   • Display all processes running with the name processname.

   ps aux | grep processname
   

4. Combining find with xargs
   • Find all .log files in the current directory and its subdirectories, and pass them to rm to delete them.

   find . -name "*.log" | xargs rm
   

5. Combining sort with pipes
   • Sort the contents of a file and display only unique lines.

   grep "pattern" file.txt | sort | uniq
   

6. Combining awk with pipes
   • Use awk to print the first and third fields of each line, and then pass them to sort.

   awk '{print $1, $3}' file.txt | sort
   

7. Combining wc with pipes
   • Count the number of lines, words, or bytes passed through the pipe.

   ls -l | wc -l
   

8. Combining head with tail and pipes
   • Display the first few lines of a file, and then extract the last word from each line.

   head -n 10 file.txt | tail -n 5 | cut -d' ' -f2
   

9. Combining diff with pipes
   • Compare differences between two files and pass the output to grep to find specific patterns.

   diff file1.txt file2.txt | grep ">"
   

10. Combining ssh with pipes
    • Use ssh to remotely execute a command and pass the output to a local command through the pipe.

    ssh user@remotehost 'command' | grep "pattern"
    

The power of pipes lies in their ability to allow users to combine simple commands into complex scripts to perform advanced tasks. By doing so, users can leverage existing command-line tools to create powerful automation scripts.

Shell Shortcut Keys

1. Ctrl + C – Terminate the currently running command.
2. Ctrl + Z – Put the current command in the background and pause execution.
3. Ctrl + A – Move the cursor to the beginning of the command line.
4. Ctrl + E – Move the cursor to the end of the command line.
5. Ctrl + L – Clear the terminal screen, equivalent to the clear command.
6. Ctrl + R – Search command history, press Ctrl + R and enter a keyword, the Shell will display the most recent matching command.
7. Ctrl + U – Delete everything from the cursor position to the beginning of the line.
8. Ctrl + K – Delete everything from the cursor position to the end of the line.
9. Ctrl + W – Delete the word before the cursor.
10. Ctrl + Y – Paste the text recently deleted with Ctrl + C, Ctrl + U, Ctrl + K, or Ctrl + W.
11. Alt + B – Move backward (left) oneword.
12. 12. Alt + F – Move forward (right) one word.

 

1. Tab – Auto-complete file names or commands, can also complete command options.
2. !! – Repeat the last command executed.
3. history – Display command history, you can use !number to execute a command from history.
4. history | grep search term – Search the command history.
5. sudo !$ – Re-execute the last command with sudo.
6. Ctrl + D – Close the Shell session if nothing is currently being input, can also indicate the end of a line, especially in multi-line commands or scripts.
7. Ctrl + T – Swap the two characters at the cursor.
8. Ctrl + X, Ctrl + E – Move the cursor to the end of the command line, if there is a ~ key above the ESC key, you can use Ctrl + X, Ctrl + X.
9. Ctrl + X, Ctrl + U – Move the cursor to the beginning of the line.
10. Ctrl + X, Ctrl + S – Lock the terminal screen.
11. Esc then b – Move backward (left) one word.
12. Esc then f – Move forward (right) one word.

Writing Simple Shell Scripts

Shell scripts are sequences of commands that can be executed automatically. Here are the basic steps for writing and running a simple Shell script:

1. Create the script file: Use a text editor to create a new file, such as script.sh.
2. Add a Shebang: Add the interpreter path at the first line of the file to tell the system which Shell to use to execute the script.

   #!/bin/bash
   

3. Write commands: Add the desired commands, which can be any command used in the command line.

   echo "Hello, World!"
   ls -l
   

4. Save and close the file.
5. Give execute permission: Use the chmod command to add execute permission to the script file.

   chmod +x script.sh
   

6. Run the script: Run the script by typing the script filename with execute permission in the command line.

   ./script.sh
   

7. Debug the script: If the script does not work as expected, you can use bash -x script.sh to trace the execution process.

By mastering these basic Shell commands and operations, users can effectively interact with the Linux system, automate tasks, and improve efficiency through scripting.

Running Programs in the Background

In Linux, there are several methods to run programs in the background:

1. Using the & Symbol

This is the simplest way to put a command into the background. Just add the & symbol at the end of the command:

command &

For example, to run the ping command in the background:

ping www.example.com &

2. Using the nohup Command

nohup allows you to run a command that will continue to run even after the terminal that started it is closed. It does this by ignoring the SIGHUP signal. It is often used with &:

nohup command &

For example:

nohup ping www.example.com &

Commands Related to Background Processes

command > output.txt 2>&1 &

In Linux, “jobs” refer to processes running in the background. Here are some common commands for managing background jobs:

1. jobs
   • List the background jobs in the current session.
2. fg [jobID]
   • Bring a background job to the foreground. The jobID can be obtained through the jobs command.
3. bg [jobID]
   • Continue running a suspended job in the background.
4. kill [jobID or PID]
   • Send a signal to terminate a job or process. The default signal is SIGTERM.
5. pkill [options] [pattern]
   • Kill processes that match the pattern. The pattern can be a jobID, process name, or user.
6. killall [program name]
   • Kill all processes that match the specified program name.
7. ps
   • Display the status of processes in the current system. It can be combined with grep to find specific background jobs.
8. top or htop
   • Display the dynamics of processes in the system in real time.
9. nohup command
   • Run a command that ignores the hangup signal (SIGHUP), usually used to keep a command running after the terminal is closed.
10. &
    • Put a command into the background for execution.
11. wait [PID]
    • Wait for one or more processes to complete.
12. trap [command] [signal]
    • Set signal handling for a process.
13. pstree
    • Display processes in a tree-like diagram.
14. renice [options] [nice value] [PID]
    • Change the nice value of a process, thus affecting its scheduling priority.
15. nice [command]
    • Run a command with a lower priority.
16. ionice [options] [command]
    • Set the I/O scheduling priority of a process.
17. screen or tmux
    • Session multiplex tools, allowing you to run multiple sessions in a single terminal window.
18. jobs [options]
    • Display background jobs in the current session, option l shows job IDs.
19. fg [%job number]
    • Bring a background job back to the foreground, the job number is the number displayed in the output of the jobs command.
20. killall5 [program name]
    • Send a specific signal to all matching processes. For example, killall -9 sends the SIGKILL signal, forcibly terminating the process.

Using these commands, you can effectively manage background jobs, including starting, stopping, pausing, and killing processes.

How to Run Multiple Commands?

These concepts are important tools in the Linux command line for controlling the flow of command execution. Here is a detailed explanation of these concepts:

; Operator: Unconditional Execution

• The semicolon ; is used to sequence multiple commands, which will be executed in order regardless of whether the previous command succeeds or fails.
• This is suitable for commands that are independent of each other and where the execution of subsequent commands does not depend on the success of the previous ones.

command1; command2; command3

&& Operator: Based on the Previous Command’s Success

• The double ampersand && allows the next command to be executed only if the previous command succeeds (exit status is 0).
• This is used for scenarios where you want to confirm that the previous operation was successful before proceeding.

command1 && command2

If command1 is executed successfully, then command2 will be executed. If command1 fails (exit status is not 0), command2 will not be executed.

|| Operator: Based on the Previous Command’s Failure

• The double pipe || allows the next command to be executed only if the previous command fails.
• This is commonly used for error handling or fallback scenarios, ensuring that a corresponding command is executed even if the previous command fails.

command1 || command2

If command1 fails, command2 will be executed. If command1 is successful, command2 will not be executed.

Grouping Commands with Braces {} and Parentheses ()

• Using braces {} or parentheses () groups multiple commands together, allowing them to be treated as a single unit.
• The main difference between braces {} and parentheses () is the shell environment they create:
  • {}: Commands are executed in the current shell. This means that any changes made within the braces (such as variable assignments or changes in the working directory) will affect the environment outside the braces.
  • (): Commands are executed in a subshell. This means that changes that occur within the parentheses do not affect the environment of the parent shell.

# Grouping with braces
{ command1; command2; }
 # Grouping with parentheses (command1; command2)