At some point, your application runs on Linux. Maybe you’re SSHed into a server debugging a production issue. Maybe you’re inside a Docker container trying to figure out why something isn’t starting. Maybe a deployment failed and the logs are somewhere on a machine you’ve never touched.

You don’t need to manage servers for a living to be useful in these moments. You need enough of a mental model to understand what you’re looking at and enough command fluency to find what you need.

This is that mental model.

Everything is a file

Linux takes the concept further than most systems. Processes, sockets, pipes, devices, and even kernel state are all exposed through the filesystem in one way or another. Understanding this unlocks a lot.

A regular file you write to disk - that’s a file. A network socket your server is listening on - also a file descriptor. A pipe between two processes - file descriptors. The directory /proc contains a virtual filesystem where every running process has an entry, and inside that entry are files that tell you everything about the process: its open file descriptors, its memory maps, its command line.

# See what files process 1234 has open
ls -la /proc/1234/fd

# See the command that started process 1234
cat /proc/1234/cmdline | tr '\0' ' '

When your server says “too many open files”, it’s telling you a process has hit its file descriptor limit. When a socket stays in CLOSE_WAIT, you can see it. Everything is inspectable.

Processes

Every program runs as a process. Every process has a PID (process ID), a parent process, an owner (a user ID), and a set of resources it holds.

The ps command shows running processes. The version you want for exploring is:

ps aux

This shows all processes (a), including those not attached to a terminal (x), with the owning user (u).

More useful for interactive exploration is htop if it’s installed - it’s a real-time view of processes with CPU and memory usage, sortable, searchable.

When a process is misbehaving, you often want to know:

• Is it running? (ps aux | grep processname)
• What is it doing? (strace -p PID will show system calls in real time - useful when a process seems stuck)
• What files does it have open? (lsof -p PID)
• What ports is it listening on? (ss -tlnp or netstat -tlnp)

lsof (list open files) is one of the most useful tools for debugging. It works on processes, files, ports, and more:

# What process is using port 8080?
lsof -i :8080

# What files does PID 1234 have open?
lsof -p 1234

Signals

Signals are how you communicate with a running process. Most developers know kill -9 - the “force kill” - but there are others worth knowing.

Signal	Number	Meaning
SIGTERM	15	Please shut down gracefully
SIGKILL	9	Terminate immediately, no cleanup possible
SIGHUP	1	Reload configuration (convention, not guaranteed)
SIGINT	2	Interrupt (this is what Ctrl+C sends)
SIGUSR1/2	10/12	User-defined, application-specific

The right way to stop a process is SIGTERM first. A well-written application handles it, closes connections cleanly, flushes buffers, and exits. SIGKILL can’t be caught or ignored - the kernel forcibly terminates the process. Use it when SIGTERM doesn’t work, not as the first option.

Many daemons reload their configuration on SIGHUP. Nginx does this. This is how you tell a process to pick up a new config file without restarting it.

kill -TERM 1234    # Graceful shutdown
kill -HUP 1234     # Reload config
kill -KILL 1234    # Force kill

Permissions

Every file and directory has an owner (user), a group, and three sets of permissions: for the owner, for the group, and for everyone else. Each set has three bits: read, write, execute.

-rwxr-xr-- 1 alice developers 1234 Apr 18 10:00 script.sh

Reading this: the owner (alice) can read, write, and execute. The group (developers) can read and execute. Everyone else can only read.

The chmod command changes permissions. The symbolic form is more readable than octal:

chmod u+x script.sh      # Give owner execute permission
chmod go-w file.txt      # Remove write from group and others
chmod 755 script.sh      # Owner: rwx, group: r-x, others: r-x

chown changes ownership:

chown alice:developers file.txt

When you see “Permission denied”, the answer is usually one of: wrong user, wrong group, wrong permissions on the file, wrong permissions on a parent directory. Check each in that order.

The filesystem hierarchy

Linux has a standard layout. You don’t need to memorize all of it, but a few directories matter:

• /etc - configuration files for system and services
• /var/log - log files (this is where you look when something is broken)
• /tmp - temporary files, cleared on reboot
• /usr/bin, /bin - system binaries and commands
• /home/username - user home directories
• /proc - virtual filesystem with kernel and process state
• /sys - virtual filesystem with hardware and kernel parameters

Your application’s logs are almost certainly in /var/log/ somewhere, or written to stdout/stderr (which a process manager like systemd captures). When something is broken, start there.

systemd and services

On modern Linux systems, services are managed by systemd. The systemctl command is your interface to it.

# Check if nginx is running
systemctl status nginx

# Start, stop, restart a service
systemctl start nginx
systemctl stop nginx
systemctl restart nginx

# Reload configuration without restart
systemctl reload nginx

# Enable/disable on boot
systemctl enable nginx
systemctl disable nginx

When a service isn’t starting, systemctl status shows the last few log lines. For more:

journalctl -u nginx          # All logs for nginx
journalctl -u nginx -f       # Follow logs in real time
journalctl -u nginx --since "1 hour ago"

Environment variables

Processes inherit environment variables from their parent. When your application can’t find a configuration value it reads from DATABASE_URL, the first thing to check is whether the environment variable is actually set in the process that runs your app.

# Show all env vars in current shell
env

# Show a specific variable
echo $DATABASE_URL

# Set a variable for a single command
DATABASE_URL=postgres://localhost/mydb ./myapp

If you set a variable in your shell and then start a service through systemd, the service won’t see your shell’s environment. systemd has its own environment, configured in the service unit file. This is a common source of confusion.

Disk and memory

When something is slow or failing, you often need to check resources.

# Disk usage by directory
du -sh /var/log/*

# Free disk space on each filesystem
df -h

# Memory usage
free -h

# Real-time resource usage
top
# or if available:
htop

A server that’s run out of disk space will fail in ways that look nothing like “disk full” - log writes fail, databases crash, uploads silently disappear. Check disk space early when debugging anything unexpected.

A practical debugging flow

Something is broken in production. Here’s a starting sequence that covers most cases:

1. Is the process running? ps aux | grep myapp or systemctl status myapp
2. What do the logs say? journalctl -u myapp -n 100 or wherever your app writes logs
3. Is it listening on the expected port? ss -tlnp | grep :8080
4. Is it out of disk or memory? df -h and free -h
5. Are there permission issues? Check the log for “Permission denied”
6. What files does it have open? lsof -p PID

This covers a large fraction of production issues. The rest usually involves reading logs more carefully.

Linux rewards curiosity. Most things are inspectable, most state is visible if you know where to look, and most debugging tools follow consistent patterns once you know a few of them. You don’t need to know everything - you need to know enough to find the answer.