Malware Analysis Basics: Static and Dynamic Techniques

You’ve captured a suspicious file. Now what?

Malware analysis lets you understand what malicious software does, how it works, and how to defend against it. This guide covers the fundamentals.

⚠️ Safety First

NEVER analyze malware on your main system.

Setting Up a Safe Lab

┌─────────────────────────────────────────────────────────┐
│                    Host Machine                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │              Isolated Network                    │   │
│  │   ┌───────────────┐    ┌───────────────┐       │   │
│  │   │ Analysis VM   │    │  INetSim/     │       │   │
│  │   │ (Windows 10)  │◄──►│  FakeNet      │       │   │
│  │   │ Snapshot: Clean│    │ (Fake Internet)│      │   │
│  │   └───────────────┘    └───────────────┘       │   │
│  └─────────────────────────────────────────────────┘   │
│                    No Internet Access                    │
└─────────────────────────────────────────────────────────┘

VM Setup Checklist

□ VirtualBox or VMware (not Hyper-V for evasive malware)
□ Windows 10/11 VM (many samples target Windows)
□ Host-only or isolated network
□ Disable shared folders
□ Disable drag-and-drop
□ Take clean snapshot BEFORE analysis
□ Install analysis tools
□ Take another snapshot with tools installed

Essential Tools

# Static Analysis
- PEStudio (PE file analysis)
- Detect It Easy (DIE) (packer detection)
- Strings (extract strings)
- YARA (pattern matching)
- Ghidra/IDA Free (disassembly)

# Dynamic Analysis
- Process Monitor (Procmon)
- Process Hacker
- Regshot (registry changes)
- Wireshark (network capture)
- API Monitor
- x64dbg (debugger)

# Automation
- Cuckoo Sandbox
- Any.Run
- VirusTotal

Static Analysis (Without Execution)

Static analysis examines the file without running it.

Step 1: File Identification

# What type of file is it?
file suspicious.exe
# PE32 executable (GUI) Intel 80386, for MS Windows

# Calculate hashes (for VirusTotal lookup)
md5sum suspicious.exe
sha256sum suspicious.exe

# Check VirusTotal
# https://www.virustotal.com/gui/file/<SHA256>

Step 2: String Extraction

# Extract readable strings
strings -n 8 suspicious.exe > strings.txt

# Look for:
# - URLs and IPs
# - File paths
# - Registry keys
# - Function names
# - Error messages
# - Encoded data (base64)

grep -E "(http|ftp|\\\\|HKEY_|\.exe|\.dll)" strings.txt

Common indicators:

# C2 URLs
http://malicious-domain.com/gate.php
https://192.168.1.100/beacon

# Persistence
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run
%APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup

# Anti-analysis
IsDebuggerPresent
CheckRemoteDebuggerPresent
VirtualBox
VMware
SbieDll.dll  (Sandboxie)

Step 3: PE Header Analysis

Use PEStudio or pefile:

import pefile

pe = pefile.PE('suspicious.exe')

# Check compilation timestamp
print(f"Compiled: {pe.FILE_HEADER.TimeDateStamp}")

# Check imports (what APIs does it use?)
for entry in pe.DIRECTORY_ENTRY_IMPORT:
    print(f"\n{entry.dll.decode()}")
    for imp in entry.imports:
        print(f"  {imp.name.decode() if imp.name else 'ordinal'}")

Suspicious imports:

# Process injection
VirtualAllocEx, WriteProcessMemory, CreateRemoteThread

# Keylogging
SetWindowsHookEx, GetAsyncKeyState

# Screen capture
BitBlt, GetDC

# Persistence
RegSetValueEx, CreateService

# Evasion
IsDebuggerPresent, GetTickCount, Sleep

Step 4: Packer/Obfuscation Detection

# Using Detect It Easy
diec suspicious.exe
# Output: UPX(3.96)[NRV2E]

# High entropy sections = likely packed/encrypted
# Use entropy analysis in PEStudio

Common packers:

  • UPX (easy to unpack: upx -d)
  • Themida/WinLicense (hard)
  • VMProtect (very hard)
  • Custom packers

Dynamic Analysis (Execution)

Dynamic analysis runs the malware in a controlled environment.

Step 1: Prepare Monitoring

# Start Process Monitor (filter to your sample)
# Start Wireshark (capture all traffic)
# Start Regshot (first snapshot)

# In INetSim/FakeNet: simulate internet
# This catches C2 callbacks without real connection

Step 2: Execute and Monitor

# Run the sample
.\suspicious.exe

# Watch for:
# - New processes spawned
# - Files created/modified
# - Registry changes
# - Network connections
# - Loaded DLLs

Step 3: Process Monitor Analysis

Filter for your process and look for:

Operation: CreateFile
Path: C:\Users\...\AppData\Roaming\malware.exe
Result: SUCCESS
# Malware copies itself

Operation: RegSetValue
Path: HKCU\Software\Microsoft\Windows\CurrentVersion\Run\backdoor
# Persistence mechanism

Operation: TCP Connect
Path: 192.168.1.100:443
# C2 communication

Step 4: Network Analysis

# In Wireshark, look for:
# - DNS queries (domain C2)
# - HTTP requests (data exfiltration)
# - Unusual ports
# - Beaconing patterns (regular intervals)

# Export HTTP objects
# File > Export Objects > HTTP

Step 5: Registry Changes

# Using Regshot:
# 1. First snapshot (before execution)
# 2. Run malware
# 3. Second snapshot
# 4. Compare

# Common malicious changes:
HKCU\Software\Microsoft\Windows\CurrentVersion\Run
HKLM\SYSTEM\CurrentControlSet\Services
HKCU\Software\Classes\*\shell

Behavioral Indicators

Persistence Mechanisms

Registry Run Keys:
HKCU\Software\Microsoft\Windows\CurrentVersion\Run
HKLM\Software\Microsoft\Windows\CurrentVersion\Run

Scheduled Tasks:
schtasks /create /tn "Update" /tr "malware.exe" /sc onlogon

Services:
sc create MalService binPath= "C:\malware.exe"

Startup Folder:
%APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup

Defense Evasion

Process Injection:
- CreateRemoteThread
- QueueUserAPC
- SetWindowsHookEx

Anti-VM:
- Check for VMware/VirtualBox processes
- Check MAC address prefixes
- CPUID instruction

Anti-Debug:
- IsDebuggerPresent
- Timing checks (RDTSC)
- Exception-based detection

Data Exfiltration

HTTP POST to suspicious URLs
DNS tunneling (long subdomains)
ICMP data exfil (ping with payload)
Cloud storage APIs (Dropbox, Google Drive)

YARA Rules

Create detection signatures:

rule Suspicious_Keylogger
{
    meta:
        description = "Detects potential keylogger"
        author = "Security Team"
        
    strings:
        $api1 = "GetAsyncKeyState" ascii wide
        $api2 = "SetWindowsHookEx" ascii wide
        $api3 = "GetKeyboardState" ascii wide
        $log = "keylog" ascii wide nocase
        
    condition:
        uint16(0) == 0x5A4D and  // MZ header (PE file)
        2 of ($api*) and
        $log
}
# Scan with YARA
yara -r rules.yar suspicious.exe

Quick Analysis Workflow

## 5-Minute Triage
1. Hash lookup on VirusTotal
2. String extraction + grep for IOCs
3. PE header check (imports, entropy)
4. Submit to sandbox (Any.Run, Hybrid Analysis)

## 30-Minute Analysis
1. All of the above
2. Unpack if packed (UPX, etc.)
3. Dynamic analysis in VM
4. Document IOCs

## Deep Dive (hours/days)
1. Reverse engineering (Ghidra/IDA)
2. Protocol analysis
3. Decryption of configs
4. Full capability mapping

Online Sandboxes

When you can’t run locally:

  • Any.Run: Interactive sandbox, watch execution live
  • Hybrid Analysis: Automated analysis with report
  • VirusTotal: Hash lookup + community comments
  • Joe Sandbox: Detailed behavioral analysis
  • Triage: Quick classification

Conclusion

Malware analysis is a skill built through practice:

  1. Start with static analysis — it’s safer
  2. Use VMs religiously — one escape can compromise everything
  3. Document everything — IOCs, behaviors, timestamps
  4. Share findings — contribute to threat intelligence
  5. Stay curious — every sample teaches something new

The goal isn’t to become a reverse engineering expert overnight. It’s to understand enough to defend your organization effectively.

Related: Linux Log Analysis for Security Monitoring