Self-host MediaCrawler on a Netcup VPS for social media research

TL;DR: Running MediaCrawler on Netcup in 5 minutes

Self-host MediaCrawler on a Netcup VPS and you get an always-on research box that collects public posts and comments from Chinese social platforms while your laptop stays closed. MediaCrawler is a Python + Playwright toolkit that drives a real browser session instead of reverse-engineering each platform's JavaScript, which is why it keeps working when API-based scrapers break.

• Seven platforms — Xiaohongshu (RedNote), Douyin, Kuaishou, Bilibili, Weibo, Baidu Tieba, and Zhihu, each with keyword search, post-detail crawling, second-level comments, and creator-page collection.
• Browser-automation based — Playwright drives Chromium with a cached login state; a CDP mode can attach to a real Chrome instance on port 9222.
• Flexible storage — CSV, JSON, JSONL, Excel, SQLite, or MySQL. SQLite is the sane default on a single VPS.
• Python 3.11 + Node.js ≥ 16 — installed with uv in about a minute.
• Non-commercial license — the project ships under a Non-Commercial Learning License 1.1: research and learning use only, no commercial deployments, and explicitly no large-scale crawling. Plan accordingly.
• Modest footprint — the Python process is light; headless Chromium typically wants 1–2 GB RAM per browser instance, so 8 GB is a comfortable home.

git clone https://github.com/NanmiCoder/MediaCrawler.git
cd MediaCrawler
uv sync
uv run playwright install chromium
uv run main.py --platform xhs --lt qrcode --type search

Primary pick: VPS 1000 G12 — 4 vCPU, 8 GB RAM, 256 GB NVMe, around €10/month.

• First month free: 5799nc17843120991
• Second code: 5799nc17835990830
• Third code: 5799nc17843120990

Introduction

Anyone studying Chinese social media — academics tracking discourse, data journalists following a story, students building an NLP corpus — runs into the same wall: the platforms have no useful public APIs, and commercial data vendors charge enterprise prices for access to what is, on screen, public content. MediaCrawler exists in that gap. It has collected roughly 59,000 GitHub stars by doing one thing well: pretending to be you, in a real browser, and writing down what it sees.

The catch is that browser-driven collection is slow by design. A keyword crawl across a few hundred posts with comments enabled runs for hours, not seconds. Running that on a laptop means keeping the lid open, the Wi-Fi stable, and the machine awake — and losing your cached login state every time something interrupts it. A small server is the obvious fix: crawls run unattended, the IP address stays constant (platforms notice when your session hops between networks), and the login cache survives between sessions.

This is where a cheap Netcup VPS earns its keep. Netcup runs its own hardware in German and Austrian data centers, puts NVMe storage on every tier, and charges €5.91–€19.25/month for the sizes relevant here — less than most providers charge for half the specs. For a workload that mostly waits on network I/O and occasionally spikes a Chromium process, that is exactly the right kind of box.

By the end of this article you will have MediaCrawler installed on a hardened Debian VPS, a working crawl writing into SQLite, a scheduled job running it unattended, and a clear answer on which Netcup tier fits your usage. One thing to keep in view throughout: the project's license restricts it to learning and research, and its own documentation warns against aggressive crawling. Treat both as constraints, not suggestions.

What is MediaCrawler?

MediaCrawler is an open-source, self-hosted social media crawler that collects public posts, comments, and creator profiles from seven Chinese platforms: Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Baidu Tieba, and Zhihu. It is written in Python and maintained by NanmiCoder, with an unusually active contributor community for a scraping project — the platforms change constantly, and the project keeps pace.

Architecture

The design decision that defines MediaCrawler: it does not reverse-engineer each platform's private API and signature algorithms. Instead it uses Playwright to drive a real Chromium browser, keeps a logged-in browser context alive, and extracts data from the pages and network responses that context produces. A CDP (Chrome DevTools Protocol) mode goes one step further and attaches to an actual Chrome instance with remote debugging enabled on port 9222, making the automation nearly indistinguishable from a person browsing.

This costs RAM and speed — a headless browser is a heavier tool than an HTTP client — but it buys resilience. When a platform rotates its JavaScript signing scheme, request-forging scrapers die instantly; a browser-based crawler mostly keeps working.

The runtime requirements follow from the architecture: Python 3.11 (3.9.6 is the documented floor), Node.js 16 or newer, and the Chromium build that Playwright installs. Each supported platform gets the same feature set: keyword search, crawling specific posts by ID, second-level (nested) comments, creator homepage collection, login-state caching, and optional IP proxy pool support. There is also a comment word-cloud generator for quick exploratory analysis, and a WebUI (a FastAPI backend plus a Node frontend) that arrived in late 2025 for people who prefer clicking to flags.

Collected data lands wherever you point it: CSV, JSON, JSONL, or Excel files for small jobs; SQLite or MySQL when you want to query across crawls. On a single VPS, SQLite is the right call — one file, no daemon, trivial backups.

The license, plainly

MediaCrawler ships under a Non-Commercial Learning License 1.1. That is not MIT with extra steps — it is a genuine restriction: use is limited to learning and research, commercial use requires written consent from the author, and the license text explicitly prohibits large-scale crawling or anything that disrupts platform operations. If your use case is a commercial monitoring product, this is not your tool. If it is a dissertation, a newsroom investigation, or learning how browser automation works, you are the intended audience.

The honest comparison points are writing your own Playwright scripts per platform (weeks of work, permanent maintenance) or paying a commercial data provider (four-figure monthly bills). MediaCrawler is the middle path: free, maintained by someone else, and constrained to research use.

How to use MediaCrawler

Core concepts

Four nouns cover the mental model:

• Platform — which site to crawl, passed as --platform with a short code: xhs, dy, ks, bili, wb, tieba, zhihu.
• Crawler type — --type search crawls by keyword, --type detail fetches specific posts by ID, and creator mode collects a profile's output.
• Login state — most content requires a logged-in session. You authenticate once (QR code scan or imported cookies) and MediaCrawler caches the browser context, so subsequent runs skip the login dance.
• Storage backend — where results go, configured in config/base_config.py alongside keywords, platform specifics, and crawl limits.

Day-to-day workflow

A working session is short: edit the config to set your keywords and limits, kick off uv run main.py with the right flags, and come back when it finishes. The crawl writes rows as it goes, so a run interrupted halfway still leaves usable data. Most people converge on a small set of standing configs — one per research question — and rerun them on a schedule to build a longitudinal dataset.

Comment collection deserves a special mention because it is the expensive part. Fetching a post is one page load; fetching its full comment tree is many. MediaCrawler ships with comment crawling disabled for exactly this reason.

Integrations and extensibility

The 2025-era WebUI wraps the crawler in a FastAPI backend (uvicorn api.main:app --port 8080) with a separate Node frontend — useful on a workstation, but on a server treat it as a dev tool, not something to expose publicly. For pipelines, the JSONL and SQLite outputs are the integration points: point pandas, DuckDB, or your NLP tooling at them directly. Proxy pool support exists for people whose institutional network requires it; configure it rather than hammering from one address if your crawls are anything beyond small.

Backup and operational reality

Two things on disk matter. First, the data directory (or SQLite file) holding your crawled results — that is your actual research output, back it up off-box. Second, the cached browser login state: losing it means re-authenticating with a phone in hand, which is annoying when the server is headless. Everything else — the repo, the venv, Chromium — is reproducible in minutes. Expect to git pull with some regularity; when a platform changes its frontend, the fix lands upstream and stale checkouts silently collect less data.

Quick Start Guide

Tested flow for a fresh Debian 12 VPS. Total time: about fifteen minutes, most of it Chromium downloading.

1. Provision the box

Sign up for the VPS 1000 G12 at netcup.com using coupon 5799nc17853181290 for one free month. Pick the Debian 12 image, drop in your SSH public key on creation, then SSH in and lock the box down:

# /etc/ssh/sshd_config.d/00-hardening.conf
PasswordAuthentication no
PermitRootLogin prohibit-password

systemctl restart ssh
apt update && apt -y upgrade

2. Install system dependencies

MediaCrawler needs Python 3.11 (Debian 12's default), Node.js ≥ 16 (Debian 12 ships 18), and uv:

apt install -y git nodejs npm python3 python3-venv curl
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.local/bin/env

uv handles the Python toolchain and virtual environment in one tool — no manual venv juggling, and installs are fast enough that rebuilding from scratch is a non-event.

3. Clone and install MediaCrawler

git clone https://github.com/NanmiCoder/MediaCrawler.git
cd MediaCrawler
uv sync
uv run playwright install --with-deps chromium

uv sync resolves and installs the Python dependencies into a project-local environment. The --with-deps flag matters on a server: it pulls in the system libraries Chromium needs to run headless on a box with no display — skip it and the browser dies on launch with a cryptic missing-library error.

4. Configure the crawl

Open config/base_config.py and set the handful of values that define your run: the search keywords, crawl limits, whether comments are collected (ENABLE_GET_COMMENTS), and the storage backend. Start with SQLite and conservative limits:

# config/base_config.py — the values worth touching first
KEYWORDS = "your search terms"
ENABLE_GET_COMMENTS = True
CRAWLER_MAX_NOTES_COUNT = 50

Since the server has no display, also make sure the browser runs headless — the config exposes a toggle for it. Keep the first crawl small; you want to validate the pipeline, not download a platform.

5. Log in and run the first crawl

uv run main.py --platform xhs --lt qrcode --type search

QR login on a headless server is the one genuinely awkward step: the code needs to be scanned with the platform's mobile app, and there is no screen to show it on.

Once authenticated, the crawl runs on its own and writes results as it progresses. Check the output directory (or query the SQLite file) after a few minutes to confirm rows are landing.

6. Schedule unattended runs

Crawls are batch jobs, not daemons — a systemd timer fits better than a service that restarts forever:

# /etc/systemd/system/mediacrawler.service
[Unit]
Description=MediaCrawler scheduled crawl

[Service]
Type=oneshot
WorkingDirectory=/root/MediaCrawler
ExecStart=/root/.local/bin/uv run main.py --platform xhs --lt cookie --type search

# /etc/systemd/system/mediacrawler.timer
[Unit]
Description=Run MediaCrawler daily

[Timer]
OnCalendar=daily
RandomizedDelaySec=1h

[Install]
WantedBy=timers.target

systemctl enable --now mediacrawler.timer

The randomized delay keeps your crawl from firing at the exact same second every day — both polite and less conspicuous. Resist the urge to schedule hourly: the license prohibits large-scale crawling, and platforms rate-limit accounts that behave like machines.

7. Back up what matters

# from your workstation — pull the data and login cache
rsync -az root@your-server:/root/MediaCrawler/data/ ./mediacrawler-backup/

Run this after significant crawls or drop it in cron. The dataset is the thing you cannot recreate — a deleted post is gone from the platform, and your copy becomes the only record.

Choosing the Right Netcup Server for cheap MediaCrawler hosting

The sizing question for MediaCrawler is really a Chromium question. The Python process is a rounding error; the headless browser is where the RAM goes, typically 1–2 GB per instance once a session has some history. Add the OS, and optionally MySQL if you outgrow SQLite, and the tiers sort themselves out.

VPS 500 G12 — the minimum viable crawler

Specs: 2 vCPU, 4 GB RAM, 128 GB NVMe, traffic included. Around €6/month (verify on netcup.com).

Why it works for MediaCrawler: one platform, one browser instance, SQLite storage, scheduled runs — 4 GB covers that with room left over. It gets tight if you enable comment crawling on large result sets or try to run two platforms concurrently. Fine for a student project or a proof of concept; expect to feel the ceiling if the research grows.

Coupons (each pops a different code at build time — €5 off any order, since this tier has no dedicated voucher):

• 36nc17718015540
• 36nc17718015545
• 36nc17718015549

VPS 1000 G12 — recommended for most MediaCrawler deployments

Specs: 4 vCPU, 8 GB RAM, 256 GB NVMe, traffic included. Around €10/month (verify on netcup.com).

Why it works for MediaCrawler: 8 GB means Chromium, the OS, and a comment-heavy crawl coexist without swap; 4 vCPUs absorb the page-render spikes; and 256 GB NVMe holds years of text-centric datasets. This is the tier where you stop thinking about resources and start thinking about your research question — which is the point.

Coupons (each pops a different code at build time):

• 5799nc17851668510
• 5799nc17830589190
• 5799nc17843120991

VPS 2000 G12 — for multi-platform and MySQL setups

Specs: 8 vCPU, 16 GB RAM, 512 GB NVMe, traffic included. Around €19/month (verify on netcup.com).

Why it works for MediaCrawler: 16 GB comfortably runs several browser instances plus a MySQL server, which is the shape of a project crawling three or four platforms on staggered timers into one queryable database. Also the right call if the same box doubles as your analysis environment — pandas and a Jupyter kernel over a few million comment rows want that headroom.

Coupons (each pops a different code at build time):

• 5800nc17802654091
• 5800nc17853363630
• 5800nc17851884640

RS 1000 G12 — dedicated cores for sustained automation

Specs: 4 dedicated cores (AMD EPYC 9645), 8 GB RAM, 256 GB NVMe. Around €13/month (verify on netcup.com).

Why it works for MediaCrawler: VPS vCPUs are shared, and browser automation is the rare "cheap hosting" workload that keeps a core genuinely busy while pages render. The RS line's dedicated Zen 5 cores make long crawls noticeably more consistent — no throttling when a noisy neighbour spikes. For €2–3 over the VPS 1000 G12, it is the pick when crawls run daily and their completion time matters, and the coupon below is worth two free months rather than one.

Coupons (each pops a different code at build time):

• 5159nc17718015443
• 5159nc17718015441
• 5997nc17843120990

Which one should you pick? If you are trying MediaCrawler for the first time, the VPS 500 G12 proves the concept for the price of a coffee — just accept you may migrate later. If you are running it seriously for a research project, take the VPS 1000 G12 and stop thinking about it; single-platform, comment-enabled crawls fit it exactly. If you are crawling multiple platforms on daily timers, or your analysis lives on the same box, go VPS 2000 G12 or RS 1000 G12 — the former for RAM, the latter for consistent long-run performance.

Conclusion

You now have a repeatable path from empty VPS to unattended, scheduled social media data collection: Debian 12, uv, Playwright's Chromium, a cookie-based login, a systemd timer, and off-box backups of the one directory that matters.

The bill is small. The recommended VPS 1000 G12 runs €10.37/month, and the coupon codes above make the first month free — so a semester-long research project costs roughly 50 €, less than a single hour of most commercial data-API pricing. If you land on a different product entirely, the €5-off-any-order code 36nc17718015548 applies storewide.

Two things deserve ongoing attention once you are running. First, stay current: git pull before blaming yourself when a crawl comes back thin, because platform frontends change and the fixes land upstream weekly. Second, stay within the lines: the Non-Commercial Learning License and the platforms' own terms both bound what this setup is for. Small, polite, scheduled crawls for research are the design center — treat the rate limits as part of the methodology, not an obstacle to it. More self-hosting guides live on the blog.