Migrating content from websites hosted on the same domain

Recently came a situation where I needed to copy manually a small amount of content when migrating sites, on different servers but both configured to only serve the website in question from the same domain name. It turned out to be a little trickier than expected to access the two sites on one machine.

The tools and such I used may change over time, so this guide cannot be fully relied upon. While the commands and such worked on a Windows machine, they should be transferable to run on most operating systems.

mitmproxy

I ended up using mitmproxy in order to have some control over requests on the browser which was targeting the server the DNS wasn’t currently pointing towards. While there was an example at some point, it seems “DNS Spoofing” isn’t a popular feature in mitmproxy.

I ended up adapting a script and producing a custom plugin, which i’ve annonymized as follows:

from mitmproxy import http


def request(flow: http.HTTPFlow) -> None:
	# pretty_host takes the Host header of the request into account,
	# which is useful in transparent mode where we usually only have the IP
	# otherwise.
	if flow.request.pretty_host == "somesite.net":
		flow.request.host = "some_other_host_on_other_server.com"
		flow.request.headers['Host'] = 'somesite.net'
Code language: Python (python)

To run mitmproxy with this file it can be ran like so:

$ mitmweb.exe -p 1024 --web-port 1025 -s C:\Users\Joe\mtim_migrate.pyCode language: Shell Session (shell)

Firefox

Firefox has a lot of useful options under the hood. One of them is the ability to run multiple Firefox instances on your machine from different profiles. And being able to override the system proxy details.

While I’m sure there’s likely a way to get Google Chrome to do the same, I couldn’t figure out how to do this without spending many hours on this. And having already had tried 6 different extensions to acomplish this task, Chrome was out of the window.

To run Firefox with a new profile, at least on Windows:

cd "C:\Program Files\Mozilla Firefox"
.\firefox.exe -profile C:\Users\Joe\AppData\Local\Temp\fftemp\Code language: Shell Session (shell)

From here you can configure the proxy through mitmproxy and have one browser for one server and another for the other.

Summary

It’s a bit of a faff, but following this guide if you’re ever in this strange situation of migrating sites on the same domain, it should help get the task completed. And yes, I could pull the database, do some work and re-upload it but when there’s 20 items which are mostly copy and paste fields between two very different database schemas, it’s really not worth the effort.

Please send any comments through the contact form.

---