A Method for Identifying and Confirming the Use of URL Filtering Products for Censorship

All confirmed URL filtering deployments—McAfee SmartFilter in UAE and Netsweeper in Yemen, UAE, and Qatar—block content across at minimum six of seven tested human-rights-sensitive categories: media freedom, human rights, political reform, LGBT, religious criticism, and minority groups/religions. Netsweeper in both Qatar (Ooredoo) and UAE (Du) blocks all seven categories. This content is protected under Article 19 of the Universal Declaration of Human Rights.

§5, Table 4 policy keyword-filtering saae

In YemenNet (AS 12486), URL filtering was observed to be intermittently offline: proxy URLs accessible in one test run were blocked in others and vice versa. A prior ONI measurement found a Yemeni ISP running Websense whose filtering ceased entirely when concurrent user count exceeded the product's license capacity. This inconsistency required larger URL test sets and repeated measurement runs to establish blocking with high confidence.

§4.4 evaluation keyword-filteringmeasurement-platform

In every ISP where URL filtering was empirically confirmed, the 'proxy anonymizer' category was actively blocked. Netsweeper blocked 6/6 submitted proxy domains in YemenNet (AS 12486), 5/6 in Du UAE (AS 15802), and 6/6 in Ooredoo Qatar (AS 42298); McAfee SmartFilter blocked 5/5 anonymizer-category submissions in Etisalat UAE (AS 5384). Blue Coat in UAE and Qatar did not confirm—Etisalat appears to use SmartFilter for URL filtering atop a Blue Coat proxy appliance for traffic management.

§4.3–4.5, Table 3 detection keyword-filteringip-blocking ae

URL filtering appliances are frequently misconfigured to be externally visible on the global Internet, enabling passive identification via Shodan keyword searches on product-specific HTTP headers and management console paths (e.g., 'cfru=' for Blue Coat, '8080/webadmin/' for Netsweeper). This technique discovered previously unknown installations in Finland, Sweden, Philippines, Thailand, Taiwan, Argentina, and Chile, as well as large U.S. ISPs including AT&T, Verizon, Bell South, Comcast, and Sprint.

§3, Figure 1 evaluation measurement-platformdpi

The paper presents a repeatable method for confirming which specific URL filtering product is used for censorship: create test domains under researcher control, submit a subset to the vendor's public URL categorization interface, then retest within 3–5 days to observe whether submitted domains become blocked. This technique confirmed McAfee SmartFilter in UAE (Etisalat, AS 5384) and Saudi Arabia (Bayanat Al-Oula AS 48237, Nournet AS 29684), and Netsweeper in Qatar (Ooredoo AS 42298), UAE (Du AS 15802), and Yemen (YemenNet AS 12486).

§4.2, Table 3 evaluation keyword-filteringmeasurement-platform saae