dnsbl_redir A Squid DNSBL Redirector program Current Version: 1.0 Last Updated: March 24, 2007 Site Refresh: January 24, 2010 (Nothing to revise, still works and still being downloaded!) Contents: Description Why make another Redirector? And why one like this? How we solved the large list problem Using a RHS DNSBL Creating and Maintaining the blocklist Source History Downloads Installation The error HTML Page What about the DNSBL itself? RBLDNSD Testing your DNSBL Configuring Squid to use the Redirector Credits, Links, License and Me Description: dnsbl_redir is a shamelessly derived redirector based on the asqredir redirector written by Thomas Zippo < thomas at zippo dot ch >. Thanks! Its been rewritten to use a DNSBL list (RHS type) to check for and block sites listed in the DNSBL. It will redirect your users to the page/site listed in the top of the dnsbl_redir.h file. dnsbl_redir is written in C. It consists of one small source file and a small(er) .h file. It has been tested and runs on the current Squid versions. It performs very well on Linux (RedHat/Fedora), BSD and technically should compile and run on any gnu/linux. Back to the Top Why make another Redirector? And why one like this? This redirector solved a problem we had here at our ISP and hosting company. We not only provide proxy service for our customers, we also provide some filtering of content, should they want to use it (voluntary and they are able to turn it on and off at will). Over time filter lists of domains, URLs and regex lines get long, files grow and memory and CPU usage grows along with it. The result - our Squid slowed down, and started using more and more resources to service the same number of people. Unacceptable. The problem was the filter lists were huge and the scripts/plugins used to make them work had to deal with them. Mind you, the plugins worked and the scripts functioned as stated - but at the expense of longer and longer lookup times and long response times from the Squid proxy when it was busy. Back to the Top How we solved the large list problem Short of re-writing scripts, pruning the lists and redesigning the methods we currently used - why not off load as much of the lists as possible into an external process more suited to the task? Why not a lightweight and efficient DNSBL system to manage and list the "forbidden" URLs? DNS requests are quick, small and even a lot of them will not load down a server like script lookups will. To make a long story short - I did it and it worked. By using this redirector and the DNSBL to manage and serve the long lists of URLs/Domains we wanted to filter, we lowered the memory squid and Squidguard required, we dramatically sped up the lookup and response times of the proxy and our users are happy. NOTE: The lists of domains and URLs we were filtering (using SquidGuard) was swamping the 2gHz AMD processor and using a lot of memory at times. That system worked as designed, but at a price. We took the Squidguard 'domain' files only, converted them for rbldnsd and by using this redirector and rbldnsd, the DNSBL process fully loaded with all the Zone files is using about 1.6-1.7% of the memory of the 512MB available to it. CPU usage does not even show on 'top' when its running. Squid runs much better now, is noticeably faster and still blocks what needs to be blocked. We still use Squidguard but with a much smaller set of block information. Your mileage may vary depending on your installation, size of your Squid server and number of users. Back to the Top Using a RHS DNSBL Why a RHS (Right Hand Side) DNSBL? Mainly for the ease of converting the lists of Domains and URLs. A wonderful side effect is that it makes far more sense to us humans to block the.site.is.blocked.com then com.blocked.is.site.the in a list, and its easier to manage. The current list is created from SquidGuard lists and other lists we found on the 'net. We compiled a couple other smaller ones from local input. To block, say: www.nasty-site.com the DNSBL would have an entry like: .nasty-site.com That exact listing will block "anything.nasty-site.com" "nasty-site.com" along with anything to the right - such as /index.html. All derivatives of that site-name get blocked with a 127.0.0.2 or similar response being returned. Read the example http://www.frws.com/squid_block/down-loads/rbldnsd.squidblock.example file to get a better understanding. The creation of a list from your own sources is left up to you, but its simple enough to write a Perl or BASH script to take one of the URL block lists and make it usable by RBLDNSD. There is a small BASH script here http://www.frws.com/squid_block/down-loads/convert.squidguard that will take a SquidGuard 'domain' list and convert it for use with this redirector. There is also an example of the 'top' of the Zone file that you have to modify and then you can use to complete the Zone file. The script will take a 'domains' file found in the current directory, add a '.' to the beginning of each line and write that file to 'new-domains'. The script looks like this: #!/bin/bash grep '\.' domains | awk '{print "."$1}' > new-domains 'cat' the modified 'top' and the converted 'new-domains' list together to create a complete Zone file. Back to the Top Creating and Maintaining a block list To maintain your own blocklist, all you have to do is add/remove/edit the listings in the created Zone file. There is no need to stop and restart the DNSBL unless it makes you feel better. RBLDNSD checks the "zone files" every 60 seconds or so and will reload and increment the list sequence if it senses the zone has changed. NOTE: As far as the block list on our website goes - simply put - we will not guarantee an up-to-date list! Treat that list as a sample list, or at least one with errors. Grab a more recent one from another site. There is a well kept SquidGuard based block list and other information at: http://cri.univ-tlse1.fr/documentations/cache/squidguard_en.html These gentlemen do a fine job, and their lists look current and work well in our environment. To make it simple, the Zone file and block list you need can look similar to this: $SOA 86400 squidblock.yourdomain.net webmaster@yourdomain.net 1 3600 900 1209600 43200 $NS 86400 dns.yourdomain.net dns2.yourdomain.net squidblock.yourdomain.net # # :2:Squid Domain Block # .domain-to-block.com .another-domain-blocked.net .yet-another-one.info And just keep adding and removing these domains as needed. Back to the Top Source History: 2006-02-02 Released v1.0 Code cleanup Created Webpage Added BASH script to create usable zone files 2005-11-14 Added: NOTHING MOVED site to http://www.frws.com/squid_block 2004-07-20 Released v0.2 Added: dnsbl_redir.h for easier configuration Added: Makefile, README and GNU License Added: Posted the .gz on the website 2004-03-15 Added: v0.1 Added: Initial release (not public) To Do: (eventually...) Add a conf file of sorts Add multiple DNSBL choices Rewrite some of the search code so that searches are more sensible Back to the Top Downloads: Current source, license, scripts and a sample block-list can be found at: http://www.frws.com/squid_block/down-loads Back to the Top Installation: Change to the directory where the downloaded package is located and unpack it: cd /home/your-dir/downloads tar -zxvf dnsbl_redir-0.2.tar.gz cd dnsbl_redir-0.2 Make changes to the two entries in dnsbl_redir.h. The DNSBL domain name - good to have one of these. // The DNSBL you will query char DNSBL[] = ".somednsbl.at-a-domain.com"; // The page to redirect your users to if the site is blocked char RedirectTo[] = "http://www.complete-url.com/squid-error-page.html"; Then type: make To install the resultant binary to /usr/local/bin/dnsbl_redir Just type: make install Note: If you wish to compile it on your own and install it manually: 1. Just cd to the directory with the source code 2. Edit the dnsbl_redir.h file as per above 3. type in gcc -o dnsbl_redir dnsbl_redir.c 4. Copy the dnsbl_redir application to the directory of your choice! You can path the squid.conf file entry to where the binary is built or manually copy it anywhere. It needs nothing else. Back to the Top The error HTML Page Just create an HTML page with appropriate contact info, and the reason and post it on a web-accessible site. The one we use can be found HERE: squid_blocked.html PLEASE do not link to this error page - WRITE your own, its not tough! Use that as an example. Back to the Top What about the DNSBL itself? THAT is the tricky part. We chose to use RBLDNSD and run it on the same server that runs squid. A GZIPPED version of Zone file we use is at: http://www.frws.com/squid_block/down-loads/rbldnsd.squidblock.gz NOTE: This file can be 10MB+ in size at times! Not huge - but large for a zone file! Unzip this file to access it. VERY IMPORTANT!!! Do not use this file as-is. Edit the Top portion of the file to reflect your DNSBL and DNS/Domain information prior to using it! Setting up and running this DNSBL is not hard, though setting up the DNS to use it may be tricky. If you use BIND 9.x/8 as your main DNS server, this is how we did it: 1. Created a zone called squidblock.yourdomain.net 2. Entry as follows in the whatever-yourdomain-is SOA file: squidblock IN A 10.0.1.1 (use a real IP here) squidblock IN NS squidblock.yourdomain.net 3. Increment the Zone file (however your DNS server makes you do this) 4. Add the following to your named.conf file (again for BIND) zone "squidblock.yourdomain.net" { type forward; forward_only; forwarders { 10.0.0.1; }; (same IP as above) }; 5. Now reload the DNS server and watch for errors. 'rndc reload' for Bind 9 (Insert your real Domain name and working IPs as needed!) 6. At this point you should be able to have that Zone working! Back to the Top Setup RBLDNSD: You can grab RBLDNSD at: http://www.corpit.ru/mjt/rbldnsd.html This should be a simple build! READ the docs! And compile it as stated in those docs and/or the website. There is an example startup start.squidblock script at: http://www.frws.com/squid_block/down-loads/start.squidblock It looks like this: #!/bin/bash ## where we installed the binary cd /home/dnsbl ./rbldnsd -b your.ip.goes.here -u dnsbl:dnsbl -r /home/dnsbl -l ./dnsbl.log squidblock.yourdomain.net:dnset:rbldnsd.squidblock This file needs some editing also - read it carefully. 1. The IP after the -b is the IP it will bind to - change it to your server's IP 2. The -r is the directory where the RBLDNSD binary resides. 3. The -l is the log file if you want one 4. This part: 'squidblock.yourdomain.net:dnset:rbldnsd.squidblock' is your squidblock domain name, the :dnset: rule and the name of the zone file. That is about it. Start the rbldnsd using that script and watch for errors in the /var/log/messages file and on the screen. Read the RBLDNSD instructions and see how it works! Add the script name and path in your /etc/rc.d/rc.local file or other start-up script so it starts when your server boots once you confirm its working correctly. Back to the Top Testing the DNSBL: As follows:. 1. Using an URL from the Zone file, simply type in: host www.thaturl.com.squidblock.domain.net 2. And it should come back with a 127.0.0.2 if all is well. 3. If not, try adding the IP of the DNSBL server to the end of the host command. 4. Check your DNS setup for the domain its in, and see if all is well. Back to the Top Configuring Squid to use the Redirector: You will need to specify dnsbl_redir as a redirect_program in the squid config file (typically squid.conf) Like so: redirect_program /full/path/to/dnsbl_redir NOTE (10/1/2007): We use the 'wrapzap' script on our main Squid proxy so we can use dnsbl_redir AND SquidGuard at the same time. We place the dnsbl_redir at the 'top' so it gets called first, and SquidGuard is called at the bottom. Works like a champ! You can find 'wrapzap' on the Squid website, probably where you found this dnsb-redir information. If you have a busy Squid, change the redirect_children to 5 or more. (we use 8 on a moderately used Squid) redirect_children 8 Ports and ACLs and stuff: We use 2 ports on our Squid, we use 3128 for regular proxying and 3129 for filtered proxying. But that is a whole 'nother lesson for another day! As it sits right now, the DNSBL redirector will act as your squid.conf is set to make it act. If you wish more detailed instructions, search the Squid docs and the lists. If that does not provide the answers, then I would subscribe to and use the Squid Lists for your questions. In a pinch, email me (address below). Cannot guarantee a quick response and I am no Squid expert, but I got this to work. Anything that does not relate to this redirector, I would stick to the mainstream SQUID lists and documentation! You have been warned. :) Back to the Top Credits: Special thanks to Thomas Zippo for his redirector. And to the squid folks for their great product as well as our loyal users that use this daily. Links: http://www.frws.com/squid_block for the latest files and errata. http://www.squid-cache.org/ for the Squid program http://www.corpit.ru/mjt/rbldnsd.html for RBLDNSD http://cri.univ-tlse1.fr/documentations/cache/squidguard_en.html SquidGuard lists and other useful info License: The dnsbl_redir is free software. You can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation. Enjoy. (Never a guarantee it will work for you as it does for us!) Back to the Top Me: JP Potucek Owner - Systems and Network Administrator: http://www.frws.com http://www.frws.net Last Content Update: 1/24/2010 Website Copyright © 2004-2010, All rights reserved. Front Range Web Services, LLC. (FRWS) Website Copyright © 2010, All rights reserved. JP Potucek This site is Hosted at FRWS.