Remove obsolete docs from web site and set up redirects; add robots.txt; add README for web site admins