Input would be a web page. First it would find broken links. Then it would search to see if the missing page had moved somewhere, e.g. hacking off bits from the RHS and then reconstructing the URL.
If it could not reconstruct the URL, it would turn to search engines using linkto: and other techniques. If it still could not find the URL, it might suggest an alternative based on keywords.
Finally, it should actually output a copy of the web page with the links fixed, that the user may simply copy it over the old one, or selectively cut and paste.
Possible enhancements:
- Take as input an entire website. Run overnight. Generate a report of the broken links listed in order of the site with the most broken links to it. Because often a change must be made globally through many of your pages. e.g. "http://akebono.stanford.edu/yahoo/" has moved to "http://www.yahoo.com/" - change needed throughout the following 53 of your web pages.
- Build a script that, when run, makes these changes.
- Nice user interface output, showing old page, new page, and selective buttons to press to "Fix link" on disk. User in control at all times.
- Use archive.org to find old page. Then search in Google to find where it has gone to.
- Suggest alternative links. e.g. Page on Shakespeare has gone. Suggest wiki/Shakespeare instead. "Click this to make change". User always in control.
Search engines have a linkto: facility,
so you can see who links to you,
but it takes forever to browse the list
so you don't bother.
This would be a
standalone program to
find all pages that link to the user or reference them,
download all these pages (will take a long time),
perhaps sort them by the topic or page referenced,
and present them all in a nice readable output,
with all the references highlighted (as in Google's cache).
How to highlight a phrase using the bold tag (view source code to see how this is done).
| How to highlight a phrase using | tables | (view source code to see how this is done). |
Extract keywords from page. (How? Need idea of dictionary frequency.)
Use search engines to find similar pages on Web.
Implement as CGI script so that I can automatically add a "What is like this?" link at the top of every URL.
- Web page enhancer - Links
CGI script. Input is a web page.
Extracts all proper nouns
(perhaps words that are not in the UNIX dictionary
- see "man spell"
and /usr/share/lib/dict/).
For all these proper nouns, link them to a search engine
so you can just click on the word
to search for it on the web.
Output is this new web page with all these words linked.
Enhancement - Perhaps check first to see if you can find a
Yahoo category
for the word.
If so, link to it.
If not, link to a search engine.
Perhaps the CGI script just functions as a spell-checker.
The basic enhancements of this and the other CGI scripts below is that images should still display from the original page. Also when links are followed, the new page should be put through the CGI script as well.
CGI script. Input is web page.
Tries to extract all the locations on the page
and link them to some online map.
Output is this new web page with all these words linked.
Difficult bit is identifying the place names
(as opposed to just any proper nouns).
Perhaps can identify Irish and UK locations
by the presence of a trailing Co.--- (one of the 32 counties)
or --Shire (one of the UK counties).
Enhancement might be to actually include the map on the page
(make sure no copyright infringement!).
Alternatively, link to a choice
of online maps.
Also see basic CGI enhancements above.
As above. Identify all dates on the page,
in any format, and, using "cal" program,
convert them all to format showing the day of the week,
like: "Sun 1st Nov 1818".
Enhancement - Link each month to a second CGI script
which displays calendar for that month.
Also see basic CGI enhancements above.
For all of these CGI scripts, I have many web pages with proper nouns, locations and dates, that could be used as test cases.
The Liquid Information project
(or via here
or here)
does something like this.