Code Sample for "Using S3, EC2, SQS, Lucene and Ruby for Web Spidering"

Sample Code & Libraries>Ruby>Code Sample for "Using S3, EC2, SQS, Lucene and Ruby for Web Spidering"
Community Contributed Software

  • Amazon Web Services provides links to these packages as a convenience for our customers, but software not authored by an "@AWS" account has not been reviewed or screened by AWS.
  • Please review this software to ensure it meets your needs before using it.

Details

Submitted By: Robert Dempsey
AWS Products Used: Amazon EC2, Amazon S3, Amazon SQS
Language(s): Ruby
License: GNU General Public License (GPL version 2)
Created On: December 23, 2007 3:02 PM GMT
Last Updated: September 21, 2008 8:51 PM GMT
Download
This code sample goes along with the "Using S3, EC2, SQS, Lucene and Ruby for Web Spidering" article. The article and accompanying code show how Ruby and Rails developers can use freely available tools and simple techniques to spider sites and then, using Lucene, index and search the results.
©2014, Amazon Web Services, Inc. or its affiliates. All rights reserved.