blogging and scraping

install firewatir on ubuntu 10.04

Mon, 12 Jul 2010 22:53:04 +0000

I am a newbie with ubuntu, I tried and failed several times to install firewatir on ubuntu 10.04, and finally got following codes by google:
sudo su apt-get install build-essential apt-get install ruby rdoc libopenssl-ruby wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz tar zxvf rubygems-1.3.7.tgz cd rubygems-1.3.7 ruby setup.rb ln -s /usr/bin/gem1.8 /usr/local/bin/gem gem install firewatir

Frankly I did not understand the function of the first 2 apt-get, I just copy it for free.

The poor support of the firewatir 1.6.5 to Chinese characters

leleba — Thu, 07 Jan 2010 09:05:41 +0000

I cannot figure out how to enter the Chinese characters into the textfield of a webpage by firewatir 1.6.5, I tried to change the encoding of the .rb file and the encoding of the firefox browser, but failed.

require ‘rubygems’
require ‘firewatir’

ff = FireWatir::Firefox.new
ff.goto “http://www.google.com“

sleep 1
ff.text_field(:name, ‘q’).set(‘?’)

For Watir 1.6.5, the Chinese charaters can be supported by replacing the script WIN32OLE.codepage = WIN32OLE::CP_UTF8 with WIN32OLE.codepage = WIN32OLE::CP_ACP in the file of C:\Ruby\lib\ruby\gems\1.8\gems\watir-1.6.5\lib\watir\win32ole.rb

Check the version of the watir and the firewatir

leleba — Thu, 07 Jan 2010 08:35:14 +0000

We can check the version of the watir and firewatir by executing an rb file with the following script:

require ‘watir’

require ‘firewatir’

puts Watir::IE::VERSION

puts FireWatir::Firefox::VERSION

It is safe require “rubygems” in the begining of the watir codes

leleba — Mon, 28 Dec 2009 04:39:10 +0000

Maybe it is safe to add the code of require ‘rubygems’ in the beginning of the watir codes, otherwise, strange results may happen sometimes .

the fucntion ie.collect in watir

Tue, 10 Nov 2009 00:44:51 +0000

Just copy a coding found in the internet for future reference,

$ie.divs.collect {|div|
if div.class_name == “data”
puts div.text
end
}

The fast way to fill a textfield in watir

Mon, 09 Nov 2009 15:30:55 +0000

It is quite slow to fill a text field by ie.text_field().set(content), the fast way is by ie.text_field().value = content.

There is no property of class for iframe in watir

Mon, 02 Nov 2009 15:21:24 +0000

I encountered an iframe in a webpage which say “iframe class = “the_iframe”, I tried but failed to access the iframe by:
ie.frame(:class, “the_iframe”)

and then I figured out there is no property of class for iframe in watir, and find the iframe by index:
ie.frame(:index, 1)

we can use ie.show_frames to find how many frames are there in a webpage.

Be careful to “include Watir”

Thu, 29 Oct 2009 06:12:39 +0000

When we have the “include Watir”, then we cannot load another ruby file.

I spent more than 1 hour to figure out this.

Multiple ruby files work together

Thu, 29 Oct 2009 04:06:05 +0000

When writing the first few Ruby programs, there is a tend to place all the code in a single file. But as time passes by and the Ruby programs grow, it is nature that at some point we have to break our code up into logical groupings and place each group in a separate file or files.

following is an examle:

in foo.rb:
puts “it is foo”
$foo = 3 # $ for global variable

in bar.rb:
puts “it is bar”
$bar = 3 # $ for global variable

in test.rb:
require ‘foo’ # pay attention –no “.rb”
load ‘bar.rb’ # pay attention — “.rb” is there
test = 1 + $foo + $bar

when execute test.rb, it shows :
it is foo
it is bar
9

watir to count the links in a search list on a webpage

Tue, 27 Oct 2009 11:10:25 +0000

Some times I need to check the amount of the links in a search list, at first I try and failed by:
ie.links(:id, /bra bra/).length

and then I find a solution by:
ie.div(:id, “searchList”).links

and then by googles, I find a better solution:
links_searchlist = ie101.links.find_all { |link| link.class_name == ‘permalink’ }
puts “the links in the searchlist:”
puts links_searchlist.length

and in another case, following code also works:
searchlistlinks = ie102.links.find_all { |link| link.id =~ /hitURL/ }
links_searchlist = searchlistlinks.length
puts links_searchlist

following can also work:
searchlistlinks = ie33.links.find_all { |link| link.href =~ /something/ }
but be careful that the following (url) will not work:
searchlistlinks = ie33.links.find_all { |link| link.url =~ /something/ }

Recently I found the best solution by google:
links = ie51.div(:id, ‘threadlist’).html.scan(/something/).count
puts links