[mikebanderson]

I want to screen scrape volume and quote data from an online broker I have an account with for a hobby trading system I'm building. The quote/volume screen uses frames and I must login to get to it. Also, when I try to access the quotes/volumes screen using a URL the site logs me out and takes me to the homepage, i.e. it is using JavaScript to prevent systems from hitting links directly and scraping the data.

I need a Java framework/utility that will let me write a program to login to the site and step by step access the quotes/volumes screen as though a human user with a browser were doing the same thing so I can scrape the screen. Can someone please recommend a utility that can do this?

[glennji]

It really sounds like they're going out of the way to stop things like that -- so even if you get it working you're probably going to be breaking some user agreement, and you're at the mercy of them changing their app and breaking yours...

Most brokers have a quotestream service (for a price), and if it's just for a "game" or "hobby" you might be able to find time-delayed quotestreams for free.

Sorry if that's not what you wanna hear ;-)

[Haraldo]

You might be able to do it using this: HttpComponents - HttpComponents Overview or, more specifically the previous stable version HttpClient - HttpClient Home.

[msk_1980]

Try using web harvest, its an open source java based scripting that allows you to code in xml to login into secure websites and scrap data. It also allows to do additional processing such as saving them to database or additionally do any more processing with the data. It is fully developed in java.

Hope that helps you.

[jwenting]

Contact the data owners. Stealing their data is NOT the way to go (and screen scraping amounts to stealing), and is a violation of the law.

All data owners of such data have APIs available (usually at a price, that data doesn't come cheap).

[mikebanderson]

I decided to just scrape everything from Yahoo finance and live with the 15 minute delay on the options data. I'm manually parsing the HTML, it only took about 1 night to write a parser.

Separate question, any suggestions on tuning my parser? It takes about 90 minute to pull the options quote pages, parse them and insert the data for 5000 symbols into mysql. I guess this isn't too bad, but I'd be happy if I could improve the speed. I haven't done much tuning outside of using eclipses profiling tools. Any suggestions on how to approach tuning this?

[tofarr]

The Selenium test framework has java APIs - it may be possible to use it to screen scrape, though it would do it through your browser (You would see your browser pop up and go and retrieve the data.)

I dunno about the legal issues, and you are at the mercy of the site owner changing their layout...

[jwenting]

The legal issues are simple: it's almost always against the TOS of a site to redistribute its content.

That may not be a criminal case in most countries, but it's certainly a civilian case of breach of contract.

[SurfMan]

Originally Posted by mikebanderson

I want to screen scrape volume and quote data from an online broker .[...] i.e. it is using JavaScript to prevent systems from hitting links directly and scraping the data. [...] so I can scrape the screen.

Do you read anything wrong here?