[Tutorial] Using HttpWebRequests to browse forums.

Tutorial: Using HttpWebRequests
Written by Jason of MPGH. None of this tutorial was taken from any other site. Do not post this tutorial elsewhere.

Hey guys, someone asked about simulating a forum login and navigation with HttpWebRequests. Midway through replying I realized I may as well just post a tut about it.

What you need:
My web scraping toolkit: http://www.mpgh.net/forum/33-visual-...s-library.html
LiveHTTPHeaders (Just google it, it's the first result)
A brain that works.
Fundamental VB knowledge: (where class-level declarations go, where Imports go)

Alright, now that you have the necessary equipment, it's time to start the coding.

First thing is first, we needa set up which namespaces to import and what class-level variables to declare. We need to add the WebClasses.dll as a reference before any of that, so go Project->Add Reference and then browse to wherever you saved my .dll.

Once that is referenced we need to import the WebClasses namespace, as well as the System.Net namespace. After the imports we'll make a class-level new instance of the WebClass class as well as a new CookieContainer.
See below:

[highlight=vb.net]
Imports WebClasses
Imports System.Net

Public Class Form1 '<< That is irrelevant, it will be the name of your current form so whatever it is, don't change it...this is just to show where to import/declare
Private wClass As New WebClass()
Private cCont As New CookieContainer()
[/highlight]

Now, you're going to need to get some info about the site so that you can login! It's time to get LiveHTTPHeaders crankin' (I'll be

calling LiveHTTPHeaders "LHH" from now on 'cos cbf typing).

Crash course on getting the info:

Step 1: Open up LHH in Firefox, uncheck the capture button and press "clear" if there is any text in there already
Step 2: Navigate firefox to your forum of choice (on the login page)
Step 3: Enter your forum credentials into the login box but DO NOT PRESS SUBMIT YET.
Step 3.1: [troll]email me your forum credentials[/troll]
Step 4: [NOTE: STEP 4+5 ACHIEVE BEST RESULTS WHEN DONE QUICKLY, SO READ BOTH BEFORE PROCEEDING] Go back to LHH and check the 'capture' box
Step 5: Swap back to the forum and click the login button
Step 6: As soon as the next page of the forum loads, go back to LHH and uncheck the 'capture'
Step 7: Scroll right to the top of LHH, if you did it quickly enough, the first or second entry should be from the forum and the METHOD should be "POST"
Step 8: Leave LHH open on that entry for now, we'll go back to it soon.

Okay, back to coding.

Now, we must begin our login method.

[highlight=vb.net]
Private Function tryLogin(ByVal username As String, ByVal password As String) As Boolean
End Function
[/highlight]

There's our function declaration. It's a function because we're going to 'try' and login (will return true if the login was successful, false otherwise)

Now to start filling out that function. Primarily we're going to be utilizing the postData function of the WebClass class, so we'll start working away at the fields required to construct that function.

Parameter One: SiteToPost
This is by far the easiest, go to LHH and the very first line of the entry will be the website link, right click it and select copy. for

the purposes of this tutorial my site will be "http://www.example.com/forum/login.php?do=login"
In vb:
[highlight=vb.net]
Dim site As String = "http://www.example.com/forum/login.php?do=login"
[/highlight]

Parameter Two: PostString
This one is the hardest one. Go to LHH, about midway down the entry there should be a line called "Content-Length" with an indented line beneath it. The indented line is the one we want, it contains all the info that was posted to the site. Depending on the forum, this can be a huge issue (some sites use random salts, some have strange encryptions...etc so yeah, this isn't a guaranteed fail-safe method. Either way, right click->copy the indented line.

For this example we'll say my indented line was simply "username=Jason&password=trololol", now we need to make this dynamic so other people can use their credentials, for your own you're going to have to search through the string till you see the credentials you used, then remove them and input the 'username' and 'password' variables as shown below
[highlight=vb.net]
Dim data As String = "username=" & username & "&password=" & password
[/highlight]

This means that your data string will depend on the credentials the user input.

Parameter Three Referer
This is optional, you can find the Referer in LHH, so if you want to set it..you can. All the referer does is basically describes what page directed you to the login page. I.e if you try to post on some forums without being logged in, that page will REFER you to the login page, once you login the site processes where you were sent from originally and puts you back there, that's basically what referer does.
[highlight=vb.net]
Dim referer As String = "" 'I'm not going to have a referer so I shall leave this blank.
[/highlight]

Parameter Four: Cookies
This is your cookiecontainer, you need this if you want to browse around the forum after you login with all your 'logged-in' priviledges. Remember that class-level CookieContainer we made before? That's what we'll use it for.

Okay, it's time to put that all together, with the postdata function included!

[highlight=vb.net]
Private Function tryLogin(ByVal username As String, ByVal password As String) As Boolean
Dim site As String = "http://www.example.com/forum/login.php?do=login"
Dim data As String = "username=" & username & "&password=" & password
Dim referer As String = ""

Dim requestString As String = wClass.postData(site, data, referer, cCont)
End Function
[/highlight]

You may be wondering why I declared requestString and why postData returns a string, this is to help you authenticate your login, the return string is the page source of the response from the website, this is how you validate the login: you test the resulting string to see if certain phrases or whatever are in there. For example, on VBulletin forums when you log in it comes up with the "thank you for logging in, Jason", so I'm going to test that here to validate the login

[highlight=vb.net]
Private Function tryLogin(ByVal username As String, ByVal password As String) As Boolean
Dim site As String = "http://www.example.com/forum/login.php?do=login"
Dim data As String = "username=" & username & "&password=" & password
Dim referer As String = ""

Dim requestString As String = wClass.postData(site, data, referer, cCont)
Dim retVal As Boolean = False 'default return is false
If requestString.Contains("Thank you for logging in,") Then retVal = True 'if the login was successful, change retVal to true

Return retVal 'return the result.
End Function
[/highlight]

So now you can test if you logged in correctly

[highlight=vb.net]
Dim loggedIn As Boolean = tryLogin("Jason", "trololol")
If loggedIn
MessageBox.Show("You were successfully logged in buddy!

")
Else
MessageBox.Show("Sorry, you were not logged in, please try again")
End If
[/highlight]

And there you have it. Because the CookieContainer was passed ByRef(erence) to the postData function, it will be full to the brim of Cookie goodness. After this, you can just make use of the CookieContainer and the GetPageSource function of the WebClass class to make requests to various pages on the forum.

[highlight=vb.net]
Dim forumSrc As String = wClass.GetPageSource("http://www.example.com/forum/33-visual-basic/", cCont)
[/highlight]

That is all from this tutorial, I hope someone get something out of this. It is a very handy skill to have.

Cheers.
Jason.

PS. Please note that I wrote all of this in notepad (including the code) so if there are any coding errors, don't flame me...just point them out, I'm bound to have made a syntax error somewhere.

Would take 10 second's if you know what you are doing.

Originally Posted by Lawllypawp

Would take 10 second's if you know what you are doing.

Um...your point?

thanks for writing this for me

Originally Posted by justiman

thanks for writing this for me

not for you So gtfu

It was for me /me /joking xD

Thanks Jason ^^ , But still using a webbrowser i think, The web scrap thing xD

Originally Posted by Lyoto Machida

not for you So gtfu

It was for me /me /joking xD

Thanks Jason ^^ , But still using a webbrowser i think, The web scrap thing xD

no it actually was for me jason posted it in my http wrapper thread

Originally Posted by Lyoto Machida

not for you So gtfu

It was for me /me /joking xD

Thanks Jason ^^ , But still using a webbrowser i think, The web scrap thing xD

It was for justiman by the way.

You should NEVER have to use a fagbrowser for simple scraping.

Good job Jason this could be usefull! I would rather go with fagbrowser though just becuase I'm a fag user.

^fagbrowser?
webbrowser.

Even though they pretty much fail completely at what you need to do in this instance they aren't bad for grabbing information and submitting forms SIMPLE-ly. (sic)

Great Tutorial. I don't believe the noobs still persisting to use webbrowsers.

Thanks for the comments guys.

WebBrowsers are a really inefficient and inaccurate way of doing any type of web-scraping. The only time they are a feasible alternative is when you can't simulate the login yourself (due to lack of knowledge about the various encryptions the site uses on its postString...or other factors such as general laziness)

Thanks, can come in useful

[Tutorial] Using HttpWebRequests to browse forums.

Tags for this Thread