[Help]Case in order?

**Jason** · 09-11-2010

Originally Posted by Web-Designer

fast facts

Alright I think i've managed to make it extract all the fact, much hardcoding and messy shit there but it works I'm pretty sure. It's hardly elegant but I never pretended to be good at string manipulation

First, make a new "Specialized.StringCollection" in your project settings. Remember what you called it. I want you to save this internally because it takes a few second to extract all the info every time you load which is gay. So this way it only has to do it once.

Okay here's all the horrible code to extract the facts (please no harshness, I'm sorry

)

[php]
Private Function GetPageSource(ByVal Website As String) As String

Dim webReq As Net.HttpWebRequest = Net.WebRequest.Create(Website)
Dim webResp As Net.HttpWebResponse = webReq.GetResponse

Using sRead As New IO.StreamReader(webResp.GetResponseStream)
GetPageSource = sRead.ReadToEnd
End Using

End Function

Private Function ExtractFacts(ByVal website As String) As String()

Dim strArray() As String
Dim tempFile As String = My.Computer.FileSystem.GetTempFileName & Rnd() * 9999
Dim totalString As String
Dim i As Integer = 0
totalString = GetPageSource(website)

Dim opener As String = "<div class=""entry"">"
Dim openIndex As Integer = (totalString.LastIndexOf(opener) + opener.Length)
Dim closeIndex As Integer = totalString.IndexOf("</div")

Try
totalString = totalString.Substring(openIndex - opener.Length)

totalString = totalString.Substring((totalString.LastIndexOf(ope ner) + opener.Length), closeIndex - (totalString.LastIndexOf(opener) + opener.Length))
Catch ex As Exception

End Try

My.Computer.FileSystem.WriteAllText(tempFile, totalString, False)

Using sRead As New IO.StreamReader(tempFile)
Dim curline As String

Do Until sRead.EndOfStream
curline = sRead.ReadLine
Try
If curline.Contains("") Or curline.Contains(opener) Then
If Not curline.Contains(" ") Then

If Not curline.Contains(opener) Then
curline = TrimIt(curline)
ReDim Preserve strArray(i)
strArray(i) = curline
i += 1
End If

Else
sRead.ReadLine()

End If

End If
Catch
End Try
Loop
End Using

Try
Return strArray
Catch
Return Nothing
End Try
Try
My.Computer.FileSystem.DeleteFile(tempFile)
Catch

End Try

End Function

Private Function TrimIt(ByVal str As String)
Try

Dim endStr As String = ""
Dim currentString As String = str
Dim fIndex As Integer = currentString.IndexOf("") + 3
Dim lIndex As Integer = currentString.LastIndexOf("")

If currentString <> "" Then
Try
endStr = currentString.Substring(fIndex, lIndex - fIndex)
Catch ex As Exception

End Try

If endStr.Contains("<a href=") Then
Dim section As String = endStr.Substring((endStr.IndexOf("<a href=")), (endStr.LastIndexOf("</a>") + 4) - (endStr.IndexOf("<a href=")))
Dim contents As String = section.Substring((section.IndexOf(">") + 1), (section.LastIndexOf("</a>") - (section.IndexOf(">") + 1)))
endStr = endStr.Replace(section, contents)
End If

If endStr.Contains("’") Then endStr = endStr.Replace("’", "'")

If endStr.Contains("“") Then endStr = endStr.Replace("“", "'")

If endStr.Contains("”") Then endStr = endStr.Replace("”", "'")

If endStr.Contains("–") Then endStr = endStr.Replace("–", "-")

If endStr.Contains("″") Then endStr = endStr.Replace("″", "'")

If endStr.Contains("…") Then endStr = endStr.Replace("…", "...")

If endStr.Contains("</a>") Then endStr = endStr.Replace("</a>", "")

Return endStr
Else
Return Nothing
End If

Catch ex As Exception
Return Nothing
End Try
End Function
[/php]

(Yes, I know it's messy as shit, I didn't have time to clean it up)

Alright presuming you have all that code, add this to your form_load event (IMPORTANT)

[php]
Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

If My.Settings.StrCollect Is Nothing Then

My.Settings.StrCollect = New System.Collections.Specialized.StringCollection

Dim thearray() As String = ExtractFacts("https://didyouknow.org/fastfacts/animals/")
For i As Integer = 0 To thearray.Length - 1

Try
My.Settings.StrCollect.Add(thearray(i))
Catch
End Try

Next
My.Settings.Save()
My.Settings.Reload()
Else

'if you've already saved the facts to your string collection...what do you want to do? Leave this blank if you don't need them til later.

End If
End Sub
[/php]

Do that for all the different sections of FastFacts (body..etc)

if you don't know what I mean by that I mean:

[php]
Dim thearray() As String = ExtractFacts("https://didyouknow.org/fastfacts/animals/")
For i As Integer = 0 To thearray.Length - 1

Try
My.Settings.StrCollect.Add(thearray(i))
Catch
End Try

Next
[/php]

NOTE: You'll have to choose a different array name each time (i.e TheArray2()) remember the () it's important.

Good luck

**Web-Designer** · 09-11-2010

thank you!

**Jason** · 09-11-2010

Originally Posted by Web-Designer

thank you!

No problem.

EDIT: Made it a lot cleaner adding each website change your form_load to something like this:

[php]
Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

If My.Settings.StrCollect Is Nothing Then

My.Settings.StrCollect = New System.Collections.Specialized.StringCollection
addArray("https://didyouknow.org/fastfacts/body/")
addArray("https://didyouknow.org/fastfacts/animals/")
addArray("https://didyouknow.org/fastfacts/movies/")
addArray("https://didyouknow.org/fastfacts/money/")
addArray("https://didyouknow.org/fastfacts/politics/")

End If

For Each s As String In My.Settings.StrCollect
If s <> "" Then
ListBox1.Items.Add(s)
End If
Next
End Sub
[/php]

And add this sub:

[php]
Private Sub addArray(ByVal website As String)

Dim yourArray() As String
yourArray = ExtractFacts(website)
For i As Integer = 0 To yourarray.Length - 1

Try
If yourarray(i) <> "" Then
My.Settings.StrCollect.Add(yourarray(i))
End If
Catch ex As Exception

End Try
Next

My.Settings.Save()
My.Settings.Reload()
End Sub
[/php]

Now, if you want to add another website all you do is go:

[php]
addArray("http.........whatever")
[/php]

and that's all, nothing more. Just add it below this existing "addArray" calls.

**Web-Designer** · 09-12-2010

Okay but now do you think you can explain how you made this? lol would help me later on, I'm sure

**Jason** · 09-12-2010

Originally Posted by Web-Designer

Okay but now do you think you can explain how you made this? lol would help me later on, I'm sure

Sure thing, just give me around 4 hours because I have an exam in 20 mins so I don't have time to start writing yet.

**Jason** · 09-13-2010

Okay sorry for the double post but I promised to explain it to him

Okay basically what I did was look at the page source of the "didyouknow.org/fastfacts/animals" and then see how all the facts were laid out. To look at the page source just right click the page and choose "view source"

[php]
<div class="entry">
Mammals are the only animals with flaps around the ears.
African elephants only have four teeth to chew their food with.
There are about one billion <a href="https://didyouknow.org/animals/cows/">cattle</a> in the world of which 200 million are in India.
A house fly lives only 14 days.
A dog was the first in space and a sheep, a duck and a rooster the first to fly in a hot air balloon.
The <a href="https://didyouknow.org/animals/bigfive/">Big Five</a> is a group of animals of Africa: cape buffalo, elephant, leopard, lion and rhino.

The oldest breed of dog is the Saluki.
The bee hummingbird of Cuba is the smallest bird in the world.
An ostrich can run up to 43mph (70 km/h).
An annoyed camel will spit at a person.
The world’s smallest dog is the Chihuahua, which means “tiny dog in the sky.”
Pea crabs (the size of a pea) are the smallest crabs in the world.
75% of wild birds die before they are 6 months old.
The pig is rated the fourth most intelligent animal but are mentioned only twice in the Bible

Sheep are mentioned 45 times and goats 88 times in the Bible. Dogs are mentioned 14 times and lions 89 times, but 
domestic cats are not mentioned.
Pork is the world’s most widely-eaten meat.
In Denmark there are twice as many <a href="https://didyouknow.org/animals/pigs/">pigs</a> as people.
<a href="https://didyouknow.org/dinosaurs/">Dinosaurs</a> did not eat grass: there weren’t any at that time.

The coyote is a member of the dog family and its scientific name, “canis latrans” means barking dog.
A giraffe can clean its ears with its 50cm (20 in) tongue.
A group of geese on the ground is a gaggle – a group of geese in the air is a skein. <a href="https://didyouknow.org/animals/animalnames/">More animal collective nouns</a>
The South American giant anteater eats more than 30,000 ants a day.
It is impossible to out-swim a shark – sharks reach speeds of 44 *** (70 km/h). Humans can run about 21 *** (35 km/h).

The sailfish is the fastest swimmer, reaching 68 *** (109 km/h), although a <a href="https://www.youtube.com/watch?gl=US&v=mD7t057XIi8">black marlin</a> has been clocked at 80 *** (128 km/h).
The slowest fish is the Sea Horse, which moves along at about 0.01 *** (0.016 km/h).
Dolphins can reach 37 *** (60 km/h).
Of the 650 types of leeches, only the Hirudo medicinalis is used for medical treatments.
The heart of a blue whale is the size of a small car.
The tongue of a blue whale is as long as an elephant.
A blue whale weighs as much as 40 rhinos.

The scales of a crocodile are made of ceratin, the same substance that hooves and fingernails are made of.
A crocodile’s tongue is attached to the roof of its mouth and cannot move it.
A snail has two pairs of tentacles on its head. One pair is longer than the other and houses the eyes. The shorter pair is used for smelling and feeling its way around. (Some snail species have only one pair of tentacles, thus they have just one eye.)
The heaviest crustacean ever found was a lobster weighing 42 lb (19 kg), caught in 1934.
The largest jellyfish ever caught measured 7’6″ (2,3 m) across the bell with a tentacle of 120 ft (36 m) long.
The largest giant squid ever recorded was captured in the North Atlantic in 1878. It weighed 4 tons. Its tentacles measured 10 m (35 ft) long.
The giant squid has the biggest eyes of any animal: its eyes measure 16 inches (40 cm) in diameter.

Domestic <a href="https://didyouknow.org/animals/cats/">cats</a> purr at about 26 cycles per second, the same frequency as an idling diesel engine.
Sharks are immune to all known diseases.
Sharks and rays also share the same kind of skin: instead of scales, they have small tooth-like spikes called denticles. The spikes are so sharp that shark skin has long been used as sandpaper.
Animals also are either right-handed or left-handed. Polar bears are left-handed – and so is Kermit the Frog.
There are 701 types of pure breed dogs. There are about 54 million dogs in the US, and Paris is said to have more dogs than people.
Some <a href="https://didyouknow.org/birds.htm">bird species</a>, usually flightless birds, have only a lower eyelid, whereas pigeons use upper and lower lids to blink.

Fish and insects do not have eyelids – their eyes are protected by a hardened lens.
Flatfish (halibut, flounder, turbot, and sole) hatch like any other “normal” fish. As they grow, they turn sideways and one eye moves around so they have two eyes on the side that faces up.
Measured in straight flight, the spine-tailed swift is the fastest bird. It flies 106 *** (170 km/h). Second fastest is the Frigate, which reaches 94 *** (150 km/h).
Millions of trees are accidentally planted by squirrels who bury nuts and then forget where they hid them.
There are more than 150 million sheep in Australia, a nation of 21 million people.
New Zealand is home to 4 million people and 70 million sheep.

Also see <a href="https://didyouknow.org/animalsabc.htm"> A-Z of animals</a>
</div>
[/php]

That's how the source is laid out.

But how do we get to this source with your program? This is where the GetPage function comes in:

[php]
Private Function GetPageSource(ByVal Website As String) As String

Dim webReq As Net.HttpWebRequest = Net.WebRequest.Create(Website)
Dim webResp As Net.HttpWebResponse = webReq.GetResponse

Using sRead As New IO.StreamReader(webResp.GetResponseStream)
GetPageSource = sRead.ReadToEnd
End Using

End Function
[/php]

We make a web request to the specified website and then get its response and from that we get the responsestream. This is the page source.

Okay, now we have the pagesource, how do we get only the info we need? This bit is where you need to think.

Firstly, we look for a nice header to the block of facts, something unique only to that specific section. In this case I noticed that

Code:

<div class="entry">

Was the only instance of "<div class="entry">" and it preceded the facts (obviously it means it's an entry to the webpage). Now we know where to start looking, how do we know where to end? Well knowing a litttttllleee bit about the basics of HTML or w/e language pagesource is written in, I knew that all headers must be closed at some point, scrolling down I found this immediately after the block of facts:

Code:

</div>

That's essentially what this part of the extract function is doing:

[php]
Private Function ExtractFacts(ByVal website As String) As String()

Dim strArray() As String
Dim tempFile As String = My.Computer.FileSystem.GetTempFileName & Rnd() * 9999
Dim totalString As String
Dim i As Integer = 0
totalString = GetPageSource(website)

Dim opener As String = "<div class=""entry"">"
Dim openIndex As Integer = (totalString.LastIndexOf(opener) + opener.Length)
Dim closeIndex As Integer = totalString.IndexOf("</div")

Try
totalString = totalString.Substring(openIndex - opener.Length)

totalString = totalString.Substring((totalString.LastIndexOf(ope ner) + opener.Length), closeIndex - (totalString.LastIndexOf(opener) + opener.Length))
Catch ex As Exception

End Try
[/php]

I just used a simple substring to cut out just that big block of facts from the mass of page source. We want to be able to go through it line by line so I wrote it to a temporary file so we can use StreamReader to go through it.

Now we begin the lengthy process of extracting each line one by one.

First we look at the similarities between each line that contains a fact. What similarities do you notice? Well for starters every line begins with

Code:

<p>

and ends with

Code:

</p>

Okay so we know that the contents are contained between those two points, excellent. Now we have the basis of our extraction, we get the text between those two points...however this is where it gets annoying. Some lines in the source are written into two lines with the extension to identify them. Seeing as I couldn't be bothered with the few lines it would take to build the two lines together, I just skipped it entirely if it contained . This brings us to this section of the code.

[php]
Using sRead As New IO.StreamReader(tempFile)
Dim curline As String

Do Until sRead.EndOfStream
curline = sRead.ReadLine
Try
If curline.Contains("") Or curline.Contains(opener) Then
If Not curline.Contains(" ") Then

If Not curline.Contains(opener) Then
curline = TrimIt(curline)
ReDim Preserve strArray(i)
strArray(i) = curline
i += 1
End If

Else
sRead.ReadLine()

End If

End If
Catch
End Try
Loop
End Using
[/php]

You don't actually need the string array so you can just ignore that if you don't understand it and I'll post the modified code at the end.

Basically what this is doing is reading one line at a time. If it contains "" (remember that's our starting identifier) then check if the line contains "" if not, continue reading the line and then trim it down to what we can use. If it contains "< /br" we need to skip the next line by doing sRead.ReadLine() to set the marker down another line.

Okay this is the Trimit function. It returns your fact from the raw crap that you input

[php]
Private Function TrimIt(ByVal str As String)
Try

Dim endStr As String = ""
Dim currentString As String = str
Dim fIndex As Integer = currentString.IndexOf("") + 3
Dim lIndex As Integer = currentString.LastIndexOf("")

If currentString <> "" Then
Try
endStr = currentString.Substring(fIndex, lIndex - fIndex)
Catch ex As Exception

End Try

If endStr.Contains("<a href=") Then
Dim section As String = endStr.Substring((endStr.IndexOf("<a href=")), (endStr.LastIndexOf("</a>") + 4) - (endStr.IndexOf("<a href=")))
Dim contents As String = section.Substring((section.IndexOf(">") + 1), (section.LastIndexOf("</a>") - (section.IndexOf(">") + 1)))
endStr = endStr.Replace(section, contents)
End If
If endStr.Contains("&#821(7);") Then endStr = endStr.Replace("&#821(7);", "'")

If endStr.Contains("&#822(0);") Then endStr = endStr.Replace("&#822(0);", "'")

If endStr.Contains("&#822(1);") Then endStr = endStr.Replace("&#822(1);", "'")

If endStr.Contains("&#821(1);") Then endStr = endStr.Replace("&#821(1);", "-")

If endStr.Contains("&#824(3);") Then endStr = endStr.Replace("&#824(3);", "'")

If endStr.Contains("&#823(0);") Then endStr = endStr.Replace("&#823(0);", "...")

If endStr.Contains("</a>") Then endStr = endStr.Replace("</a>", "")

Return endStr
Else
Return Nothing
End If

Catch ex As Exception
Return Nothing
End Try
End Function [/php]

What you see here is just a load of hardcoding and extracting. For some reason the page source didn't cater for symbols like "-" etc so they have funny little things like "…" to represent symbols so I had to manually enter each one and tell the program what it SHOULD be. Also, some lines contain internal links to some of the words so I had to extract all the linking code and just leave the name.

That's basically all there is. It takes a little but of practice to write your own but all you really need to do is what I said in the first few steps:

1. Find what you want to extract in the page source
2. Figure out how all these are laid out (what's common? what can seperate it from the rest of the page? make sure it's an established pattern)
3. Figure out how you're going to extract each line (again, what's common?)

Main thing is you don't want to be hardcoding every single line.

Okay here's the modified full code:

[php]
Private Function GetPageSource(ByVal Website As String) As String

Dim webReq As Net.HttpWebRequest = Net.WebRequest.Create(Website)
Dim webResp As Net.HttpWebResponse = webReq.GetResponse

Using sRead As New IO.StreamReader(webResp.GetResponseStream)
GetPageSource = sRead.ReadToEnd
End Using

End Function

Private Sub ExtractFacts(ByVal website As String)

Dim tempFile As String = My.Computer.FileSystem.GetTempFileName & Rnd() * 9999
Dim totalString As String
Dim i As Integer = 0
totalString = GetPageSource(website)

Dim opener As String = "<div class=""entry"">"
Dim openIndex As Integer = (totalString.LastIndexOf(opener) + opener.Length)
Dim closeIndex As Integer = totalString.IndexOf("</div")

Try
totalString = totalString.Substring(openIndex - opener.Length)

totalString = totalString.Substring((totalString.LastIndexOf(ope ner) + opener.Length), closeIndex - (totalString.LastIndexOf(opener) + opener.Length))
Catch ex As Exception

End Try

My.Computer.FileSystem.WriteAllText(tempFile, totalString, False)

Using sRead As New IO.StreamReader(tempFile)
Dim curline As String

Do Until sRead.EndOfStream
curline = sRead.ReadLine
Try
If curline.Contains("") Or curline.Contains(opener) Then
If Not curline.Contains(" ") Then

If Not curline.Contains(opener) Then
My.Settings.StrCollect.Add(TrimIt(curline))
End If

Else
sRead.ReadLine()

End If

End If
Catch
End Try
Loop
End Using

Try
My.Computer.FileSystem.DeleteFile(tempFile)
Catch

End Try
My.Settings.Save()
My.Settings.Reload()
End Sub

Private Function TrimIt(ByVal str As String)
Try

Dim endStr As String = ""
Dim currentString As String = str
Dim fIndex As Integer = currentString.IndexOf("") + 3
Dim lIndex As Integer = currentString.LastIndexOf("")

If currentString <> "" Then
Try
endStr = currentString.Substring(fIndex, lIndex - fIndex)
Catch ex As Exception

End Try

If endStr.Contains("<a href=") Then
Dim section As String = endStr.Substring((endStr.IndexOf("<a href=")), (endStr.LastIndexOf("</a>") + 4) - (endStr.IndexOf("<a href=")))
Dim contents As String = section.Substring((section.IndexOf(">") + 1), (section.LastIndexOf("</a>") - (section.IndexOf(">") + 1)))
endStr = endStr.Replace(section, contents)
End If

If endStr.Contains("&#821(7);") Then endStr = endStr.Replace("&#821(7);", "'")

If endStr.Contains("&#822(0);") Then endStr = endStr.Replace("&#822(0);", "'")

If endStr.Contains("&#822(1);") Then endStr = endStr.Replace("&#822(1);", "'")

If endStr.Contains("&#821(1);") Then endStr = endStr.Replace("&#821(1);", "-")

If endStr.Contains("&#824(3);") Then endStr = endStr.Replace("&#824(3);", "'")

If endStr.Contains("&#823(0);") Then endStr = endStr.Replace("&#823(0);", "...")

If endStr.Contains("</a>") Then endStr = endStr.Replace("</a>", "")

Return endStr
Else
Return Nothing
End If

Catch ex As Exception
Return Nothing
End Try
End Function

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

If My.Settings.StrCollect Is Nothing Then

My.Settings.StrCollect = New System.Collections.Specialized.StringCollection
ExtractFacts("https://didyouknow.org/fastfacts/animals/")
ExtractFacts("https://didyouknow.org/fastfacts/body/")
'//continue like this for each page.
End If

For Each s As String In My.Settings.StrCollect
If s <> "" Then
ListBox1.Items.Add(s)
End If
Next
End Sub
[/php]
I removed the string arrays because I realized I could just save it to the string collection straight away. Silly me.

Hope this helps.

EDIT: Wow, coding wrappers are just screwing up my replacements. They're recognizing them just as html does and converting them to symbols as you see them when really they should be different. I had to modify those lines a little but so you could actually see them

.

Where you see these lines:

[php]
If endStr.Contains("&#821(7);") Then endStr = endStr.Replace("&#821(7);", "'")

If endStr.Contains("&#822(0);") Then endStr = endStr.Replace("&#822(0);", "'")

If endStr.Contains("&#822(1);") Then endStr = endStr.Replace("&#822(1);", "'")

If endStr.Contains("&#821(1);") Then endStr = endStr.Replace("&#821(1);", "-")

If endStr.Contains("&#824(3);") Then endStr = endStr.Replace("&#824(3);", "'")

If endStr.Contains("&#823(0);") Then endStr = endStr.Replace("&#823(0);", "...")

If endStr.Contains("</a>") Then endStr = endStr.Replace("</a>", "")
[/php]
Remove the brackets around the last number. Sorry that's the best I could do.

**Web-Designer** · 09-14-2010

Thanks. . .I know what I'm gonna be doing the the next few hours. . .Lol

Did you get banned just for me? Lol

**Imported** · 09-14-2010

Originally Posted by Web-Designer

Thanks. . .I know what I'm gonna be doing the the next few hours. . .Lol

Did you get banned just for me? Lol

I have no idea why I got banned lol. All it said was "Flame" in the message....dunno where from. Perhaps from when I told Bayley60 to grow a fucking brain. I'm guessing it was KING who banhammered me.

Ahh well.

Thread: [Help]Case in order?

Thread Tools

Display

The Following User Says Thank You to Jason For This Useful Post:

The Following 2 Users Say Thank You to Jason For This Useful Post:

The Following User Says Thank You to Jason For This Useful Post:

The Following User Says Thank You to Jason For This Useful Post:

Similar Threads

BLACK HISEC CASE HELP NEEDED

[Help Request] Help my!

Help with which ones cases are better?

Need help with ca---- Help me and i will gift u a gp case!

[help request]Need some info in order to make my own aimbot