Help with making program

[RedByte1337]

Can anyone make a program that saves all the usernames to a .txt file from this website: (Can't post links :/ google this:"theminecraftserverlist players")

[MarkHC]

Well, i don't play Minecraft but I assume it has some kind of web API to retrieve the player list on a server, and the server list so you should probably use those.

In any case, if you really want to, you can use some HTML parser for C# (like HtmlAgilityPack) to do some scraping on that website (do a request, get the result, parse it) and voilà.

[RedByte1337]

sorry I placed this on the wrong section :/ I tought I was in the other one.
I'm not so good in programing but isn't there a way that it goes from page to page and copies the row from the usernames:
link/pagenum=2 (on page2)
link/pagenum=3 (on page3)
link/pagenum=4 (on page4)
...
105741 pages
This is what I learned on school:

for(i=1;i<=105741;i++){
link=link+"/pagenum="+i}


As you can see it's not a lot so can anyone help me?
and I think you can't do this with java so...

[RedByte1337]

Sorry I placed this on the wrong section :/ I tought I was in the other one.
I'm not so good in programing but isn't there a way that it goes from page to page and copies the row from the usernames:
link/pagenum=2 (on page2)
link/pagenum=3 (on page3)
link/pagenum=4 (on page4)
...
105741 pages
This is what I have learned on school:

for(i=1;i<=105741;i++){
link=link+"/pagenum="+i}


As you can see it's not a lot so can anyone help me?
and I think you can't do this with java so...

[MarkHC]

You can do what you want using HtmlAgilityPack. But if you don't know how it works, then it gets hard.

public const string RootPage = "https://blablabl*****m/player";
public int MaxPages = 10;
for(int index = 0; index < MaxPages; index++){
     string page = RootPage + "?pagenum=" + index.ToString();
     
     //Download the new page, parse the values you want from the table and store them on a list or something
}

https://stackoverflow.com/questions/13005098/parsing-html-table-in-c-sharp

[abuckau907]

 

Public Class Form1
    Private _baseUrl As String = ""
    Private _outputFilename As String = ""
    Private _webRequest As System.Net.WebRequest
    Private _webResponse As System.Net.WebResponse
    Private _startTime As DateTime
    Private _forceStop As Boolean = False


  

    Private Sub cmdStart_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles cmdStart.Click
        Dim startPage As Int32 = 0
        Dim endPage As Int32 = 0
        Dim totalUsers As Int64 = 0 '' sum total of all usersnames found

        If Not Int32.TryParse(txtStartPage.Text, startPage) Then
            MessageBox.Show("Invalid start page number. Must be 0 < x > 1000000")
            txtStartPage.Focus()
            txtStartPage.SelectAll()
            Exit Sub
        End If
        If Not Int32.TryParse(txtStopPage.Text, endPage) Then
            MessageBox.Show("Invalid end page number. Must be 0 < x > 1000000")
            txtStopPage.Focus()
            txtStopPage.SelectAll()
            Exit Sub
        End If
        If endPage <= startPage Then
            MessageBox.Show("Invalid end page..must be greater than start page.")
            txtStopPage.Focus()
            txtStopPage.SelectAll()
            Exit Sub
        End If
        If txtOutputFilename.TextLength = 0 Then
            MessageBox.Show("Invalid output file name.")
            txtOutputFilename.Focus()
            Exit Sub
        End If
        ''User start/stop pages were valid
        _baseUrl = txtBaseUrl.Text
        _outputFilename = txtOutputFilename.Text
        Dim _pageUsers() As String = Nothing
        Dim _tmpSpan As TimeSpan
        _startTime = Date.Now()

        For xx As Int32 = startPage To endPage
            If _forceStop Then
                DoOutput("Force stop. Shutting down.")
                DoOutput("Last page successfully read #:" & (xx - 1).ToString())
                Application.DoEvents()
                _forceStop = False
                Exit Sub
            End If
            lblCurrentPage.Text = "Current Page: " + FormatNumber(xx, 0)
            Try
                _pageUsers = GetUsernamesByPage(xx) '' this function parses the raw page data and pulls out the usernames
            Catch ex As Exception
                '' server was busy? Page unreachable.
                DoOutput("Server busy/unreachable! page # " & xx.ToString() & " (wait 10sec.)")
                Application.DoEvents()
                xx -= 1
                Threading.Thread.Sleep(10000)
                Continue For
            End Try

            System****.File.AppendAllLines(_outputFilename & ".txt", _pageUsers)
            totalUsers += _pageUsers.Length()
            lblTotalUsers.Text = "Total Users: " + FormatNumber(totalUsers, 0)
            lblCurrentUser.Text = "Current User: " & _pageUsers(_pageUsers.Length - 1)
            ProgressBar1.Value = CInt((xx / endPage) * 100)
            _tmpSpan = Date.Now.Subtract(_startTime)
            lblTotalTime.Text = "Total Time: " & Math.Floor(_tmpSpan.TotalMinutes).ToString & " m, " & CInt(_tmpSpan.TotalSeconds Mod 60).ToString() & " s"
            Application.DoEvents() '' hmm
        Next
        _tmpSpan = Date.Now.Subtract(_startTime)
        DoOutput("Done. Time taken: " & Math.Floor(_tmpSpan.TotalMinutes).ToString() & " minutes, " & CInt(_tmpSpan.TotalSeconds Mod 60).ToString() & " seconds.")
        MsgBox("DONE. Time taken: " & Math.Floor(_tmpSpan.TotalMinutes).ToString() & " minutes, " & CInt(_tmpSpan.TotalSeconds Mod 60).ToString() & " seconds.")
    End Sub
    Private Sub cmdStop_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles cmdStop.Click
        _forceStop = True
    End Sub
    Private Function GetUsernamesByPage(ByVal pageNumber As Int32) As String()
        Dim _userNames As New List(Of String)

        _webRequest = System.Net.HttpWebRequest.Create(_baseUrl + pageNumber.ToString())
        _webRequest.Timeout = 10000 '' timeout = 10 seconds
        _webResponse = _webRequest.GetResponse()
        Dim _entirePageText As String = New System****.StreamReader(_webResponse.GetResponseStream, System.Text.Encoding.ASCII).ReadToEnd
        Dim _tableText As String = ""

        '' Find the table that contains all users
        Dim tableStartIndex As Int32 = 0
        Dim tableEndIndex As Int32 = 0
        tableStartIndex = _entirePageText.IndexOf("<tbody>")
        If tableStartIndex = -1 Then
            MessageBox.Show("Unable to find user table inside webpage. Format is out of date! This program needs to be updated!")
            _userNames.Add("ERROR Page #: " & pageNumber.ToString())
            Return _userNames.ToArray()
        End If
        tableEndIndex = _entirePageText.IndexOf("</table>", tableStartIndex)
        _tableText = _entirePageText.Substring(tableStartIndex, tableEndIndex)
        ''Find each row
        Dim _rowStartIndex As Int32 = _tableText.IndexOf("</th><th><a href=")
        Dim _rowEndIndex As Int32 = 0
        Do While _rowStartIndex > -1
            _rowEndIndex = _tableText.IndexOf(" />", _rowStartIndex) + 3
            _userNames.Add(_tableText.Substring(_rowEndIndex, _tableText.IndexOf("</a>", _rowEndIndex) - _rowEndIndex)) '' lol 
            _rowStartIndex = _tableText.IndexOf("</th><th><a href=", _rowEndIndex)
        Loop
        Return _userNames.ToArray()
    End Function

    Private Sub DoOutput(ByVal msg As String)
        txtOutput.AppendText("[" & Date.Now.ToLongTimeString & "] " & msg & Environment.NewLine)
    End Sub
End Class

link to project files msvs2010 (no executable): http : / / www . file dropper . c o m /redbyteprojectvb

link to executable: http : / / www . file dropper . c o m /redbyteproject
(requires .net framework 4.0)

 

jotti malware scan: https://virusscan.jotti.org/en/scanre...553f6f69111a05
MD5: af789dd2d3a32016715610c642e64226
SHA1: dce54b0b17fbde1bf72f3396129e022893a63fde

Virus-Total scan: https://www.virustotal.com/en/file/4...is/1402300927/
SHA256: 4d7fc501fc30f7a25de4f6020961cb86eda195714dac47a409 a50f077d84c47c

this is loading every page manually...with a ton of junk data...if they do have some api to give you that information in a smaller size, it'd be much faster/more efficient than this. Depending on sever load I was getting anywhere from 1 to 10 pages per second: 105,000+ pages will take a long time (and a lot of data? ~31kb per page?).

[RedByte1337]

TY so much but one more litlle thing?
how to open the things?

EDIT: nvm I think I can open it with Visual Studio.net 2008

[abuckau907]

just create a new project, copy the user-interface, and copy code over as needed. 2 Buttons and 2 functions are used. Hopefully you can manage to copy-paste 4 times : p


version2 would check the page and figure out what the actual max page # is..but u don't pay well enough for that : )
also reminds me...the user-interface hangs...o well.

I'm curious to know how long it takes to get all 105k pages, and how large the username file is. : p Gl.

edit: you'll need a special text editor once the output file is over a few MB. Notepad will crash.
and if the format of the webpage changes, the program probably won't work.

[RedByte1337]

hahaha ok,
will notepad++ work?

wtf windows smartscreen prevents me to open the file -_-
nvm I opened it as admin and everything works now

[abuckau907]

Not sure..google "open very large .txt file"

Idk man..you're on windows 8? It's targeting .net framework 4.0, you have that?

It's my bed time. Gl. Will check back in in the morning.

If all else fails, re-create the project yourself..that's why I posted a text copy of the source. Only 4 functions.

[RedByte1337]

Yes I have windows 8 and the program works fine now it takes 2~5 sec per page

[abuckau907]

1 second per page * 105k pages = 105k seconds

105,000 seconds / 60 = 1750 minutes

1750 minutes / 60 = 29 hours

at times I would see speed increases..maybe as much as 5-10 pages / second. Hopefully you get some of that too. 29 hours lol.. : (

-tested with another server (google) and transferred 1000 pages in 2m 27s
1000 / 147s = 6.8 pages per second

[Drokechas]

You use with webb strings with file on vbulletin or other web html..
Also is only text to text easy for EMAIL or other social..

Can anyone make a program that saves all the usernames to a .txt file from this website: (Can't post links :/ google this:"theminecraftserverlist players")

[abuckau907]

You use with webb strings with file on vbulletin or other web html..
Also is only text to text easy for EMAIL or other social..

Do you see that as being helpful...?