Pixel searching in vb.net

[abuckau907]

Anyone interested in pixel bots? Never as good as memory reading / injecting bots, but can sometimes be safer.

(Ie. take a screenshot, then scan the screenshot for certain colors - red for health, etc etc. but slightly more complex than just red lol)

Could make up a tut. if there is enough interest?

(Saw a tut in the C++ section, thought vb.net might want one too?)

[Pingo]

Me and a few others have released a pixel bot, we had some interest in them.

It wouldn't hurt to have a tut. Im personally interested in the method you'l use.

When i first started my project, i tried the GetPixel method and the unsafe method.
GetPixel was too slow, unsafe was hard to read but fast.
I ended up making my own method from scratch, easy to read and fast.

[abuckau907]

Funny you mentioned that. Few years ago when I first tried pixel bots, I used GetPixel.
Now .Net has Graphics.CopyFromScreen()
I was going to do a speed test, and include both versions in the project. I assume m$ did good and .CopyFromScreen() is as fast as GetDesktopWindow(), GetDC(), GetPixel(),ReleaseDC(). Will test in the next few days.

with Graphics, it's way easier now, almost not enough to make a full tut. I was thinking more along the lines of...specific implementations / possible ideas. Nothing complex, but sometimes ppl have problems not with the code, but with the basic logic behind the code (ie. defining the problem well)

For example, you could have the user *set* 1 point for a health bar (button_click), then scan left and right to find the bar.width etc. Again, it's not hard, but a lot of ppl have trouble with the logic behind it.

Even just scanning for some small image inside the entire screenshot -- some ppl aren't even sure how to loop over the entire screenshot..let alone do the comparisons to a 2nd image.(how well did they learn the language?) If someone can't understand the logic behind something, I probably can't help them too much, but maybe seeing it step-by-step with pics. will help things click? Not sure, that's why I asked if there was interest.

[Pingo]

I never really had an idea what i was doing either when i started.
But instead of following a tut, i just worked things out for myself. Thats how i got it to click.

I ended up using Marshal.Copy

Marshal.Copy(BmpData.Scan0, Bytes, 0, Bytes.Length);

Then i could loop through the array to view each pixel

int Loc = (int)((double)y * BmpData.Stride) + (x * 3);
R = Bytes[Loc + 2], G = Bytes[Loc + 1], B = Bytes[Loc]

Using a reverse loop would allow me to scan just as fast as the unsafe method.
This is in C# though, easy enough to convert to vb.

[abuckau907]

same here. but for others, it could be helpful.

ya, I've thought about just returning an array of int32 instead of a bitmap -- would probably (definately?) be faster than returning bitmap, but since I don't really use it for anything..I don't know what the speed limits are / if it would actually matter -- working with the bitmaps is 'nicer', but if the speed was important, it could just pass around large arrays. I haven't given the class structure more thought (I'm sure this has been done many times before, probably even here at mpgh) to the actual structure/relationships --> healthbar struct, basicSquareImage, etc etc. Will have something to post in a few days. As much as I've been typing lately, not a whole lot of code is getting done.

[abuckau907]

(Started writing one anyway. If anyone is interested, please ask questions)

The Goal
Using pixel searching, be able to know the value of a progress bar (health,mana,ammo etc).

The drawbacks
As with any bot that monitors pixels, usually the user has to 'define' a few points on the screen, so the program knows WHERE to check for specific colors. It sucks, and I don't like it. There are ways to get around this, but it involves scanning the entire screen which is relatively very slow and should generally be avoided.

For example, most games (?) display the Local Player's health-bar at the top left of the screen (sometimes very bottom, centered usually). And most games use red. We could take a screenshot of the entire screen and scan it for red (or whatever color), but if your screen resolution is 1200x1000 ..that's a LOT of pixels! and checking each and every one will take a long time. Not actually a lot time, but in terms of milliseconds and fps, it's long.

Instead, it's better to have the user 'tell our program' where (about) the health bar is, so our scan area goes from 1200x1000 pixels to maybe 400x400 for example. The CPU will be happy it doesn't have to scan 1.2 million pixels every time your function executes But, forcing the user to tell where the progress bar is, is..not very cool. But it works. Something to consider when using pixel bots.

Anyway, on to the code.
.Net makes taking a screen shot (getting the pixel's color at a specific x,y location on the screen) very very easy. If you use a bitmap. I'm talking about Graphics.CopyFromScreen(). This function essentially takes a screenshot (you specify the location/size) and it writes the data into a bitmap.
Something along the lines of..

Dim _myBitmap as new Bitmap(PrimaryScreen.Width, primaryScreen.Height)
Dim _gr as Graphics = Graphics.FromImage(_myBitmap)
''now we can use _gr object to Draw() to the bitmap
_gr.CopyFromScreen(location/size) '<-not real parameters

That's it! As far as getting pixel colors from the screen, that's it. Now we have to decide WHICH areas to check.

some real code
(I made a class called ScreenShotManager, and this is a func. inside it)

    Public Function CopyFromScreen(ByVal tl As Point, ByVal width As Int32, ByVal height As Int32) As Bitmap
        Dim _rtnBmp As New Bitmap(width, height)
        Dim _gr As Graphics = Graphics.FromImage(_rtnBmp)
        _gr.CopyFromScreen(tl, New Point(0, 0), New Size(width, height), CopyPixelOperation.SourceCopy)
        _gr.Dispose()
        Return _rtnBmp
    End Function

which would be used like:
Dim _newSS As Bitmap = CopyFromScreen(new point(0,0),PrimaryScreenWidth,primaryscreenHeight)
to take a screenshot of the entire screen.

CopyFromScreen() is over-loaded a few times, the one I used is:

CopyFromScreen(upperLeftSource, upperLeftDestination, regionSize)

upperLeftSource: point on monitor where source will start
upperLeftDestination: where in our bitmap to copy to. 0,0 basically means start at top left (generally means use the whole bitmap)
regionSize: width and height -> ie. size of area to copy (1200x1000, 400x400 etc)

upperLeftSource:
(0,0) is the top-left of the screen.
(ScreenWidth,ScreenHeight) is the bottom-right

Why does this matter? Because we're not going to take a screenshot of the entire screen, but only of select areas.
Something that is still slow, but slightly better than taking a screenshot of the entire screen --> figure out where on the screen the Target Window is, and how big it is, and take a screenshot of just that area. I can't stress enough how slow this might be, but* it's better than capturing the entire screen. you might not actually use the code below. Just creating it for...learning's sake.

Copy only a specific window

 Public Function GetWindowScreen(ByVal windowTitle As String, Optional ByVal includeBorders As Boolean = True) As Bitmap
        Dim _targetWindowHandle As IntPtr = FindWindow(Nothing, windowTitle) 'try to find window based on title.
        If _targetWindowHandle = 0 Then
            ''window not found
            MessageBox.Show("ScreenShotManager::GetScreenOfApp windowTitle not found! --> '" & windowTitle & "'" & Environment.NewLine _
                            & "Are you sure that program is running?")
            Return New Bitmap(2, 2) ''why not (1,1). choice.
        Else
            ''window found
            Dim _targetWindowInfo As WINDOWINFO
            GetWindowInfo(_targetWindowHandle, _targetWindowInfo)
            If includeBorders Then
                Dim _tl As New Point(_targetWindowInfo.rcWindow.Left, _targetWindowInfo.rcWindow.Top)
                Return CopyFromScreen(_tl, _targetWindowInfo.rcWindow.Width, _targetWindowInfo.rcWindow.Height)
            Else
                Dim _tl As New Point(_targetWindowInfo.rcClient.Left, _targetWindowInfo.rcClient.Top)
                Return CopyFromScreen(_tl, _targetWindowInfo.rcClient.Width, _targetWindowInfo.rcClient.Height)
            End If
        End If
    End Function

*short explain:
Windows(R) likes to store things as 'handles', basically it's just a unique id number.
*
includeBorders --> each window has a 'working area', which doesn't include the borders. I added in option for both, really won't matter.
FindWindow() will return a window handle, if windowTitle is found.
GetWindowInfo() will return a structure (we define it in our code in a moment) which holds lots of useful information about a window (size, location, border style, few other things)

So I guess you need the API declarations..

    Private Declare Function FindWindow Lib "user32" Alias "FindWindowA" (ByVal lpClassName As String, ByVal lpWindowName As String) As IntPtr
        Private Declare Function GetWindowInfo Lib "user32" (ByVal hWnd As IntPtr, ByRef pWinInfo As WINDOWINFO) As Boolean

and the WINDOW_INFO structure..

    Private Structure RECT_win32 'not bit compatable with system.drawing.rectangle.
        Dim Left As Integer
        Dim Top As Integer
        Dim Right As Integer
        Dim Bottom As Integer
        Public Property Height() As Integer
            Get
                Return Bottom - Top
            End Get
            Set(ByVal value As Integer)
                Bottom = value - Top
            End Set
        End Property
        Public Property Width() As Integer
            Get
                Return Right - Left
            End Get
            Set(ByVal value As Integer)
                Right = value + Left
            End Set
        End Property
    End Structure
    ''' <summary>
    ''' Represents a Windows.Form 's properties like size and border_style
    ''' </summary>
    Private Structure WINDOWINFO
        Dim cbSize As Integer
        Dim rcWindow As RECT_win32
        Dim rcClient As RECT_win32
        Dim dwStyle As Integer
        Dim dwExStyle As Integer
        Dim dwWindowStatus As UInt32
        Dim cxWindowBorders As UInt32
        Dim cyWindowBorders As UInt32
        Dim atomWindowType As UInt16
        Dim wCreatorVersion As Short
    End Structure

Notice the other structure in there? RECT_win32.
To make a long story short, .NET already has a 'Rectangle' data type, but it's variables (left,top,right,bottom) aren't defined the same way, so they're not interchangable on the bit level.

So what can we do so far? Well, nothing to do with a health-bar yet, but we can copy the pixels from any area on the screen, and we have a special function to copy a specific window based on title. It's not a lot, but it's a start.

Because each game does it's health bars a little differently, I don't really want to write a specific guide for a specif game. Instead I'll just go over the general concept (at least how I do it).

So, first thing's first: we need the user to tell us WHERE the health bar is. Personally I add a new Button to my project --> The user must tab to the button, then move the mouse to the correct location, and press {Enter} on the keyboard (ie. pressing the button) and my button code saves the cursor.position. Again, it's crude..I call it "GUI configuration" and as far as bots go, it's def. not professional, but it works well enough for personal projects. Or if it doesn't, stop reading.

The basic idea is this:
If we know the left-most point on the health bar, and the right-most point, then we know how wide it is.
If red pixels don't go all the way to the right-most point..we know health is less than 100%!
We can actually figure out (a really close approximation) of the bar's value by measuring the width of red pixels and comparing it to the width the bar is supposed to be.

1. Starting at left-most point and scanning right, once you find a non-red pixel, you know 'health pixels width'.
2. Divide 'health pixels width' by 'bar width' and you have a ratio like 1/2, which is your health.

pretty simple concept, but if all you need is to know when your health is < 50 or whatever, it works.

I guess that means you have a choice --> to know how wide the 'health bar' is:
The way I do it, is when the user presses the 'set mouse location' button, it (assumes health is full) and scans as far left, and as far right as it can find red pixels. Then it knows width.
Another option would be to make the user set 2 points -- healthBar.FarLeft and healthBar.FarRight
Since we're making the user set points anyway, choice is up to you. For a personal bot, it shouldn't feel like too much work? Once I have my profiles saved, re-setting up usually takes a minute or two, and bot goes for a few hours, so it works for me personally.


Hope this helps someone*

---------- Post added at 01:36 PM ---------- Previous post was at 01:32 PM ----------

Somewhat un-related, but, it's possible to know when you have a target AND how much health the target has.

*most games*, when you have a target, will display a 'banner' with their name and health ...usually this is in the same spot on the screen, every time.

Knowing this: check the 'enemy banner location' to know if you have an enemy
check the 'enemy banner health loc' to know how much health they have.


not a good bot, but basically {spam} tab until you have a target, {spam} some keys until target is dead. repeat.

^^edit* alternatively, if you *know* the health bar will be at the top-left (or wherever)...you could just take a screen shot of the app, and 'try to figure out where the health bar is' by knowing it's *somewhere* in the top-left. This works. And doesn't require the user to set any points, but it's more cpu intensive to find it's location, and slightly more complex/error prone.

[abuckau907]

@Pingo

Just looking at some old code. the way I did screenshots, before Graphics.CopyFromScreen was....

Public Shared Function GetPixelColor(ByVal x As Int32, ByVal y As Int32) As Color
        Dim hWnd As IntPtr
        Dim hDC As IntPtr
        hWnd = GetDesktopWindow()
        hDC = GetWindowDC(hWnd)

        Dim lColor As Int32 = GetPixel(hDC, x, y)
        ReleaseDC(hWnd, hDC)

        Return ColorTranslator.FromWin32(lColor)
    End Function
    Public Shared Function GetScreenArea(ByVal tl As Point, ByVal br As Point) As Bitmap
        Dim hWnd As IntPtr
        Dim hDC As IntPtr
        hWnd = GetDesktopWindow
        hDC = GetWindowDC(hWnd)
        Dim tmpColor As Color
        '''''''''''''''''''''''''''''''''''''
        Dim areaWidth As Int32 = br.X - tl.X
        Dim areaHeight As Int32 = br.Y - tl.Y
        Dim rtnBtmp As New Bitmap(areaWidth, areaHeight)
        'scans top-to-bottom, left-to-right
        For xx As Int32 = 0 To areaWidth - 1
            For yy As Int32 = 0 To areaHeight - 1
                tmpColor = ColorTranslator.FromWin32(GetPixel(hDC, tl.X + xx, tl.Y + yy))
                rtnBtmp.SetPixel(xx, yy, tmpColor)
            Next
        Next
        ReleaseDC(hWnd, hDC)
        Return rtnBtmp
    End Function

^^again, because it's storing it into a Bitmap, it's not AS fast as it could be -- it would be slightly more efficient to just store the color data as an array of Int32 and leave bitmaps out of it completely.

[Pingo]

@abuckau907
I did it alittle different. Mine would either scan around the cursor or within a resizable rectangle.

        public Bitmap CaptureBmp()
        {
            if (Visible)
            {
                Point P = RetLocation();
                Rect = new Rectangle(Location.X + 8, Location.Y + 8, S.Width - 17, S.Height - 17);
            }
            else Rect = new Rectangle(Cursor.Position.X - (SizeW / 2), Cursor.Position.Y - (SizeH / 2), SizeW, SizeH);
            Bmp = new Bitmap(Rect.Width, Rect.Height, PixelFormat.Format32bppPArgb);
            G = Graphics.FromImage(Bmp);
            G.CopyFromScreen(Rect.X, Rect.Y, 0, 0, Rect.Size, CopyPixelOperation.SourceCopy);
            BmpData = Bmp.LockBits(new Rectangle(0, 0, !Visible ? SizeW : Rect.Width, !Visible ? SizeH : Rect.Height), ImageLockMode.ReadOnly, PixelFormat.Format24bppRgb);
            Bytes = new byte[BmpData.Stride * BmpData.Height];
            G.Dispose();
            return Bmp;
        }

Then used marshal to copy the data.

Marshal.Copy(BmpData.Scan0, Bytes, 0, Bytes.Length);

[IMG]https://i203.photobucke*****m/albums/aa29/Baxter_esa/PixiScreen-1.png[/IMG]

[abuckau907]

@Pingo lol..maybe it's cuz I just woke up but you don't need the marshal.copy

Bytes = new byte[BmpData.Stride * BmpData.Height]; //declared, but never used?

I believe ..

        public Bitmap CaptureBmp() // should use RetLocation as parameters so func. is more flexible
        {
            if (Visible)
            {
                Point P = RetLocation();
                Rect = new Rectangle(Location.X + 8, Location.Y + 8, S.Width - 17, S.Height - 17);
            }
            else Rect = new Rectangle(Cursor.Position.X - (SizeW / 2), Cursor.Position.Y - (SizeH / 2), SizeW, SizeH);
            Bmp = new Bitmap(Rect.Width, Rect.Height, PixelFormat.Format32bppPArgb);
            G = Graphics.FromImage(Bmp);
            G.CopyFromScreen(Rect.X, Rect.Y, 0, 0, Rect.Size, CopyPixelOperation.SourceCopy);
            G.Dispose();
            return Bmp;
        }

should work exactly the same, but slightly faster?

-I'm not sure why your Marshal.Copy wasn't actually in the sub? You can't access 'Bytes' if it's not. Maybe

[Pingo]

@abuckau907
Bytes = new byte[BmpData.Stride * BmpData.Height];
It is used, just not there. The size is dynamic so i needed to declare the size somewhere.

I think i did need to use marshal. It was awhile since i made this and i think i tried it without marshal, something wasn't right.
Can't remember exactly.

Dont expect it to be a perfect scanner, it was my first attempt and i coded it from scratch without help online.

I tested the speed against the unsafe method which was supposed to be the fastest in C#, both were near the same speed.

[abuckau907]

"It is used, just not there." -- heh, probably cuz u were first starting out..I'm sure u scope variables differently now Anyway...this post is dead. later.