HelpHow to Read/Scan Screen?

Posts 1–10 of 10 · Page 1 of 1

How to Read/Scan Screen?

Hello,

I was recently thinking of some programs I would like to make, and I have stumbled across a problem. I have been searching for a way to read/scan the screen for information. Say if "Hello" displayed itself in a text box or something I could detect that it said "Hello" and obviously base some sort of action on it. I would also like to do this with colors, if something green was on the screen I would like be able to recognize that. Also if its possible I would like to be able to do this for the whole desktop, and for specific area's on the screen. One last thing I would like to be able to do this for the desktop(This is the most important for me), and maybe if it is possible I would like to look at a fullscreen application.

I don't even know if some of this is even possible or if anyone has even thought about it, or even done it here. I just was looking to see if I could get some input, maybe even some source code if someone has done this.

P.S. The closest thing I have seen to what I want to do is a 'Color Bot', but when I tried to look for some code so I could understand how it works, I came up with nothing. Anyway just a fun idea I want to mess around with. Thanks for reading and any responses in advance.

There are a number of different ways you could go about this. The easiest would be taking a screenshot-see here- and

Normally, to detect changes in an application you have to first hook the process. A lot of malware hook browser functions. For example, lets say a malware's goal is to steal your internet passwords. It might hook the CredEnumerate function which gets called when auto-completing forms. In your case you essentially need to write a hook that recognizes when the user inputs data into a form.
For a browser its easy-just check for a <form> element. You can also look into event hooking.
For desktop applications you would just have to hook CreateWindowEx and check the first parameter is WS_EX_CLIENTEDGE. If it is, then you can just check the data it holds with SendMessage.

One other option involves taking a screenshot of the desktop and using CImg or OpenCV to run image analysis. Take two screenshots, two seconds apart.
1. Compare for any differences.
2. If there is a difference, is it a text difference?
3. If it is a text difference, convert it to a data type
OpenCV has machine learning algorithms which may be handy.

Thanks for the reply. I started looking into software that took screenshots and changed them to bitmap images and such last night. This looks like a much more viable option for what im doing. So I think the desktop should be covered for me. But about the application side of things, If i don't want to write hooks could I just use the same method as the desktop if say I am looking for a specific word/color to show up. I play CSGO, and I went through the hacks that were posted on here, and I found things like 'External Color Trigger Bots'. How would they pull this sort of thing off without hooking into the application. and Thanks so much for the reply, you are saving me alot of time by pointing me in the right direction.

Those external color aimbots work by interfacing with esp/cham hacks, that re-color the players in the game. Then your aimbot looks for that color on the screen and moves the mouse to that spot. (player model recolored, your hack searches for a color on the screen, if it finds the color it moves the mouse to that spot, rinse/repeat)

You would be better off creating a hack based on memory reading, there's a few easy guide tuts out there for CS:GO (Try this). What you're wanting to do sounds a lot like what people usually do in Autoit/AHK, and you can do it in C++, but you would be doing yourself a favor by learning C++ (if you don't know it already) first, getting into some WinAPI, and then try making what you want.

It's hard, a lot more time consuming, but the final product is so much better, and the journey getting there is much better.

I am not trying to create a hack for a game. It is just what I have seen and It gave me an idea on how I could scan for different colors. im wondering if I can do that same thing outside an application. Like maybe find objects that are a certain color on my screen, then I can OCR the image or something to check for text inside the colored area.

I have done game hacking before for like assault cube and stuff for practice. I also have 3 years of c++ programming, 1 year of java, and I messed around with a few others. But I have never messed with WinAPI or even integrating my code with other programs. Its all been school work and a few fun projects on the side. Pretty much I want to create a general program/tool that has the capability to search for color/text(and maybe be able to compare images with the use of OpenCV) that I can use for other, more specified programs that I want to use to automate some of my daily work and what not.

So I guess after all that, it looks like I cant use the same code that those color bots do, but you think it would be possible to take a screen shot, then search for colors, and using those colors I could maybe create a 'hitbox' that I could then OCR or OpenCV to check for words or changes to the text inside. I could then run like a macro, or a series of commands after that to get it what I need to do. This all seems possible, just gonna need to do a lot of research I guess.

If anyone still has any input on this, or tools I could use I always welcome a little help.

Originally Posted by stfuandh0p

I am not trying to create a hack for a game. It is just what I have seen and It gave me an idea on how I could scan for different colors. im wondering if I can do that same thing outside an application. Like maybe find objects that are a certain color on my screen, then I can OCR the image or something to check for text inside the colored area.

I have done game hacking before for like assault cube and stuff for practice. I also have 3 years of c++ programming, 1 year of java, and I messed around with a few others. But I have never messed with WinAPI or even integrating my code with other programs. Its all been school work and a few fun projects on the side. Pretty much I want to create a general program/tool that has the capability to search for color/text(and maybe be able to compare images with the use of OpenCV) that I can use for other, more specified programs that I want to use to automate some of my daily work and what not.

So I guess after all that, it looks like I cant use the same code that those color bots do, but you think it would be possible to take a screen shot, then search for colors, and using those colors I could maybe create a 'hitbox' that I could then OCR or OpenCV to check for words or changes to the text inside. I could then run like a macro, or a series of commands after that to get it what I need to do. This all seems possible, just gonna need to do a lot of research I guess.

If anyone still has any input on this, or tools I could use I always welcome a little help.

I'm only basing this off of what I've done in specifically Java:

I know that you can search the screen for a specific color, but the problem is, millions of things on your screen could match that color (so this would require a limit of size that it could be in, i.e if resolution is 1920x1080, you may only want 800x600 to be searched and in a specific location)

I'm working on linux so it's a bit hard for me to test windows applications, but I'll see what I can do and I'll respond back to you

Originally Posted by liquidsystem

I'm only basing this off of what I've done in specifically Java:

I know that you can search the screen for a specific color, but the problem is, millions of things on your screen could match that color (so this would require a limit of size that it could be in, i.e if resolution is 1920x1080, you may only want 800x600 to be searched and in a specific location)

I'm working on linux so it's a bit hard for me to test windows applications, but I'll see what I can do and I'll respond back to you

Anyway i can see one of your projects, im still running into some problems, and I would like to see another persons project.

Originally Posted by stfuandh0p

Anyway i can see one of your projects, im still running into some problems, and I would like to see another persons project.

Is there any reason you couldn't use an address to find when it changes, or something of the sorts? If you're hoping for a just general, read ANYTHING on the screen, then I guess you couldn't do this, but do you have a specific example that YOU are using it for? (Or was that example in the OP your use for it?)

I think you're wanting to look for something like Image search. AHK and Autoit use it. They both use the imagesearch.dll file that's floating around on the internet (source code for it is somewhere around the internet too.

You wouldn't be looking for just one color on the screen but scanning the screen with an image you've already captured. Essentially, if the text is all uniform and the same, you can use it to read too (if you are trying to keep it general and not specific to one program)

You would have to already have all the images you're looking for (in some kind of supported format)

Hmm, I don't think the image search would work in my situation. This is just a general program, and I don't want to search for specific images. Also I don't want to search for addresses of specific programs because I think it would be to much work to do every time I want to add something to look for. My Idea is whenever i have the program running I want to create a list of things to look for. First I will scan for colors, then I want that region containing that color to be OCR to look for text. If it matches the text I will have it click it highlight it or whatever I want to do. I feel like if I can get this more general way of looking for changes to work, instead of scanning addresses i could adapt a lot more easy and it would not be much of a problem to add a new search function. Thanks for all your responses, and I have been working on this program slowly but surly. I am at the point where I have a color picker, screenshot function, and I can set a margin of error to look for. Now I am working on processing the screenshots and trying to create regions that I can OCR. If anyone has any input or sources that could help that would be great.

Posts 1–10 of 10 · Page 1 of 1

Tags for this Thread

None

HelpHow to Read/Scan Screen?

Post a Reply

Similar Threads

Tags for this Thread