Rectangular shapes coordinates detection

[Combozavr]

Greetings!

Im making a program for converting Diablo 3 items stats into text from image, i made things work on AH but now i want it to scan the game in real time mode and grab all the item windows that player sees (that requires 3-5 screenshots per second).

I have some ideas on how to make recognizing item windows easier for the machine, as so:
set brightness and contrast of each screenshot to max, this should remove most of gradient places (wich are bad for my goal)
turn screenshot into monochrome image, this should make things ALOT faster, since each pixel may have only 2 states - black or white (please correct me if im wrong)

here are steps that ive made (in paint.net so far but i think it wont be hard to make same things in ImageMagick):
1) Original image:

2) Setting input level to 1 (i dont understand what it does, but it does what i need lol):

3)monochrome:

4) I need coords of these red spots (or at least top left spots, since width is fixed and height is very easy to detect by going from top left corner down till first white pixel):


I think 1 of possibilities is to remove all noise by, lets say, hard blurring image at some point and then monochroming it, but im not sure. But if i manage to totally remove all the black shapes of small size, then i could just search for first black pixels with functions that work with binaries.

Any way if some1 can help me or give an advice or link - this will be much appreciated.

Edit: i dunno why it cutted all the images, here are links to them on image shack:
1) http://imageshack.us/photo/my-images/651/49038602a.jpg/
2) http://imageshack.us/photo/my-images/829/23798932.jpg/
3) http://imageshack.us/photo/my-images/842/14352362.jpg/
4) http://imageshack.us/photo/my-images/688/18924584.jpg/

[fmw42]

Setting input level to 1 (i dont understand what it does, but it does what i need lol):

We need more help to know what this is doing? Can you make a screen snap of your tool showing the controls for this command?

Is there no documentation about the command?

In IM, try

convert image.jpg -level 0x1 result.png

But that does not give exactly the same result. The image you provided seems to have a lot of junk colors around the text, that make it very hard to resolve without bringing out other areas.

So far my best effort is:

convert 49038602.jpg -colorspace gray -level 0x1% -threshold 20% 49038602_gray_l0x1pct_t20pct.png

or just

convert 49038602.jpg -colorspace gray -level 0x150 49038602_gray_l0x150.png


Or adding a little noise filtering at the beginning.

convert 49038602.jpg -enhance -colorspace gray -level 0x150 49038602_enh_gray_l0x150.png

convert 49038602.jpg -enhance -colorspace gray -level 0x1% -threshold 20% 49038602_enh_gray_l0x1pct_t20pct.png

[Combozavr]

Thanks for your reply! ill test all your suggestions whenever i will be able to.

On screenshot #2 you can see interface of Paint.net with that settings (thats why i gave souch a short comment on that step, cuz ive shown interface).

Still, thanks. Say, will any of those functions give me coords of smthn? If not - do you know how that can be done?

Also thanks in advance!

sorry if some of what i wrote wa sstupid cuz i wasnt even looking on functions, dont have much time right now.

P.S. Sp is that true that working with monochrome images is ALOT faster for engine? Or i need to use certain commands to make calcullations faster, or what?

[fmw42]

Sorry your images displayed above cut off the screen shot showing the command window. But I realize now I can open the image and see more of it. It looks like the standard interface to level as in Photoshop. In IM the command would be -level 0,1,1.02 of using Q8 IM (the third value from your screenshot is the gamma value) but my attempts with IM did not reproduce your result very closely.

I tried similar GUI commands in both Photoshop and GraphicConverter and they as well as IM always leave the red rectangles in the result.

Are you sure you did nothing before the first operation you show or something afterwards to change the image you posted?

So, since I could not reproduce the results from similar commands, I tried other things.

Note if you are using IM Q16 (16 bit compile) then the values for -level will be between 0 and 65535 for the first two values (even if your image is only 8-bits). You can also use percent values. Your screen shot shows values between 0 and 255 (8-bit). So 1/255 would be 0.4% and not 1 for -level.

Coordinates of what? In general IM has no particular smarts to find objects in a picture. You would have to find some way to crop those parts and then IM can give you the crop offsets and sizes.

IM -monochrome may work fast, but your image will be binary by the use of -threshold, also. So -monochrome is not needed. Furthermore it dithers the image.

[Combozavr]

omg lol im so sorry i didint mention that i modified 1st slide right be4 posting, to hide my ingame contacts DDD Making this prog might be violation of blizz rules...

Ok, ive edited 1st slide, evrything should be ok now^^ And btw ive been using all operations on BMP files but ive uploaded jpg so all results on modifying those might still be slightly different cuz of compression. ALSO ive drawn red dots manually on the final slide!

And regarding coordinates... that is the most important part. I know that IM can find subimage on an image and return coordinates of it on the original picture, so as return 2 resulting pictures that show probable positions of subimage on the original image. I guess coordinates that it should return are first coordinates of best matching position, but it doesnt work for me. here are some quotes:

If the reconstructed image is a subimage of the image, the compare program returns the best match offset. In addition, it returns a similarity image such that an exact match location is completely white and if none of the pixels match, black, otherwise some gray level in-between:

Code: Select all

$ compare -metric RMSE -subimage-search logo.png wizard.jpg similarity.gif
85.05 (0.00129778) @ 353,157

^ from here: http://www.imagemagick.org/script/compare.php

And ive found some post recently on another forum, will be too hard to find the link now since i closed it and have alot of data in my history since then, but the meaning was that it is possible to get coordinates from IM. The guy was looking for piramids on image that have several piramids, all of his images were monochrome. BUT HE DIDT SAY HOW TO GET THOSE COORDS!:((((

Code: Select all

353,157

i think that those numbers are coords that i need but i couldt make IM to return them in any form in windows 7 command line call or call from my program (IM only makes 2 files and doesnt return any sort of numbers)..

Thats why im using im for generating grayscale map first and then im looking for the first pure black pixel (thats weird cuz i thought that best match pixel is pure white, but only looking for pure black worked) on it with pixel compare loop in my prog, but it works rly slowly, i will never be able to check 5 slides per second, especially with high resolution. Thats why im looking for faster ways to analyze image. I guess ill try to play with monochrome bitmaps today and search more info on .bmp specification or other formats that will be analyzed faster.

Right now the speed of finding 3x3 full color subimage on my resolution is reeeeeeeeally slow, its like 10+ mins... i will only be able to load all those slides in some kind of buffer (temp folder) and then let prog analyze all then in background, but this will take x50+ more time than a game time and alot of HD space also, its like if player played for 1 hour and program took 1 screenshot evry second, it wil have 18000 slides after that, 10 mins for each slide...

i guess for my game schedule it will take a couple of centuries to analyze all:D

[fmw42]

Are you saying that

compare -metric RMSE -subimage-search logo.png wizard.jpg similarity.gif
85.05 (0.00129778) @ 353,157

does not work on your system. Note that -subimage-search was added at 6.6.3-6. You don't mention what version of IM you are using?

The match coordinate for the upper left corner of the wizard in the logo image is 353,157

[Combozavr]

product version of compare.exe is 6.7.7

-subimage-search works but it only returns 2 images, no coords info (the most important info that i need, to crop text region from screenshot and feed it to OCR engine)

APPEND:

im using
execute_program(convert.exe,*COMMAND LINE TEXT*,true)

3rd argument (true) means that the code have to wait prog to finish and return some results be4 it continues execution. Also this command (execute program) returns anything that the execute dprogram can give to it (not sure about this, but i believe it is called STDOUT). This command worked perfectly with alot of programs but with IM compare -subimage-search commands it returns absolutely nothing and thats one of my main problems. But the main problem is to find region that i need fast enough.

[fmw42]

are you saying that the terminal information is not showing, namely,

85.05 (0.00129778) @ 353,157

If not then it is your system setup regarding where standard error is being sent.

try
compare -metric RMSE -subimage-search logo.png wizard.jpg similarity.gif 2>&1

[Combozavr]

yeah ive tried 2>&1 and also adding in the end "null: 2>&1" and a buhnch of other operations to try getting STDERR stream but nothing worked.

Do you recommend to search more ways of getting STDERR in my system?

also this is only half of problem, the main problem is speed of -subimage-search and thats why im looking for alternatives, as souch - working with monochrome, looking for horisontal black line of set width (cuz this is the best unique attribute of a region that im looking for) on it, but im almost sure that simply making 395x1 subimage sample and looking for it on the monochrome original image wont make it faster, i gotta tell IM that the task is much much simplier than it thinks, then it will do stuff much faster. But how...

[fmw42]

yeah ive tried 2>&1 and also adding in the end "null: 2>&1" and a buhnch of other operations to try getting STDERR stream but nothing worked.

Do you recommend to search more ways of getting STDERR in my system?

also this is only half of problem, the main problem is speed of -subimage-search and thats why im looking for alternatives, as souch - working with monochrome, looking for horisontal black line of set width (cuz this is the best unique attribute of a region that im looking for) on it, but im almost sure that simply making 395x1 subimage sample and looking for it on the monochrome original image wont make it faster, i gotta tell IM that the task is much much simplier than it thinks, then it will do stuff much faster. But how...

I have no idea about your system? Have you tried upgrading your version of IM?

compare is going to be very slow. You might find that using my script, normcrosscor, in HDRI mode via FFT is faster. See the link below.

[Combozavr]

ive downloaded IM about a week ago from off site, so i dont think i need to upgrade it.

So your THAT guy? Ive looked all your scrypts be4 writing this post, but i guess i wasnt accurate enough. Ill check out what you said right now

[fmw42]

I just tested compare on Mac OSX Snow Leopard Mac mini under IM 6.7.7.10 Q16:


time compare -metric RMSE -subimage-search logo.png logo_wizard.png similarity.gif
0 (0) @ 327,130

real 1m52.387s
user 1m51.906s
sys 0m0.473s

[vijjay29]

Hey,
compare -metric rmse -subimage-search <original-Image> <SubImage> result-subimage.jpg

Above cli gives the starting point of subImage in original image,

How to get the ending point of the founded sub image ??

I need 4 coordinates of the sub-image, anyone to help ??

[snibgo]

To the found coordinates, add the width and height of the subimage.

[vijjay29]

@snibgo,
Thanks for your reply.
Is there any command to get the four coordinates like "compare -metric rmse -subimage-search <original-Image> <SubImage> result-subimage.jpg" ??

Because I've only resized subImages.