A more efficient proxy...

Well I've been playing around with Ikulip's java source code that I've updated to the current build of rotmg. As a first project, I decided to use the source to make a multibox character sync'er. That is, to get all the multiboxer characters to sync up and follow each other. To do this, I ended up using a client modified specifically for the proxy to handle the actual movement of the characters (added new packets to it and routines for dealing with them including the follow routine) while having the proxy reroute the location of my leader to all my followers. I wanted to get it to sync up as close as possible which means I needed to transfer that location information quickly to the follower clients. Well I quickly found out that Ikulip's proxy wasn't going to be as quick as I wanted even with removal of all the unneeded code.

The proxy does what a proxy should do, read in a packet, interpret it, make decisions/changes based on that packet (which includes sending other packets if needed), and send it out to its destination modified or not (if it was determined not to block the packet). The problem with this, as it applies to my project, packets are sent in at irregular time intervals, sometimes far longer than I wanted to wait to request a new position update from my leader client. I couldn't use this procedural loop to request the position updates. My solution was to make a separate thread that sent out a new position update requests to the leader client at regular intervals which eventually worked after dealing with threading related issues with the use of variables (this is really my first experience with threading).

The other problem I ran into is that I needed to send out packets quickly. The queuing system setup in the proxy is far too slow to get updates out to the followers (relies on the irregular input of packets). Ikulip mostly used the socket output stream directly, bypassing the queuing system, to send out packets. This works as long as one didn't try to cross threads to send out information. By cross threads I mean if one received a packet from the server (meaning your currently in the "server to client" thread) one shouldn't try to write a packet to the server using the "client to server" thread. The reason one doesn't want to do this is because the packets need encrypting and only one packet can be encrypted at a time. I needed to cross threads so that I could update my followers as to the whereabouts of my leader quickly. So my solution was to create another thread to handle the output of packets and this worked well.

Now after playing around with this I noticed that when the proxy is loaded with many clients and cpu usage starts to rise to 100%, shit hits the fan. Everything starts to bounce around (much like how @flyrocket's oryxdom would lag when around many objects, although to a lesser extent). So I figure this problem can fall into either of two categories, a) when the cpu hits 100% your fucked no matter what you do, or b) there is something wrong with how the proxy handles the packets. I'm hoping it is the latter and can be fixed. One idea I had was to create yet another thread that only reads in packets to a queue although I don't see much advantage to this as you'll need to look at them one by one anyways unless yet more threads are created to deal with multiple packets at a time. Anyone here have experience with this sort of stuff and can shed some light on what can be done to combat this lag?

this looks very interesting.. if we could actually make a release out of this you can become invincible

If you don't mid, could you describe your queing system, possibly with some code?
EDIT: how many explicit threads is your proxy using?

From what i see in the source i have (old furyhunter updated, thks again marinepower), there is indeed "2 thread" per connection (one reading from server and sending to client and one reading from client and sending to server) sharing the same instance of packetprocessor.
Its all i can say for sure.

Now i wonder something (not related) : how do you exactly use the MOVE packet to change the player position? Not that i really care or need but i am curious

were you on uswest2 yesterday? If so, thanks for a wis and attk.
My cpu can't handle 1 rotmg client >.< it's worth $150 new.
its ok on adobe flash player and low quality though. Still takes 100%. But somehow doesn't lag me. Just really low fps.

Originally Posted by krazyshank

If you don't mid, could you describe your queing system, possibly with some code?
EDIT: how many explicit threads is your proxy using?

Well 2 main threads, the policy server thread (don't know much about this, didn't mess with it) and the thread looking for new connections. Then for each client connected I have 4 threads: 1 connection thread (doesn't really do anything other than start the packet threads), 2 threads to handle client to server packets and server to client packets, and 1 packet writer thread. For the leader client, identified by its accountId, has an additional thread to request position updates. With 9 clients connected, I have 39 threads running.

The queuing system is just another class that extends the thread class. All packet writes go through this class:

Code:

import java****.DataOutputStream;
import java********Exception;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

import net.clarenceho.crypto.RC4;

import packets.Packet;

public class PacketWriter extends Thread {
	private static final int NANOSLEEP = 10000; // used so thread doesn't eat up cpu cycles
	private volatile boolean run = true;
	private Connection con;
	private RC4 rcENCR;
	private DataOutputStream out;
	private List<Packet> queue;
	
	PacketWriter(Connection con, RC4 rcEncr, DataOutputStream out) {
		Thread.currentThread().setPriority(Thread.MAX_PRIORITY);
		this.con = con;
		this.rcENCR = rcEncr;
		this.out = out;
		
		ArrayList<Packet> al = new ArrayList<Packet>();
		queue = Collections.synchronizedList(al);
	}
	
	public void run() {
		try {
			while (run) {
				if (queue.size() != 0) {
					//if (queue.size() > 1) { System.out.println("Queue: " + queue.size()); }
					Packet p = queue.remove(0);
					if (p != null) {
						writePacket(p);
					} else {
						System.err.println("How the fuck did this happen?");
					}
				}
				Thread.sleep(0, NANOSLEEP);
			}
		} catch (IOException | InterruptedException ioe) {
			//System.out.println("Error with packet writer.");
			//ioe.printStackTrace();
		}
		con.close();
	}
	
	private void writePacket(Packet p) throws IOException {
		byte[] encoded = rcENCR.rc4(p.data);
		out.writeInt(encoded.length + 5);
		out.writeByte(p.id);
		out.write(encoded);
		out.flush();
	}
	
	public void add(Packet p) {
		queue.add(p);
	}

	public void close() { // use to stop thread
		run = false;
	}
}

Originally Posted by JustAnoobROTMG

Initially I tried moving the followers via packets but this was a no go because the only packet that can actually move your character on screen is the goto packet. The movement provided by this was very choppy and created desyncs with what the client saw versus what the server saw and the server would disconnect me after a short bit of time doing this. So to answer your question, the client handles the move packet. All I did was looked at the existing code in the client used for movement and made a follow routine based off that code. So when the proxy starts sending packets to my followers letting them know the position of my leader, it first checks to see if the leader is on the same map as them and if so just walks towards the leader till within a small distance from.

---------- Post added at 03:09 PM ---------- Previous post was at 03:05 PM ----------

Originally Posted by 75dragon

Yea that was me. Your welcome.

I made a vid of me multiboxing (http://www.mpgh.net/forum/654-realm-...ml#post8637503). This client is heavily modified. If you pay attention you can spot several different hacks/changes that haven't been seen in the public.

Originally Posted by nilly

Code:

import java****.DataOutputStream;
import java********Exception;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

import net.clarenceho.crypto.RC4;

import packets.Packet;

public class PacketWriter extends Thread {
	private static final int NANOSLEEP = 10000; // used so thread doesn't eat up cpu cycles
	private volatile boolean run = true;
	private Connection con;
	private RC4 rcENCR;
	private DataOutputStream out;
	private List<Packet> queue;
	
	PacketWriter(Connection con, RC4 rcEncr, DataOutputStream out) {
		Thread.currentThread().setPriority(Thread.MAX_PRIORITY);
		this.con = con;
		this.rcENCR = rcEncr;
		this.out = out;
		
		ArrayList<Packet> al = new ArrayList<Packet>();
		queue = Collections.synchronizedList(al);
	}
	
	public void run() {
		try {
			while (run) {
				if (queue.size() != 0) {
					//if (queue.size() > 1) { System.out.println("Queue: " + queue.size()); }
					Packet p = queue.remove(0);
					if (p != null) {
						writePacket(p);
					} else {
						System.err.println("How the fuck did this happen?");
					}
				}
				Thread.sleep(0, NANOSLEEP);
			}
		} catch (IOException | InterruptedException ioe) {
			//System.out.println("Error with packet writer.");
			//ioe.printStackTrace();
		}
		con.close();
	}
	
	private void writePacket(Packet p) throws IOException {
		byte[] encoded = rcENCR.rc4(p.data);
		out.writeInt(encoded.length + 5);
		out.writeByte(p.id);
		out.write(encoded);
		out.flush();
	}
	
	public void add(Packet p) {
		queue.add(p);
	}

	public void close() { // use to stop thread
		run = false;
	}
}

If you're using this proxy: http://www.mpgh.net/forum/599-realm-...roxy-base.html
I'll see what I can do. I don't know how to make one so I guess it's time to learn since i've wanted to for a while, i'll convert that source to c# and go from there.

From that code snippet, I can't really see what the problem is.
My only concern being the
Thread.Sleep(0, NANOSLEEP); where NANOSLEEP == 10000. I'm not familiar with a thread.Sleep taking two arguments,
but do me a favor and try changing that line to:
Thread.Sleep(50); That'll tell the thread scheduler to sleep the thread ATLEAST 50ms, which in reality isn't even noticeable by us, but will limit the thread greatly.
Try experimenting.

I'll get back to you later as I do some conversion and find any other improvements. Possibly add me on skype: KrazyShank.
Good luck

Originally Posted by krazyshank

Yea you found the source I updated. I believe both marinepower and orxydom were based off this source (if you are familiar with them). As for the sleep thing, 50ms is way too long. If you think about it, this means only one packet will be sent every 50ms. It needs to run fast enough to keep the queue down close to 0 at all times. Since one of the applications is a real time update of my leaders position this delay would be woefully inadequate (updated every 16ms). Even though the delay is small (10,000 nano seconds) this seems to work well enough and doesn't cause too much cpu usage. I could possibly go larger to 100,000 nanoseconds but I know 1 ms is too long.

Originally Posted by nilly

Oh for some reason I was thinking that the thread sent each item in the queue at once, not 1 packet then sleep.

Originally Posted by nilly

Now after playing around with this I noticed that when the proxy is loaded with many clients and cpu usage starts to rise to 100%, shit hits the fan. Everything starts to bounce around (much like how @flyrocket's oryxdom would lag when around many objects, although to a lesser extent). So I figure this problem can fall into either of two categories, a) when the cpu hits 100% your fucked no matter what you do, or b) there is something wrong with how the proxy handles the packets. I'm hoping it is the latter and can be fixed. One idea I had was to create yet another thread that only reads in packets to a queue although I don't see much advantage to this as you'll need to look at them one by one anyways unless yet more threads are created to deal with multiple packets at a time. Anyone here have experience with this sort of stuff and can shed some light on what can be done to combat this lag?

Unfortunately, it's looking like that is going to be the case. Ikulip's source isn't that slow, definitely not slow enough to require a full rewrite. The issue is Flash eating up all of your cpu. With as many clients as you have open, I'm surprised at how well they work as it is, judging by that video. Flash is the exact reason that there was work on a separate client in Java (among other reasons, but that was a big reason). Flash just uses way too many resources.

The only things I can suggest is using less clients (which would net more loot anyway) and, if you aren't already, setting the flash clients to low quality.

If you do figure out anything, let us know, but I'm 95% positive (without seeing source) that Flash is to blame here.

@krazyshank, I could have done that. I figured since it loops over and over there was no reason to do so (but that was before I added the delay). I think I might change it to empty the queue with each run and see how that goes.

Originally Posted by nilly

I would definitely clear the queue by sending any packets that are in it before sleeping again. With how small your sleeps are, I doubt this is the issue, and you may not see any major performance gains, but that would be the optimal way for sure.

@nilly @dwdude What is the function of the "Plotter" class? I'm having difficulties rewriting it in C#. I dont see the purpose of making new graphics object and bitmaps in a proxy?

Originally Posted by krazyshank

@nilly @dwdude What is the function of the "Plotter" class? I'm having difficulties rewriting it in C#. I dont see the purpose of making new graphics object and bitmaps in a proxy?

It isn't used. Ignore it.

Originally Posted by dwdude

@dwdude, Thanks for the input. Two of my characters died likely due to the delay induced by the proxy and the 100% cpu usage. I'm think about rewriting the proxy so that one thread handles both inputs from the client and the server and adds them to a queue and then another thread handling both outputs to the client and server. A third thread (the mediator) would be used to look at the input queues and make changes and what not and pass them to the output queues. I also want to prioritize the processing of packets so that packets like escape get sent out immediately without touching the mediating thread. Then set the priority of the input/output threads to high and maybe lessen the priority of the mediating thread. Just some ideas I had that might help some. It is unfortunate that cpu usage has such an impact on the proxy though. :/

---------- Post added at 05:48 PM ---------- Previous post was at 05:43 PM ----------

I just thought of something @dwdude. I've been multiboxing with 9 clients for a while now. I only added the proxy recently to help with character sync issues. When playing without the proxy I didn't have the type of lag that I do with the proxy. Perhaps I just never realized there was lag because player movement with w,a,s,d isn't affected by choke ups with the network input/output.

A more efficient proxy...

Post a Reply

Similar Threads

Tags for this Thread