How to decode forex.com's foreign exchange protocol

What is forex.com & What I do

forex.com is lead vendor for online foreign exchange. It trade 200 billion dollar per month.

I'm interested with foreign exchange rate data, and I want to verify my mathematical model. So I need some real data. forex.com is a good choice.

I first decoded forex.com's foreign exchange communication protocol at Jul-2007. After that, forex.com changed protocol 4 times. Last time I decoded it at Sep-2011, and the program in this article is related to this version.

The protocol decode's product is re-implement a forex.com client by python.

Protocol

Protocol of forex.com's foreign exchange have a lead protocol header, and follow with protocol body. The protocol body repeat many times. If you don't disconnect, data flow will stop after 4 hours. So you need a mechanism to reconnect your client.

The first char of protocol data flow is "S", it is not important, you can drop it. Maybe this char is just for client to confirm this protocol.

You can use regular expression to recognize protocol. It's simpler than your code by manual.

The protocol header is recognize by the regex below:

RE_HEADER=re.compile(r"""^(?P\d+)\\(?P\w{3})\/(?P\w{3})\\(?P[\.\d]+)\\(?P[\.\d]+)\\(?P[\.\d]+)\\(?P[\.\d]+)\\(?P[DR])\\(?P[AE])\\(?P\d)\\(?P[\.\d]+)\\$""")

This pattern will repeat many times, depend on how many foreign currency is in exchange. After 2010, forex.com include gold information in protocol. At Sep-2011, there are 74 currencies pair in forex.com. Protocol Header also has 74 tuples.

"keyname" is useful for protocol body to match data tuple, because they do not want repeat the currency name again in protocol body. "p1" & "p2" are two currency name, like "USD" or "AUD".

"bid", "ask", "high", "low" is past recent data. You do not need it, you will get more fresh data in protocol body. These are just initial data.

I do not know what is "dr", "ae", "num", and "close", but I think these are not important.

There is a example here for AUD/CAD rate, in python dict:

{'p2': 'CAD', 'p1': 'AUD', 'ae': 'A', 'bid': '1.01494', 'high': '1.01890', 'num': '5', 'low': '1.01390', 'ask': '1.01530', 'close': '1.01745', 'dr': 'D', 'keyname': '22'}

If you finished protocol header, there is protocol body. The regex for protocol body is below:

RE_BODY=re.compile(r"""^R(?P\d+)\\(?P[\.\d]+)\\(?P[\.\d]+)\\(?P[DR])\\\\\\(?P\d{2}\/\d{2}\/\d{4} \d{2}:\d{2}:\d{2})\\$""")

Every time your client received data from server, there are several pairs of tuple. The regex is just for 1 tuple. If you want to record these data, you can print out time now, and then print out tuple at this time.

"keyname" is same to protocol header, for recognize currency pair. "bid" and "ask" is bid and ask price. I do not know what is "dr" too. "datetime" do not need my explanation, I think it is not useful, just increase communication bandwidth.

One line of the data I caught is here, it just a short line:

11:54:06 {'GBP/JPY':{'ask':'120.923','bid':'120.876'},'JPX/JPY':{'ask':'8609','bid':'8581'},'SGD/JPY':{'ask':'61.553','bid':'61.515'},'AUD/USD':{'ask':'1.02326','bid':'1.02302'},'CAD/JPY':{'ask':'77.285','bid':'77.237'},'XAU/AUD':{'ask':'1775.27','bid':'1774.02'}}

The protocol above is from server to client. You need send a request to start a communicate. Request is just a string. One available IP and port is 74.217.51.143:443. Do not panic, this is not SSL protocol. And a available request string is:

"TKN0Hrs5KPx5H6dyboaNcjcS83Rg0sme/kbhufsh0ME4/l4LDf8v35/ZIqJUQj4aUkmr+KHwzD5WniNFhfd5K8YHPUN8TZQh2D8tXy07EAzoQNfypaOjkcubOcF8dDRil4ToBrsYugEF30mTZH843+Xyw==".

Notice, these IP/port and request string is not available now. If you want to do more interesting thing, you must get it by your sniffer.

What is the use for you

You can fetch real world foreign exchange data, and verify your mathematical model.

I think the analysis for forex.com protocol is not bad thing to forex.com. And this analysis is not danger for forex.com.

If you want to talk me about mathematical model about forex trade, contact me.