The Design Rules of API

=========================
The Design Rules of API
=========================

:Author: gashero
:Date: 2012-02-24

Good API is usable for caller, but bad API is hard to debug. This article will introduce some design rule of API in my opinion. Every one of the rules is base on some failure's lesson.

1. Separate ID-LIST and GET-ONE

Your API need one of ID-LIST or GET-ONE. The ID-LIST API will return a list of object_id, but no detail information about the object. The GET-ONE API will return only ONE object's full information (written by gashero).

If your API designed by this rule, most requirement will be cover by these two type of API. Someone get list, someone get detail. ID-LIST API will return in limit time.

If NOT? If you develop API for every requirement, you will get many API, and many repeat code. API will be not reuseful. And if a requirement need a new object's attribution, you must modify the API. But if you use ID-LIST & GET-ONE, you just only need modify the Application's code, GET-ONE API has enough attribution.

2. Dataset slice, count & offset

Every ID-LIST API need dataset slice, by count & offset. If your requirement do not mention it now, it will be appear later. So, if you have designed it, you do not need modify it later.

Notice some language allow parameter have default value, you can give a default value 0 to offset, but DO NOT give default value to count. Because caller maybe ignore it, and then cause some bug. For example, caller maybe want to get all object id list, but he forget the count parameter, the default count will lost some id.

3. Start with action

API's naming is a difficult task, because you need a meanful name, but avoid same to other API's name. Name a API start with a action is good choice. Like Hungarian notation, name lead with it's class, not data type.

Some useful prefix of API's name:

1. add: create an new object.
2. delete: delete an object.
3. update: modify an object's 1 field value.
4. get: get 1 object's full detail by object_id.
5. list: get list of object's id match some rule.
6. count: get count of object match some rule.

4. Do not call eachother

API would not call eachother, avoid depend on eachother. Independent or reuse, most time, independent is more useful. Sometimes, you may think something is reusable, but actually it not.

5. Set operate

If a API focus update set of objects. Right way is just add/delete the element in the set, not write whole set. Because, write whole set need large communication bandwidth, and this method can not get right result in concurrent environment.

6. No super API

If you provide a API have many feature, caller will abuse it everywhere. When you want to do some reconstruction, these API will kill you. Because you do not know caller how to use it.

Super API break the API, make API not clear, make caller inject the internel of your implemention.

7. More debug information, not "return NULL"

Debug is more important than performance. "return NULL" is not a good idea for debug, because it transfer nothing to caller.

You need define a exceptioin class, with some field to tell caller what is wrong. Some useful field:

1. errnum: error number, can be recognize by program.
2. errmsg: error message, can be read by people.

8. Update API, (id, field, value)

Modify some object's field just need these three parameters. Select an object by object_id, modify its field to value. Field can be select by a string, this parttern can match many requirement.

9. More string as type, no magic number

There is many type field in interface or database design. A string can be readable, it's more friendly to debug. But if a magic number as type field in a large system, caller will crash.

10. Separate interface and logic

Interface is sometimes very simple without logic, too simple to test. But if logic in interface, it's hard to test.

Automatic test is very good for your software quality. So separate interface and logic will make logic suit for test. Unittest will help you increase quality.

11. Do not mix code

Mix code will decrease your program's readability. If you write SQL in your function, code will become confusion.

You can write SQL on top of your source file, and refer in your code.

How to decode forex.com's foreign exchange protocol

What is forex.com & What I do

forex.com is lead vendor for online foreign exchange. It trade 200 billion dollar per month.

I'm interested with foreign exchange rate data, and I want to verify my mathematical model. So I need some real data. forex.com is a good choice.

I first decoded forex.com's foreign exchange communication protocol at Jul-2007. After that, forex.com changed protocol 4 times. Last time I decoded it at Sep-2011, and the program in this article is related to this version.

The protocol decode's product is re-implement a forex.com client by python.

Protocol

Protocol of forex.com's foreign exchange have a lead protocol header, and follow with protocol body. The protocol body repeat many times. If you don't disconnect, data flow will stop after 4 hours. So you need a mechanism to reconnect your client.

The first char of protocol data flow is "S", it is not important, you can drop it. Maybe this char is just for client to confirm this protocol.

You can use regular expression to recognize protocol. It's simpler than your code by manual.

The protocol header is recognize by the regex below:

RE_HEADER=re.compile(r"""^(?P\d+)\\(?P\w{3})\/(?P\w{3})\\(?P[\.\d]+)\\(?P[\.\d]+)\\(?P[\.\d]+)\\(?P[\.\d]+)\\(?P[DR])\\(?P[AE])\\(?P\d)\\(?P[\.\d]+)\\$""")

This pattern will repeat many times, depend on how many foreign currency is in exchange. After 2010, forex.com include gold information in protocol. At Sep-2011, there are 74 currencies pair in forex.com. Protocol Header also has 74 tuples.

"keyname" is useful for protocol body to match data tuple, because they do not want repeat the currency name again in protocol body. "p1" & "p2" are two currency name, like "USD" or "AUD".

"bid", "ask", "high", "low" is past recent data. You do not need it, you will get more fresh data in protocol body. These are just initial data.

I do not know what is "dr", "ae", "num", and "close", but I think these are not important.

There is a example here for AUD/CAD rate, in python dict:

{'p2': 'CAD', 'p1': 'AUD', 'ae': 'A', 'bid': '1.01494', 'high': '1.01890', 'num': '5', 'low': '1.01390', 'ask': '1.01530', 'close': '1.01745', 'dr': 'D', 'keyname': '22'}

If you finished protocol header, there is protocol body. The regex for protocol body is below:

RE_BODY=re.compile(r"""^R(?P\d+)\\(?P[\.\d]+)\\(?P[\.\d]+)\\(?P[DR])\\\\\\(?P\d{2}\/\d{2}\/\d{4} \d{2}:\d{2}:\d{2})\\$""")

Every time your client received data from server, there are several pairs of tuple. The regex is just for 1 tuple. If you want to record these data, you can print out time now, and then print out tuple at this time.

"keyname" is same to protocol header, for recognize currency pair. "bid" and "ask" is bid and ask price. I do not know what is "dr" too. "datetime" do not need my explanation, I think it is not useful, just increase communication bandwidth.

One line of the data I caught is here, it just a short line:

11:54:06 {'GBP/JPY':{'ask':'120.923','bid':'120.876'},'JPX/JPY':{'ask':'8609','bid':'8581'},'SGD/JPY':{'ask':'61.553','bid':'61.515'},'AUD/USD':{'ask':'1.02326','bid':'1.02302'},'CAD/JPY':{'ask':'77.285','bid':'77.237'},'XAU/AUD':{'ask':'1775.27','bid':'1774.02'}}

The protocol above is from server to client. You need send a request to start a communicate. Request is just a string. One available IP and port is 74.217.51.143:443. Do not panic, this is not SSL protocol. And a available request string is:

"TKN0Hrs5KPx5H6dyboaNcjcS83Rg0sme/kbhufsh0ME4/l4LDf8v35/ZIqJUQj4aUkmr+KHwzD5WniNFhfd5K8YHPUN8TZQh2D8tXy07EAzoQNfypaOjkcubOcF8dDRil4ToBrsYugEF30mTZH843+Xyw==".

Notice, these IP/port and request string is not available now. If you want to do more interesting thing, you must get it by your sniffer.

What is the use for you

You can fetch real world foreign exchange data, and verify your mathematical model.

I think the analysis for forex.com protocol is not bad thing to forex.com. And this analysis is not danger for forex.com.

If you want to talk me about mathematical model about forex trade, contact me.

Look out, if I'm angry, I can explode

2010-11-02 HangZhou, ZheJiang.

ATtiny13 as IR receiver

ATtiny13 as IR receiver

IR controller is everywhere in your home, but you can not control anything. But if you can follow me to make a very cheap PCB with ATtiny13, you can control most equipment in your home. Out of your home? Maybe more interesting.

What's that stuff?

Generally, it's a PCB can receive IR signal from your IR controller, and control other things like light or TV.Just with a relay.

Why ATtiny13?

ATtiny13 is an ATMEL product, AVR family. This chip is very cheap. If your chip is cheap enough, you can use it anywhere you want.

The chip's power:

1. Small: only 8 pin, 6 GPIO

2. Cheap: $0.3

3. Powerful: 1KB Flash, 64 Byte RAM, 64 Byte EEPROM, up to 20MHz clock

4. Save energy: 240 uA @ 1MHz, <1uA @ power-down mode

5. Most important reason: I like it, :-)

How to recognize IR signal?

Most popular IR signal is defined by NEC. First 9ms signal, 4.5ms no signal, and then is 4 byte data, front 2 byte is address, last 2 byte is reverse to each other.

If chip's software, you need State Machine to change state by input signal right now, and put it in 1 bit. Because ATtiny13 only have 64 Byte RAM, so you can not save all voltage one by one. Transform it immediately, you will only need 4 Byte RAM.

Want debug it? ATtiny13 do not have a serial port, so you can write some status to EEPROM, and read it out later. Maybe it's hard, but it's cheap.

Some IR data

My MacRemote's address code is 0x77e1, yours maybe difference. Signal of buttons:

OK=0x3a5d

Vol+=0x505d

Vol-=0x305d

<<=0x905d

>>=0x605d

MENU=0xc05d

>||=0xfa5d

More feature with IR

IR is very useful for control, you can use it transfer serial port with 9600 bps. And you can use anyother device as adapter between IR and other channel, for example your cell phone as adapter communicate with GSM/3G/4G network.

If you are interested with IR, contact me.