Socket网络编程
socket通常也称作"套接字"，用于描述IP地址和端口，是一个通信链的句柄，应用程序通常通过"套接字"向网络发出请求或者应答网络请求。
socket起源于Unix，而Unix/Linux基本哲学之一就是“一切皆文件”，对于文件用打开、读写、关闭模式来操作。socket就是该模式的一个实现，socket即是一种特殊的文件，一些socket函数就是对其进行的操作（读/写IO、打开、关闭）
socket和file的区别：
file模块是针对某个指定文件进行打开、读写、关闭
socket模块是针对 服务器端 和 客户端Socket 进行打开、读写、关闭
Socket的英文原义是“孔”或“插座”。Socket正如其英文原意那样，像一个多孔插座。一台主机犹如布满各种插座的房间，每个插座有一个编号，有的插座提供220伏交流电， 有的提供110伏交流电，有的则提供有线电视节目。 客户软件将插头插到不同编号的插座，就可以得到不同的服务。
我知道上面那段话很枯燥，那就忘记吧！我们只需要记得：两个程序通过“网络”交互数据就使用socket，它只负责两件事：建立连接，传递数据。不能再简练了！！！
一个完整的socket通信流程大致如下图所示
实际socket服务器和和客户端通信的过程有些类似日常生活中拨打电话的过程，需要有手机，有电话卡，开机，拨号，建立连接，发送消息一直到通话结束，中间也有其他人打进电话的情况出现，
这个跟socket客户端连接服务器端，服务器端正在连接客户端时会将新进来的连接放到连接池中处于等待状态，下面看下代码怎么实现的：
下面来实现最简单的一对一拨打电话过程
Client端
建立连接，输入交互命令
Server端
socket的实现就像是打电话的过程，接收电话的是server端，拨打电话是client端，send和recv中的内容就是我们要发送和接受的东西，那么准备好了要发送的东西，我们要告诉手机打电话的目标电话，看server中的sock.bind方法，这里面传了一个元组('ip','port')给bind方法，就是把自己的电话开机处于接听状态；而client端的obj_client.connect方法是告诉自己手机终端，去找哪个号码建立通信。所以这个元组中的ip和port内容必须一致，是接听和拨打电话双方约定好的。我send的内容无所谓是什么，我send什么，客户端就recv什么，因为我发送的什么，接收到的就是什么，他没有权利
左右我发送什么东西。
#!/usr/bin/env python # -*- coding:utf-8 -*- # Author: Jason Wangimport socket import subprocess #导入执行命令模块 ip_port=('127.0.0.1',9999) #定义元祖 #买手机 s=socket.socket() #绑定协议，生成套接字 s.bind(ip_port) #绑定ip+协议+端口：用来唯一标识一个进程，ip_port必须是元组格式 s.listen(5) #定义最大可以挂起链接数 #等待电话 while True: #用来重复接收新的链接conn,addr=s.accept() #接收客户端胡的接请求，返回conn（相当于一个特定胡链接），addr是客户端ip+portconn.sendall(bytes('欢迎致电 10086，请输入1xxx,0转人工服务.',encoding='utf-8'))#收消息while True: #用来基于一个链接重复收发消息try: #捕捉客户端异常关闭（ctrl+c）recv_data=conn.recv(1024) #收消息，阻塞if len(recv_data) == 0:break #客户端如果退出，服务端将收到空消息，退出#发消息p=subprocess.Popen(str(recv_data,encoding='utf8'),shell=True,stdout=subprocess.PIPE) #执行系统命令，windows平# 台命令的标准输出是gbk编码，需要转换res=p.stdout.read() #获取标准输出if len(res) == 0: #执行错误命令，标准输出为空，send_data='cmd err'# else:# send_data=str(res,encoding='gbk') #命令执行ok，字节gbk---->str---->字节utf-8print(str)send_data=bytes(str(res,encoding='utf8'),encoding='utf8')#解决粘包问题ready_tag='Ready|%s' %len(send_data)conn.send(bytes(ready_tag,encoding='utf8')) #发送数据长度feedback=conn.recv(1024) #接收确认信息feedback=str(feedback,encoding='utf8')if feedback.startswith('Start'):conn.send(send_data) #发送命令的执行结果except Exception:break#挂电话conn.close()
#!/usr/bin/env python # -*- coding:utf-8 -*- # Author: Jason Wangimport socket ip_port=('127.0.0.1',9999) #买手机 s=socket.socket() #拨号 s.connect(ip_port) #链接服务端，如果服务已经存在一个好的连接，那么挂起 welcom_msg = s.recv(200).decode()#获取服务端欢迎消息 print(welcom_msg) while True: #基于connect建立的连接来循环发送消息send_data=input(">>: ").strip()if send_data == 'exit':breakif len(send_data) == 0:continues.send(bytes(send_data,encoding='utf8'))#解决粘包问题ready_tag=s.recv(1024) #收取带数据长度的字节：Ready|9998ready_tag=str(ready_tag,encoding='utf8')if ready_tag.startswith('Ready'):#Ready|9998msg_size=int(ready_tag.split('|')[-1]) #获取待接收数据长度start_tag='Start's.send(bytes(start_tag,encoding='utf8')) #发送确认信息#基于已经收到的待接收数据长度，循环接收数据recv_size=0recv_msg=b''while recv_size < msg_size:recv_data=s.recv(1024)recv_msg+=recv_datarecv_size+=len(recv_data)print('MSG SIZE %s RECE SIZE %s' %(msg_size,recv_size))print(str(recv_msg,encoding='utf8'))#挂电话 s.close()
WEB服务应用：
#!/usr/bin/env python
#coding:utf-8
import socketdef handle_request(client):buf = client.recv(1024)client.send("HTTP/1.1 200 OK\r\n\r\n")client.send("Hello, World")def main():sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)sock.bind(('localhost',8080))sock.listen(5)while True:connection, address = sock.accept()handle_request(connection)connection.close()if __name__ == '__main__':main()更多功能
sk = socket.socket(socket.AF_INET,socket.SOCK_STREAM,0)
参数一：地址簇socket.AF_INET IPv4（默认）socket.AF_INET6 IPv6socket.AF_UNIX 只能够用于单一的Unix系统进程间通信参数二：类型socket.SOCK_STREAM 流式socket , for TCP （默认）socket.SOCK_DGRAM 数据报式socket , for UDPsocket.SOCK_RAW 原始套接字，普通的套接字无法处理ICMP、IGMP等网络报文，而SOCK_RAW可以；其次，SOCK_RAW也可以处理特殊的IPv4报文；此外，利用原始套接字，可以通过IP_HDRINCL套接字选项由用户构造IP头。socket.SOCK_RDM 是一种可靠的UDP形式，即保证交付数据报但不保证顺序。SOCK_RAM用来提供对原始协议的低级访问，在需要执行某些特殊操作时使用，如发送ICMP报文。SOCK_RAM通常仅限于高级用户或管理员运行的程序使用。socket.SOCK_SEQPACKET 可靠的连续数据包服务参数三：协议0 （默认）与特定的地址家族相关的协议,如果是 0 ，则系统就会根据地址格式和套接类别,自动选择一个合适的协议import socket ip_port = ('127.0.0.1',9999) sk = socket.socket(socket.AF_INET,socket.SOCK_DGRAM,0) sk.bind(ip_port)while True:data = sk.recv(1024)print dataimport socket ip_port = ('127.0.0.1',9999)sk = socket.socket(socket.AF_INET,socket.SOCK_DGRAM,0) while True:inp = raw_input('数据：').strip()if inp == 'exit':breaksk.sendto(inp,ip_port)sk.close()
sk.bind(address)
s.bind(address) 将套接字绑定到地址。address地址的格式取决于地址族。在AF_INET下，以元组（host,port）的形式表示地址。
sk.listen(backlog)
开始监听传入连接。backlog指定在拒绝连接之前，可以挂起的最大连接数量。
backlog等于5，表示内核已经接到了连接请求，但服务器还没有调用accept进行处理的连接个数最大为5
这个值不能无限大，因为要在内核中维护连接队列
sk.setblocking(bool)
是否阻塞（默认True），如果设置False，那么accept和recv时一旦无数据，则报错。
sk.accept()
接受连接并返回（conn,address）,其中conn是新的套接字对象，可以用来接收和发送数据。address是连接客户端的地址。
接收TCP 客户的连接（阻塞式）等待连接的到来
sk.connect(address)
连接到address处的套接字。一般，address的格式为元组（hostname,port）,如果连接出错，返回socket.error错误。
sk.connect_ex(address)
同上，只不过会有返回值，连接成功时返回 0 ，连接失败时候返回编码，例如：10061
sk.close()
关闭套接字
sk.recv(bufsize[,flag])
接受套接字的数据。数据以字符串形式返回，bufsize指定最多可以接收的数量。flag提供有关消息的其他信息，通常可以忽略。
sk.recvfrom(bufsize[.flag])
与recv()类似，但返回值是（data,address）。其中data是包含接收数据的字符串，address是发送数据的套接字地址。
sk.send(string[,flag])
#Python 2将string中的数据发送到连接的套接字。返回值是要发送的字节数量，该数量可能小于string的字节大小。即：可能未将指定内容全部发送。
#Python 3必须转换为字节形式bytes发送
sk.sendall(string[,flag])
#将string中的数据发送到连接的套接字，但在返回之前会尝试发送所有数据。成功返回None，失败则抛出异常。
#Python 3必须转换为字节形式bytes发送
内部通过递归调用send，将所有内容发送出去。
sk.sendto(string[,flag],address)
将数据发送到套接字，address是形式为（ipaddr，port）的元组，指定远程地址。返回值是发送的字节数。该函数主要用于UDP协议。
sk.settimeout(timeout)
设置套接字操作的超时期，timeout是一个浮点数，单位是秒。值为None表示没有超时期。一般，超时期应该在刚创建套接字时设置，因为它们可能用于连接的操作（如 client 连接最多等待5s ）
sk.getpeername()
返回连接套接字的远程地址。返回值通常是元组（ipaddr,port）。
sk.getsockname()
返回套接字自己的地址。通常是一个元组(ipaddr,port)
sk.fileno()
套接字的文件描述符
SocketServer模块
SocketServer内部使用 IO多路复用 以及 “多线程” 和 “多进程” ，从而实现并发处理多个客户端请求的Socket服务端。即：每个客户端请求连接到服务器时，Socket服务端都会在服务器是创建一个“线程”或者“进程” 专门负责处理当前客户端的所有请求。
ThreadingTCPServer
ThreadingTCPServer实现的Soket服务器内部会为每个client创建一个 “线程”，该线程用来和客户端进行交互。
1、ThreadingTCPServer基础
使用ThreadingTCPServer:
- 创建一个继承自 SocketServer.BaseRequestHandler 的类
- 类中必须定义一个名称为 handle 的方法
- 启动ThreadingTCPServer
#!/usr/bin/env python # -*- coding:utf-8 -*- import SocketServerclass MyServer(SocketServer.BaseRequestHandler):def handle(self):# print self.request,self.client_address,self.serverconn = self.requestconn.sendall('欢迎致电 10086，请输入1xxx,0转人工服务.')Flag = Truewhile Flag:data = conn.recv(1024)if data == 'exit':Flag = Falseelif data == '0':conn.sendall('通过可能会被录音.balabala一大推')else:conn.sendall('请重新输入.')if __name__ == '__main__':server = SocketServer.ThreadingTCPServer(('127.0.0.1',8009),MyServer)server.serve_forever()
#!/usr/bin/env python # -*- coding:utf-8 -*-import socketip_port = ('127.0.0.1',8009) sk = socket.socket() sk.connect(ip_port) sk.settimeout(5)while True:data = sk.recv(1024)print 'receive:',datainp = raw_input('please input:')sk.sendall(inp)if inp == 'exit':breaksk.close()
2、ThreadingTCPServer源码剖析
内部调用流程为：
- 启动服务端程序
- 执行 TCPServer.__init__ 方法，创建服务端Socket对象并绑定 IP 和 端口
- 执行 BaseServer.__init__ 方法，将自定义的继承自SocketServer.BaseRequestHandler 的类 MyRequestHandle赋值给self.RequestHandlerClass
- 执行 BaseServer.server_forever 方法，While 循环一直监听是否有客户端请求到达 ...
- 当客户端连接到达服务器
- 执行 ThreadingMixIn.process_request 方法，创建一个 “线程” 用来处理请求
- 执行 ThreadingMixIn.process_request_thread 方法
- 执行 BaseServer.finish_request 方法，执行 self.RequestHandlerClass() 即：执行 自定义 MyRequestHandler 的构造方法（自动调用基类BaseRequestHandler的构造方法，在该构造方法中又会调用 MyRequestHandler的handle方法）
ThreadingTCPServer相关源码：
class BaseServer:"""Base class for server classes.Methods for the caller:- __init__(server_address, RequestHandlerClass)- serve_forever(poll_interval=0.5)- shutdown()- handle_request() # if you do not use serve_forever()- fileno() -> int # for select()Methods that may be overridden:- server_bind()- server_activate()- get_request() -> request, client_address- handle_timeout()- verify_request(request, client_address)- server_close()- process_request(request, client_address)- shutdown_request(request)- close_request(request)- handle_error()Methods for derived classes:- finish_request(request, client_address)Class variables that may be overridden by derived classes orinstances:- timeout- address_family- socket_type- allow_reuse_addressInstance variables:- RequestHandlerClass- socket"""timeout = Nonedef __init__(self, server_address, RequestHandlerClass):"""Constructor. May be extended, do not override."""self.server_address = server_addressself.RequestHandlerClass = RequestHandlerClassself.__is_shut_down = threading.Event()self.__shutdown_request = Falsedef server_activate(self):"""Called by constructor to activate the server.May be overridden."""passdef serve_forever(self, poll_interval=0.5):"""Handle one request at a time until shutdown.Polls for shutdown every poll_interval seconds. Ignoresself.timeout. If you need to do periodic tasks, do them inanother thread."""self.__is_shut_down.clear()try:while not self.__shutdown_request:# XXX: Consider using another file descriptor or# connecting to the socket to wake this up instead of# polling. Polling reduces our responsiveness to a# shutdown request and wastes cpu at all other times.r, w, e = _eintr_retry(select.select, [self], [], [],poll_interval)if self in r:self._handle_request_noblock()finally:self.__shutdown_request = Falseself.__is_shut_down.set()def shutdown(self):"""Stops the serve_forever loop.Blocks until the loop has finished. This must be called whileserve_forever() is running in another thread, or it willdeadlock."""self.__shutdown_request = Trueself.__is_shut_down.wait()# The distinction between handling, getting, processing and# finishing a request is fairly arbitrary. Remember:# # - handle_request() is the top-level call. It calls# select, get_request(), verify_request() and process_request()# - get_request() is different for stream or datagram sockets# - process_request() is the place that may fork a new process# or create a new thread to finish the request# - finish_request() instantiates the request handler class;# this constructor will handle the request all by itselfdef handle_request(self):"""Handle one request, possibly blocking.Respects self.timeout."""# Support people who used socket.settimeout() to escape# handle_request before self.timeout was available.timeout = self.socket.gettimeout()if timeout is None:timeout = self.timeoutelif self.timeout is not None:timeout = min(timeout, self.timeout)fd_sets = _eintr_retry(select.select, [self], [], [], timeout)if not fd_sets[0]:self.handle_timeout()returnself._handle_request_noblock()def _handle_request_noblock(self):"""Handle one request, without blocking.I assume that select.select has returned that the socket isreadable before this function was called, so there should beno risk of blocking in get_request()."""try:request, client_address = self.get_request()except socket.error:returnif self.verify_request(request, client_address):try:self.process_request(request, client_address)except:self.handle_error(request, client_address)self.shutdown_request(request)def handle_timeout(self):"""Called if no new request arrives within self.timeout.Overridden by ForkingMixIn."""passdef verify_request(self, request, client_address):"""Verify the request. May be overridden.Return True if we should proceed with this request."""return Truedef process_request(self, request, client_address):"""Call finish_request.Overridden by ForkingMixIn and ThreadingMixIn."""self.finish_request(request, client_address)self.shutdown_request(request)def server_close(self):"""Called to clean-up the server.May be overridden."""passdef finish_request(self, request, client_address):"""Finish one request by instantiating RequestHandlerClass."""self.RequestHandlerClass(request, client_address, self)def shutdown_request(self, request):"""Called to shutdown and close an individual request."""self.close_request(request)def close_request(self, request):"""Called to clean up an individual request."""passdef handle_error(self, request, client_address):"""Handle an error gracefully. May be overridden.The default is to print a traceback and continue."""print '-'*40print 'Exception happened during processing of request from',print client_addressimport tracebacktraceback.print_exc() # XXX But this goes to stderr!print '-'*40
class TCPServer(BaseServer):"""Base class for various socket-based server classes.Defaults to synchronous IP stream (i.e., TCP).Methods for the caller:- __init__(server_address, RequestHandlerClass, bind_and_activate=True)- serve_forever(poll_interval=0.5)- shutdown()- handle_request() # if you don't use serve_forever()- fileno() -> int # for select()Methods that may be overridden:- server_bind()- server_activate()- get_request() -> request, client_address- handle_timeout()- verify_request(request, client_address)- process_request(request, client_address)- shutdown_request(request)- close_request(request)- handle_error()Methods for derived classes:- finish_request(request, client_address)Class variables that may be overridden by derived classes orinstances:- timeout- address_family- socket_type- request_queue_size (only for stream sockets)- allow_reuse_addressInstance variables:- server_address- RequestHandlerClass- socket"""address_family = socket.AF_INETsocket_type = socket.SOCK_STREAMrequest_queue_size = 5allow_reuse_address = Falsedef __init__(self, server_address, RequestHandlerClass, bind_and_activate=True):"""Constructor. May be extended, do not override."""BaseServer.__init__(self, server_address, RequestHandlerClass)self.socket = socket.socket(self.address_family,self.socket_type)if bind_and_activate:try:self.server_bind()self.server_activate()except:self.server_close()raisedef server_bind(self):"""Called by constructor to bind the socket.May be overridden."""if self.allow_reuse_address:self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)self.socket.bind(self.server_address)self.server_address = self.socket.getsockname()def server_activate(self):"""Called by constructor to activate the server.May be overridden."""self.socket.listen(self.request_queue_size)def server_close(self):"""Called to clean-up the server.May be overridden."""self.socket.close()def fileno(self):"""Return socket file number.Interface required by select()."""return self.socket.fileno()def get_request(self):"""Get the request and client address from the socket.May be overridden."""return self.socket.accept()def shutdown_request(self, request):"""Called to shutdown and close an individual request."""try:#explicitly shutdown. socket.close() merely releases#the socket and waits for GC to perform the actual close. request.shutdown(socket.SHUT_WR)except socket.error:pass #some platforms may raise ENOTCONN here self.close_request(request)def close_request(self, request):"""Called to clean up an individual request."""request.close()
class ThreadingMixIn:"""Mix-in class to handle each request in a new thread."""# Decides how threads will act upon termination of the# main processdaemon_threads = Falsedef process_request_thread(self, request, client_address):"""Same as in BaseServer but as a thread.In addition, exception handling is done here."""try:self.finish_request(request, client_address)self.shutdown_request(request)except:self.handle_error(request, client_address)self.shutdown_request(request)def process_request(self, request, client_address):"""Start a new thread to process the request."""t = threading.Thread(target = self.process_request_thread,args = (request, client_address))t.daemon = self.daemon_threadst.start()
class ThreadingTCPServer(ThreadingMixIn, TCPServer): pass
RequestHandler相关源码
lass BaseRequestHandler:"""Base class for request handler classes.This class is instantiated for each request to be handled. Theconstructor sets the instance variables request, client_addressand server, and then calls the handle() method. To implement aspecific service, all you need to do is to derive a class whichdefines a handle() method.The handle() method can find the request as self.request, theclient address as self.client_address, and the server (in case itneeds access to per-server information) as self.server. Since aseparate instance is created for each request, the handle() methodcan define arbitrary other instance variariables."""def __init__(self, request, client_address, server):self.request = requestself.client_address = client_addressself.server = serverself.setup()try:self.handle()finally:self.finish()def setup(self):passdef handle(self):passdef finish(self):pass
实例
#!/usr/bin/env python # -*- coding:utf-8 -*- # Author: Jason Wangimport socketserver import subprocessclass MyServer(socketserver.BaseRequestHandler):def handle(self):self.request.sendall(bytes('欢迎致电10086，请输入1xx,0转入人工服务',encoding='utf-8'))while True:data = self.request.recv(1024)print('------>%s'%(len(data)))if len(data) == 0:breakprint("[%s] says:%s" %(self.client_address,data.decode()))#self.request.sendall(data.upper())cmd = subprocess.Popen(data.decode(),shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)cmd_res = cmd.stdout.read()if not cmd_res:cmd_res = cmd.stderr.read()if len(cmd_res) == 0:#cmd has no outputcmd_res = bytes('cmd has output',encoding='utf-8')# self.request.sendall(data.upper()) self.request.send(cmd_res)if __name__ == '__main__':server = socketserver.ThreadingTCPServer(('0.0.0.0',9999),MyServer)server.serve_forever()
#!/usr/bin/env python # -*- coding:utf-8 -*- # Author: Jason Wang import socket ip_port=('0.0.0.0',9999) #买手机 s=socket.socket() #拨号 s.connect(ip_port) #发送消息 welcome_msg = s.recv(1024) print("from server:",welcome_msg.decode()) while True:send_data=input(">>: ").strip()if len(send_data) == 0:continues.send(bytes(send_data,encoding='utf8'))#收消息recv_data=s.recv(1024)print(str(recv_data,encoding='utf8'))#挂电话 s.close()
源码精简：
import socket
import threading
import selectdef process(request, client_address):print request,client_addressconn = requestconn.sendall('欢迎致电 10086，请输入1xxx,0转人工服务.')flag = Truewhile flag:data = conn.recv(1024)if data == 'exit':flag = Falseelif data == '0':conn.sendall('通过可能会被录音.balabala一大推')else:conn.sendall('请重新输入.')sk = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sk.bind(('127.0.0.1',8002))
sk.listen(5)while True:r, w, e = select.select([sk,],[],[],1)print 'looping'if sk in r:print 'get request'request, client_address = sk.accept()t = threading.Thread(target=process, args=(request, client_address))t.daemon = Falset.start()sk.close()
如精简代码可以看出，SocketServer的ThreadingTCPServer之所以可以同时处理请求得益于 select 和 Threading 两个东西，其实本质上就是在服务器端为每一个客户端创建一个线程，当前线程用来处理对应客户端的请求，所以，可以支持同时n个客户端链接（长连接）。
ForkingTCPServer
ForkingTCPServer和ThreadingTCPServer的使用和执行流程基本一致，只不过在内部分别为请求者建立 “线程” 和 “进程”。
基本使用：
ForkingTCPServer只是将 ThreadingTCPServer 实例中的代码：
server = SocketServer.ThreadingTCPServer(('127.0.0.1',8009),MyRequestHandler)
变更为：
server = SocketServer.ForkingTCPServer(('127.0.0.1',8009),MyRequestHandler)
SocketServer的ThreadingTCPServer之所以可以同时处理请求得益于 select 和 os.fork 两个东西，其实本质上就是在服务器端为每一个客户端创建一个进程，当前新创建的进程用来处理对应客户端的请求，所以，可以支持同时n个客户端链接（长连接）。
源码剖析参考 ThreadingTCPServer
参考：
socket详解：http://www.cnblogs.com/wupeiqi/articles/5040823.html
TCP和UDP协议的区别：http://jingyan.baidu.com/article/6dad5075df3452a123e36ecb.html
