RPC服务框架探索之Thrift

Posted 2020-07-14 huan & ping

前言
架构服务化后，需要实现一套方便调用各服务的框架，现在开源如日中天，优先会寻找开源实现，如果没有合适自家公司业务的，才会考虑从零开发，尤其是一切以KPI为准绳的公司，谁会跟钱过不去？N个月之前，公司大神就开始调研了，最后选中了Thrift这个RPC服务框架。使用不熟悉的技术，我会感到很恐惧，它就相当于一个黑盒，我对它一无所知，它是如何运转的？出了问题该如何解决？带着一丝不安，查阅了相关技术文档。

 

RPC
很早之前听说过soap,restful api,rpc之类的服务协议，一直都没有机会深入实践，对它们理解的不够深。它们的目的都是提供本地调用远程服务的能力，只是实现方式不同而已。RPC(remote procedure call)意思是远程过程调用，编码时可以把它当作本地方法一样调用，无需关心内部的实现细节，对于调用方很友好很简单。我查阅资料，发现RPC之类的东西很早很早以前就出现了，存在即是合理的，肯定有它的理由。跟本地调用相比有什么优点缺点呢？根据查阅的资料以及自己的理解总结如下：
优势
1 提高系统吞吐能力
2 业务服务解耦
3 更易构建服务分布式集群
4 基础服务重用更方便
劣势
1 因网络开销方法执行时间更长
2 系统更复杂对运维挑战很大
3 排错成本增加
4 数据序列化消耗CPU资源
现在是移动互联时代，数据无时无刻不在产生着，随着时间的推移，随着用户数的增加，随着更多业务的开展，随着功能迭代的频繁，只有业务服务化只有更易构建分布式集群的架构才能应对这些挑战，毕竟单台服务器的能力是有限的，即使是IOE这种高端设备。尤其是现在地下黑产如此猖獗的今天，时不时的遭受到类DDOS攻击，可以方便扩展节点的架构是多么重要，否则会死的很惨。
因此通过RPC协议实现微服务架构是利大于弊的。

 

Thrift
Thrift是RPC服务协议的一种实现，它是由Facebook开发，2007年开源并且2008年成为Apache开源项目。它的实现目标是支持多语言跨平台简单易使用高性能对业务开发透明屏蔽实现细节专注业务开发。大部分语言提供了类库，可以让开发人员只专注在业务接口实现部分。

实现原理
它由传输层，协议层，处理器，服务层所组成，每个部分不互相依赖，职责单一。
传输层：流协议传输数据，支持socket，http，文件等媒介
协议层：数据封包解包，支持binary，compact，json等
处理器：接口代理，请求转发给相应接口处理
服务层：组装传输层，协议层，处理器，提供RPC服务
客户端调用接口-》客户端数据封包-》传输层-》服端解包-》服端处理器-》服端接口处理。处理完成遵循同样的链路响应。

服端网络模型
简单说明，不做深入了解。
1 单进程
2 单进程多线程
3 单进程事件驱动

IDL
IDL(interface description language)接口描述语言，它包含简单数据类型定义，复杂数据类型结构体定义，复杂数据类型列表定义集合定义，异常类型定义，命名空间声明，接口定义规范等，通过这些规范的组合可以定义好任意复杂的接口，定义好之后，使用IDL编译器可以生成指定语言源码，传输层协议层处理器这些代码会自动生成，只需要专注业务接口具体实现即可。具体类型定义参考官方文档吧。

安装
安装很简单，不同的系统官方文档都有说明，不需要完全死板按照步骤执行，相关依赖如果存在可以跳过相应步骤。安装成功之后就可以使用thrift命令编译IDL文件了。

命令
定义好的IDL文件名以.thrift结尾，通过thrift编译。命令参数说明如下：

  1 Usage: thrift [options] file
  2 Options:
  3   -version    Print the compiler version
  4   -o dir      Set the output directory for gen-* packages
  5                (default: current directory)
  6   -out dir    Set the ouput location for generated files.
  7                (no gen-* folder will be created)
  8   -I dir      Add a directory to the list of directories
  9                 searched for include directives
 10   -nowarn     Suppress all compiler warnings (BAD!)
 11   -strict     Strict compiler warnings on
 12   -v[erbose]  Verbose mode
 13   -r[ecurse]  Also generate included files
 14   -debug      Parse debug trace to stdout
 15   --allow-neg-keys  Allow negative field keys (Used to preserve protocol
 16                 compatibility with older .thrift files)
 17   --allow-64bit-consts  Do not print warnings about using 64-bit constants
 18   --gen STR   Generate code with a dynamically-registered generator.
 19                 STR has the form language[:key1=val1[,key2[,key3=val3]]].
 20                 Keys and values are options passed to the generator.
 21                 Many options will not require values.
 22 
 23 Options related to audit operation
 24    --audit OldFile   Old Thrift file to be audited with \'file\'
 25   -Iold dir    Add a directory to the list of directories
 26                 searched for include directives for old thrift file
 27   -Inew dir    Add a directory to the list of directories
 28                 searched for include directives for new thrift file
 29 
 30 Available generators (and options):
 31   as3 (AS3):
 32     bindable:        Add [bindable] metadata to all the struct classes.
 33   c_glib (C, using GLib):
 34   cocoa (Cocoa):
 35     log_unexpected:  Log every time an unexpected field ID or type is encountered.
 36     debug_descriptions:
 37                      Allow use of debugDescription so the app can add description via a cateogory/extension
 38     validate_required:
 39                      Throws exception if any required field is not set.
 40     async_clients:   Generate clients which invoke asynchronously via block syntax.
 41     pods:            Generate imports in Cocopods framework format.
 42     promise_kit:     Generate clients which invoke asynchronously via promises.
 43   cpp (C++):
 44     cob_style:       Generate "Continuation OBject"-style classes.
 45     no_client_completion:
 46                      Omit calls to completion__() in CobClient class.
 47     no_default_operators:
 48                      Omits generation of default operators ==, != and <
 49     templates:       Generate templatized reader/writer methods.
 50     pure_enums:      Generate pure enums instead of wrapper classes.
 51     include_prefix:  Use full include paths in generated files.
 52     moveable_types:  Generate move constructors and assignment operators.
 53   csharp (C#):
 54     async:           Adds Async support using Task.Run.
 55     wcf:             Adds bindings for WCF to generated classes.
 56     serial:          Add serialization support to generated classes.
 57     nullable:        Use nullable types for properties.
 58     hashcode:        Generate a hashcode and equals implementation for classes.
 59     union:           Use new union typing, which includes a static read function for union types.
 60   d (D):
 61   dart (Dart):
 62     library_name:    Optional override for library name.
 63     library_prefix:  Generate code that can be used within an existing library.
 64                      Use a dot-separated string, e.g. "my_parent_lib.src.gen"
 65     pubspec_lib:     Optional override for thrift lib dependency in pubspec.yaml,
 66                      e.g. "thrift: 0.x.x".  Use a pipe delimiter to separate lines,
 67                      e.g. "thrift:|  git:|    url: [e-mail]"
 68   delphi (delphi):
 69     ansistr_binary:  Use AnsiString for binary datatype (default is TBytes).
 70     register_types:  Enable TypeRegistry, allows for creation of struct, union
 71                      and container instances by interface or TypeInfo()
 72     constprefix:     Name TConstants classes after IDL to reduce ambiguities
 73     events:          Enable and use processing events in the generated code.
 74     xmldoc:          Enable XMLDoc comments for Help Insight etc.
 75   erl (Erlang):
 76     legacynames: Output files retain naming conventions of Thrift 0.9.1 and earlier.
 77     maps:        Generate maps instead of dicts.
 78     otp16:       Generate non-namespaced dict and set instead of dict:dict and sets:set.
 79   go (Go):
 80     package_prefix=  Package prefix for generated files.
 81     thrift_import=   Override thrift package import path (default:git.apache.org/thrift.git/lib/go/thrift)
 82     package=         Package name (default: inferred from thrift file name)
 83     ignore_initialisms
 84                      Disable automatic spelling correction of initialisms (e.g. "URL")
 85     read_write_private
 86                      Make read/write methods private, default is public Read/Write
 87   gv (Graphviz):
 88     exceptions:      Whether to draw arrows from functions to exception.
 89   haxe (Haxe):
 90     callbacks        Use onError()/onSuccess() callbacks for service methods (like AS3)
 91     rtti             Enable @:rtti for generated classes and interfaces
 92     buildmacro=my.macros.Class.method(args)
 93                      Add @:build macro calls to generated classes and interfaces
 94   hs (Haskell):
 95   html (HTML):
 96     standalone:      Self-contained mode, includes all CSS in the HTML files.
 97                      Generates no style.css file, but HTML files will be larger.
 98     noescape:        Do not escape html in doc text.
 99   java (Java):
100     beans:           Members will be private, and setter methods will return void.
101     private-members: Members will be private, but setter methods will return \'this\' like usual.
102     nocamel:         Do not use CamelCase field accessors with beans.
103     fullcamel:       Convert underscored_accessor_or_service_names to camelCase.
104     android:         Generated structures are Parcelable.
105     android_legacy:  Do not use java.io.IOException(throwable) (available for Android 2.3 and above).
106     option_type:     Wrap optional fields in an Option type.
107     java5:           Generate Java 1.5 compliant code (includes android_legacy flag).
108     reuse-objects:   Data objects will not be allocated, but existing instances will be used (read and write).
109     sorted_containers:
110                      Use TreeSet/TreeMap instead of HashSet/HashMap as a implementation of set/map.
111     generated_annotations=[undated|suppress]:
112                      undated: suppress the date at @Generated annotations
113                      suppress: suppress @Generated annotations entirely
114   javame (Java ME):
115   js (javascript):
116     jquery:          Generate jQuery compatible code.
117     node:            Generate node.js compatible code.
118     ts:              Generate TypeScript definition files.
119   json (JSON):
120     merge:           Generate output with included files merged
121   lua (Lua):
122     omit_requires:   Suppress generation of require \'somefile\'.
123   ocaml (OCaml):
124   perl (Perl):
125   php (PHP):
126     inlined:         Generate PHP inlined files
127     server:          Generate PHP server stubs
128     oop:             Generate PHP with object oriented subclasses
129     rest:            Generate PHP REST processors
130     nsglobal=NAME:   Set global namespace
131     validate:        Generate PHP validator methods
132     json:            Generate JsonSerializable classes (requires PHP >= 5.4)
133   py (Python):
134     twisted:         Generate Twisted-friendly RPC services.
135     tornado:         Generate code for use with Tornado.
136     no_utf8strings:  Do not Encode/decode strings using utf8 in the generated code. Basically no effect for Python 3.
137     coding=CODING:   Add file encoding declare in generated file.
138     slots:           Generate code using slots for instance members.
139     dynamic:         Generate dynamic code, less code generated but slower.
140     dynbase=CLS      Derive generated classes from class CLS instead of TBase.
141     dynfrozen=CLS    Derive generated immutable classes from class CLS instead of TFrozenBase.
142     dynexc=CLS       Derive generated exceptions from CLS instead of TExceptionBase.
143     dynimport=\'from foo.bar import CLS\'
144                      Add an import line to generated code to find the dynbase class.
145     package_prefix=\'top.package.\'
146                      Package prefix for generated files.
147     old_style:       Deprecated. Generate old-style classes.
148   rb (Ruby):
149     rubygems:        Add a "require \'rubygems\'" line to the top of each generated file.
150     namespaced:      Generate files in idiomatic namespaced directories.
151   st (Smalltalk):
152   swift (Swift):
153     log_unexpected:  Log every time an unexpected field ID or type is encountered.
154     debug_descriptions:
155                      Allow use of debugDescription so the app can add description via a cateogory/extension
156     async_clients:   Generate clients which invoke asynchronously via block syntax.
157     promise_kit:     Generate clients which invoke asynchronously via promises.
158   xml (XML):
159     merge:           Generate output with included files merged
160     no_default_ns:   Omit default xmlns and add idl: prefix to all elements
161     no_namespaces:   Do not add namespace definitions to the XML model
162   xsd (XSD):

View Code

 

实践
平常使用PHP开发，所以使用PHP实践下，对它有更深入的了解，一般服端开发使用编译型的语言，执行效率更高，以实现简单的NoSQL功能为例子(源码里有各种语言的实现例子)。
1 接口定义(nosql.thrift)

 1/*
 3  * thrift简单示例 模仿thrift源码教程
 4  * nosql数据库简单实现
 5  */
 6  
 7 #由wadeyu创建 wadeyu.cnblogs.com
 8 
 9 /**
10  * 命名空间
11  */
12 namespace cpp NoSql
13 namespace java NoSql
14 namespace php NoSql
15 
16 /**
17  * 异常:无效参数
18  */
19 exception InvalidParametorException{
20 }
21 
22 /**
23  * 定义服务
24  */
25 service NoSqlService{
26     /**
27      * 获取key的值
28      */
29     string get(1:string key) throws (1:InvalidParametorException ex),
30     
31     /**
32      * 设置值
33      */
34     bool set_(1:string key, 2:string value) throws (1:InvalidParametorException ex),
35     
36     /**
37      * 自增
38      */
39     i32 incr(1:string key) throws (1:InvalidParametorException ex),
40 }

2 编译(生成服端代码)

 1 [wadeyu@localhost thriftdemo]$ thrift --gen php:server -out ./server ./meta/nosql.thrift 

3 编写服端启动代码

 1 <?php
 2 /*
 3  * php简单服端脚本
 4  */
 5  
 6 #author by wadeyu: wadeyu.cnblogs.com
 7 
 8 define(\'BASE_DIR\',dirname(__FILE__) . \'/\');
 9 define(\'VENDOR_DIR\',BASE_DIR.\'vendor/\');
10 
11 use Server\\NoSqlHandler;
12 use NoSql\\NoSqlServiceProcessor;
13 use Thrift\\Factory\\TBinaryProtocolFactory;
14 use Thrift\\Factory\\TTransportFactory;
15 use Thrift\\Server\\TServerSocket;
16 use Thrift\\Server\\TSimpleServer;
17 
18 $loader = include_once VENDOR_DIR.\'autoload.php\';
19 $loader->addPsr4(\'Server\\\\\',BASE_DIR.\'server/\');
20 
21 include_once BASE_DIR.\'server/NoSql/NoSqlService.php\';
22 include_once BASE_DIR.\'server/NoSql/Types.php\';
23 
24 $serverTransport = new TServerSocket(\'localhost\',9090);
25 $clientTransport = new TTransportFactory;
26 $binaryProtocol  = new TBinaryProtocolFactory;
27 $nosqlProcessor  = new NoSqlServiceProcessor( new NoSqlHandler );
28 $simpleServer    = new TSimpleServer(
29     $nosqlProcessor,
30     $serverTransport,
31     $clientTransport,
32     $clientTransport,
33     $binaryProtocol,
34     $binaryProtocol
35 );
36 echo "start listening:localhost:9090 \\n";
37 $simpleServer->serve();

4 启动服务

[wadeyu@localhost thriftdemo]$ php phpserver.php 
start listening:localhost:9090

5 编译(生成客户端代码)

[wadeyu@localhost thriftdemo]$ thrift --gen php -out ./client ./meta/nosql.thrift

6 客户端接口调用

 1 <?php
 2 /*
 3  * php简单客户端脚本
 4  */
 5  
 6 #author by wadeyu: wadeyu.cnblogs.com
 7 
 8 define(\'BASE_DIR\',dirname(__FILE__) . \'/\');
 9 define(\'VENDOR_DIR\',BASE_DIR.\'vendor/\');
10 
11 use Thrift\\Protocol\\TBinaryProtocol;
12 use Thrift\\Transport\\TSocket;
13 use NoSql\\NoSqlServiceClient;
14 
15 $loader = include_once VENDOR_DIR.\'autoload.php\';
16 
17 include_once BASE_DIR.\'client/NoSql/NoSqlService.php\';
18 include_once BASE_DIR.\'client/NoSql/Types.php\';
19 
20 $transport = new TSocket(\'localhost\',9090);
21 $protocol  = new TBinaryProtocol($transport);
22 $service   = new NoSqlServiceClient($protocol);
23 $startTime = microtime(true);
24 try{
25     $transport->open();
26     $key1 = \'test1\';
27     $ret = $service->get($key1);
28     var_dump($ret);
29     $ret = $service->set_($key1,\'test1\');
30     var_dump($ret);
31     $key2 = \'test2\';
32     $ret = $service->incr($key2);
33     var_dump($ret);
34     $transport->close();
35 }catch(Exception $ex){
36     throw $ex;
37 }
38 var_dump(\'cost:\'.(microtime(true)-$startTime));

1 [wadeyu@localhost thriftdemo]$ php phpclient.php 
2 string(4) "NULL"
3 bool(true)
4 int(1)
5 string(21) "cost:0.22478580474854"

 

示例源码下载
https://github.com/wadeyu/thriftdemo/archive/master.zip

 

后记
纸上得来终觉浅，得知此事要躬行。其实很早就开始写这篇文章了，自己很懒，拖了几个星期才完成，以后要改掉这个毛病了。

 

参考资料
【1】thrift官方文档

http://thrift.apache.org/docs/
【2】thrift安装

http://thrift.apache.org/docs/install/
【3】thrift IDL规范

http://thrift.apache.org/docs/idl

http://thrift.apache.org/docs/types
【4】thrift实现论文

http://thrift.apache.org/static/files/thrift-20070401.pdf