markdown Python库/包

Posted 2021-05-06

### Jupyter specific:

* JupyterLab (to replace Jupyter notebooks soon!): https://github.com/jupyterlab/jupyterlab
* ipyvolume: 3D plotting using WebGL on notebooks: https://github.com/maartenbreddels/ipyvolume

### GPU related:

* arrayfire-python: github: https://github.com/arrayfire/arrayfire-python
* CuPy - NumPy-like API accelerated with CUDA (chainer uses this): https://github.com/cupy/cupy
* GunRock: High performance graph primitives on GPU:
	- github: https://github.com/gunrock/gunrock
	- docs: http://gunrock.github.io/gunrock/doc/latest/index.html

### Extensions of core python modules:

- toolz: extends capabilities of itertools & functools: https://github.com/pytoolz/toolz/
- cytoolz: Cython implementation of toolz above: https://github.com/pytoolz/cytoolz/

### Profiling related:

- line_profiler: https://github.com/rkern/line_profiler

### Intel resources:

* Installing Intel Distributions for Python & Intel Performance libraries with Anaconda: https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda
* Build & Install Tensorflow on Intel architecture: https://software.intel.com/en-us/articles/build-and-install-tensorflow-on-intel-architecture
* Intel Optimized TensorFlow wheel now available: https://software.intel.com/en-us/articles/intel-optimized-tensorflow-wheel-now-available

### Distributed Computing:

* (Uber) Horovod: Launching Tensorflow & Keras on distributed computing setting (for example to run Tensorflow/Keras on Apache Spark): https://github.com/uber/horovod
* Pyspark: http://spark.apache.org/docs/2.2.0/api/python/index.html

### Utilities

* chrome-printtopdf

### GraphQL stack:

* Resouces collection: awesome-graphql: https://github.com/chentsulin/awesome-graphql#lib-py

* graphene (the main python GraphQL library): http://graphene-python.org/
	- github: https://github.com/graphql-python/graphene. Also installs:
		- graphql-core: https://github.com/graphql-python/graphql-core
		- graphql-relay: https://github.com/graphql-python/graphql-relay-py (Relay compliant graphQL server setup)
		- promise: https://github.com/syrusakbary/promise
		- RxPy (rx): https://github.com/ReactiveX/RxPY
* graphql-ws: Websocket implementation (works with aiohttp, django-channels, asyncio & gevent):
	- github: https://github.com/graphql-python/graphql-ws
* graphiql: (also packaged in graphene-django)
	- github: https://github.com/graphql/graphiql
* Python graphql clients:
	- GQL - Python graphql client (plays well with graphene): https://github.com/graphql-python/gql
	- graphqlclient: https://pypi.python.org/pypi/graphqlclient/0.2.0
* Web-Framework integration:
	- Django (including Django Rest Framework):
		- graphene-django: https://github.com/graphql-python/graphene-django
		- django-graphql-jwt: JSON web token authentication for graphql: https://github.com/flavors/django-graphql-jwt
	- aiohttp: 
		- aiohttp-graphql: https://github.com/graphql-python/aiohttp-graphql
* ORM integration:
	- graphene-sqlalchemy: https://github.com/graphql-python/graphene-sqlalchemy
	- graphene-peewee-async: https://pypi.python.org/pypi/graphene-peewee-async
* Database integration:
	- Postgres
		- PostGraphile (previously PostGraphQL): https://www.graphile.org/postgraphile/
		- PostGraphile on github: https://github.com/graphile/postgraphile

### Database related:

#### ORMs

* sqlalchemy: https://www.sqlalchemy.org/
* graphene-sqlalchemy: adds GraphQL support to sqlalchemy: https://github.com/graphql-python/graphene-sqlalchemy/
(the 2 below do not have GraphQL support at the moment, work with peewee in progress)
* peewee ORM: http://docs.peewee-orm.com/en/latest/
* Storm ORM (Canonical) (plays well with webpy): 

#### Database interfaces:

* asyncpg: A fast Postgres client library for python/asyncio: https://github.com/magicstack/asyncpg
* sqlbuilder: build sqlqueries in Python: http://sqlbuilder.readthedocs.io/en/latest/

### Web/Data scraping, mining & analytics

* pattern: https://github.com/clips/pattern (depends on beautifulsoup4)
* requests: http://docs.python-requests.org/en/master/
* scrapy: https://scrapy.org/
* nltk: https://www.nltk.org/

### ML/DL related

* myia (MILA - Theano successor): https://github.com/mila-udem/myia
* skorch - scikit-learn compatible NN library (wraps pytorch): https://github.com/dnouri/skorch
* chainer - CuPy based NN framework for DL: https://github.com/chainer/chainer
* TensorLayer - DL & RL modules based on TensorFlow - http://tensorlayer.readthedocs.io/en/latest/

### Probabilistic ML/DL

* probtorch: https://github.com/probtorch/probtorch
* edward: http://edwardlib.org/
* PyMC3: http://docs.pymc.io/
* uber/pyro: https://github.com/uber/pyro
* Pomegranate (Cython for speed):
	- Repo: https://github.com/jmschrei/pomegranate
	- Paper: https://arxiv.org/pdf/1711.00137.pdf
	- Article: https://www.techleer.com/articles/458-pomegranate-v090-now-released-flexible-probabilistic-modelling-in-python/
* pgmpy (has NetworkX interface to visualise PGMs): https://github.com/pgmpy/pgmpy#installation
* pgmpy_viz: Visualising pygmpy graphs in the browser: https://github.com/pgmpy/pgmpy_viz
* libpgm: Implementation of Koller, Friedman book: http://pythonhosted.org/libpgm/
* Lea - discrete prob dists in Python: https://bitbucket.org/piedenis/lea
* Daft - PGM visualization (matplotlib in background) - https://github.com/dfm/daft

### ML/DL pipeline related: 

* tpot: Python Automated Machine Learning tool that optimizes ML learning pipelines using genetic programming: https://github.com/EpistasisLab/tpot

### Finance/ trading/ time-series/ backtesting/ portfolios et al

#### Backetesting related:

* quantdom: https://pypi.python.org/pypi/quantdom/0.1a1
	- github: https://github.com/constverum/Quantdom

#### Time-series etc related:

* Prophet: Time-series analyis: https://facebook.github.io/prophet/
* pyflux: https://github.com/RJT1990/pyflux

#### Technical Analysis related:

* TA-Lib: https://www.ta-lib.org/
* pyti (technical indicators): https://github.com/kylejusticemagnuson/pyti

#### Trading APIs

* IB API (Python, C++, Java): https://www.interactivebrokers.co.uk/en/index.php?f=5041

### Natural Language Processing

* spacy: https://spacy.io/	
* thinc: spacy's ML library for NLP in Python: https://github.com/explosion/thinc

### Data related

#### Storage:

* feather (Wes Mckinney et al) - based on Apache Arrow (http://arrow.apache.org/)
	- https://blog.rstudio.com/2016/03/29/feather/
	- github: https://github.com/wesm/feather
* zarr: classes & functions for working with stuff like ndarrays which can be individually compressed:
	- docs: http://zarr.readthedocs.io/en/stable/tutorial.html#creating-an-array

#### Encoding, json, object serialization

* ujson (ultrajson): Lightning fast data serialization: 
	- article: https://blog.hartleybrody.com/python-serialize/
	- github: https://github.com/esnme/ultrajson
* serpy: Ridiculously fast Object serialization: 
	- readthedocs: http://serpy.readthedocs.io/en/latest/
	- github: https://github.com/clarkduvall/serpy
* ijson: Iterative json parser: https://github.com/isagalaev/ijson
* yajl: event driven json parser: http://lloyd.github.io/yajl/

(msgpack is also a very good json byte encoder)

### Images related:

* pillow: Python PIL version:
* Pillow-SIMD: faster than pillow, drop-in replacement: https://github.com/uploadcare/pillow-simd

### Async related:

* gevent: http://www.gevent.org/ (Oldest of the lot, based on greenlets). Asyncio now natively handles this..
* asyncio (built-in for python since 3.5):
	- Detailed docs: http://asyncio.readthedocs.io/en/latest/tcp_echo.html
* async_timeout: For timeout context manager with asyncio
	- pypi: https://pypi.python.org/pypi/async_timeout
	- github: https://github.com/aio-libs/async-timeout
* uvloop: ultra fast drop-in asyncio event-loop: 
	- github: https://github.com/MagicStack/uvloop
	- blog: https://magic.io/blog/uvloop-blazing-fast-python-networking/
(uvloop excels with httptools, but httptools says it has breakable api... hmmm... pycohttptools?)
* httptools: fast http parsing: https://github.com/MagicStack/httptools
* pycohttpparser: (built on picohttpparser in C): 
	- docs: http://pycohttpparser.readthedocs.io/en/latest/
	- github: https://github.com/Lukasa/pycohttpparser
	- PyPi: https://pypi.python.org/pypi/pycohttpparser/1.1.0
* websockets: https://github.com/aaugustin/websockets/ (built on top of asyncio, which can be sped up with uvloop event loop above)
* aiohttp: Async http client/server for asyncio & Python: 
	- Docs: https://aiohttp.readthedocs.io/en/stable/
	- github: https://github.com/aio-libs/aiohttp
* joblib: lightweight parallel pipelining in Python: https://pythonhosted.org/joblib/

### Tasks, queues, scheduling, async related

* rq: Redis based queuing library: https://github.com/rq/rq
* rq-scheduler: Adds task-scheduling to rq: https://github.com/rq/rq-scheduler
* gevent: coroutine based Python networking library (uses greenlets): http://www.gevent.org/
(asyncio with uvloop & httptools better than gevent now)
* celery (distributed task queue) (can use Redis or RabbitMQ as message-broker): http://docs.celeryproject.org/en/latest/index.html

Summary:
Task queues: RQ (needs Redis as message broker) or Celery (can use both Redis & RabbitMQ as message broker) 
Message broker: Redis or RabbitMQ (as above)
Task scheduler: RQ needs rq-scheduler / Celery: built-in
Database: Redis (even for Celery)

### WebDev related (other that Django & Django REST Framework)

#### Web Frameworks and REST APIs related:

* webpy: http://webpy.org/
* WebPy-GraphQL: Adds GraphQL support to web.py: https://pypi.python.org/pypi/WebPy-GraphQL
* Flask: 
* Tornado: 
	- docs: http://www.tornadoweb.org/en/stable/index.html
	- Integration with other services: http://www.tornadoweb.org/en/stable/integration.html
	- Tornado with asyncio: http://www.tornadoweb.org/en/stable/asyncio.html
	- Tornado asyncio with uvloop: https://github.com/MagicStack/uvloop/issues/35
	- Tornado with aiohttp: https://stackoverflow.com/questions/34759841/using-tornado-with-aiohttp-or-other-asyncio-based-libraries
* Twisted:
* Sanic: A flask like framework, but meant to go much faster using asyncio (uvloop & httptools):
	- github: https://github.com/channelcat/sanic
* Quart: Flask like again, but faster using asyncio:
	- github: https://github.com/pgjones/quart
	- gitlab: https://gitlab.com/pgjones/quart
	- docs: https://pgjones.gitlab.io/quart/
* Japronto (extremely fast based on uvloop & picohttpparser, but experimental):
	- github: https://github.com/squeaky-pl/japronto
	- A million requests per sec in Python: https://medium.freecodecamp.org/million-requests-per-second-with-python-95c137af319
* aiohttp: Async http client/server for asyncio & Python: 
	- Docs: https://aiohttp.readthedocs.io/en/stable/
	- github: https://github.com/aio-libs/aiohttp
	- aiohttp does poorly with json-encoded response etc. (perhaps use other serialization etc.?): https://github.com/aio-libs/aiohttp/issues/530

#### Ancillaries:

* django-cors-headers: https://github.com/ottoyiu/django-cors-headers
* websocket-client (websocket client for python): https://github.com/websocket-client/websocket-client

#### RPC (Remote procedure call frameworks)

* GRPC: Used by Tensorflow Serving 
	- page: https://grpc.io/
	- pypi: https://pypi.python.org/pypi/grpcio/1.10.1rc1
	- Python QuickStart: https://grpc.io/docs/quickstart/python.html
* Apache Thrift: 
	- Tutorials: http://thrift.apache.org/tutorial/
	- The missing guide: http://diwakergupta.github.io/thrift-missing-guide/thrift.pdf

### Graphics and Visualisation related:

* manim (used by 3Blue1Brown for his youtube math/science videos): https://github.com/3b1b/manim

### Python testing related:

* unittes: python built-in
* conttest: pypi: https://pypi.python.org/pypi/conttest
* nose: Extends unittest for easier tests:
	- docs: http://nose.readthedocs.io/en/latest/
	- github: https://github.com/nose-devs/nose
	- pypi: https://pypi.python.org/pypi/nose/1.3.7
* tox: Automation project that includes testing (can also be used with front end CI)
	- docs: https://tox.readthedocs.io/en/latest/
	- pypi: https://pypi.python.org/pypi/tox
	- github: https://github.com/tox-dev/tox