Pants includes OS X specific Python wheels -

tldr: pants fetches os x specific wheels bc i'm developing on mac. how can avoid this, or specify deploy ubuntu?

full story:

trying package python application pants. going great far, ran problem i've been stuck @ while. i'm developing on macbook deploying ec2 ubuntu.

here's i've done far:

1. created virtualenv.
2. added build files applications, suggested 3rd party pattern third party packages.
3. ran ./pants run.py backend:admin_server runs fine , generated dist/admin_server.pex
4. scp .pex onto fresh ec2 ubuntu box.

however when run application there, get:

failed execute pex file, missing compatible dependencies for:     mysql-python     pycrypto 

the problem seems pants takes os x specific wheels these 2:

pex: - mysql_python-1.2.5-cp27-none-macosx_10_11_intel.whl pex: - pycrypto-2.6.1-cp27-none-macosx_10_11_intel.whl

how can avoid that, or specify os should run on?

here's full output:

ubuntu@ip-***:~$ export pex_verbose=1 ubuntu@ip-***:~$ python admin_server.pex pex: found site-library: /usr/local/lib/python2.7/dist-packages pex: found site-library: /usr/lib/python2.7/dist-packages pex: tainted path element: /usr/local/lib/python2.7/dist-packages pex: tainted path element: /usr/lib/python2.7/dist-packages pex: scrubbing site-packages: /usr/local/lib/python2.7/dist-packages pex: scrubbing site-packages: /usr/lib/python2.7/dist-packages pex: scrubbing user site: /home/ubuntu/.local/lib/python2.7/site-packages pex: failed resolve requirement: mysql-python==1.2.5 pex: failed resolve requirement: pycrypto==2.6.1 pex: unresolved requirements: pex:   - mysql-python pex:   - pycrypto pex: distributions contained within pex: pex:   - six-1.10.0-py2.py3-none-any.whl pex:   - protobuf-2.6.1-py2.7.egg pex:   - setuptools-19.5-py2.py3-none-any.whl pex:   - mysql_python-1.2.5-cp27-none-macosx_10_11_intel.whl pex:   - pycrypto-2.6.1-cp27-none-macosx_10_11_intel.whl pex:   - futures-3.0.4-py2-none-any.whl pex:   - webapp2-2.5.2-py2-none-any.whl pex:   - requests-2.9.0-py2.py3-none-any.whl pex:   - jmespath-0.9.0-py2.py3-none-any.whl pex:   - beautifulsoup4-4.4.1-py2-none-any.whl pex:   - python_dateutil-2.4.2-py2.py3-none-any.whl pex:   - boto3-1.2.3-py2.py3-none-any.whl pex:   - webob-1.5.1-py2.py3-none-any.whl pex:   - cssutils-1.0.1-py2-none-any.whl pex:   - webapp2_static-0.1-py2-none-any.whl pex:   - paste-2.0.2-py2-none-any.whl pex:   - docutils-0.12-py2-none-any.whl pex:   - botocore-1.3.22-py2.py3-none-any.whl pex:   - protobuf_to_dict-0.1.0-py2-none-any.whl failed execute pex file, missing compatible dependencies for: mysql-python pycrypto 

ps: make sure didn't include versions of python libraries, pip uninstalled both pycrypto , mysql-python.

one of nice things distributing project pex file can prepare run on multiple platforms. example, 1 pex can run on both linux , mac platforms. many projects, there nothing special other build pex. when project has dependencies on platform specific binary code, need perform steps.

one example of library contains platform specific code psutil library. contains c code compiled shared library when module installed. create pex file contains such dependencies, must first provide pre-built version of library platforms other 1 running pants.

the easiest way pre-build libraries use pip tool build wheels.

this recipe assumes following:

• you want build multi-platform pex run on both linux , mac going pre-build libraries in linux environment, build pex on mac environment.
• your project directory lives under ~/src/cookbook

let’s take simple program references library , create pex it.

#  src/python/ps_example/main.py import psutil  proc in psutil.process_iter():     try:         pinfo = proc.as_dict(attrs=['pid', 'name'])     except psutil.nosuchprocess:         pass     else:         print(pinfo) 

with pants, can define executable defining python_binary target in build file:

# src/python/ps_example/build python_binary(name='ps_example',   source = 'main.py',   dependencies = [     ':psutil',  # defined in requirements.txt   ], )  # defines targets specifications in requirements.txt python_requirements() 

in same directory, list python libraries in requirements.txt file:

# src/python/ps_example/requirements.txt  psutil==3.1.1 

now, to make multi-platform pex, you'll need access linux box create linux version of psutil wheel. copy requirements.txt file linux machine, then, execute pip tool:

linux $ mkdir ~/src/cookbook/wheelhouse linux $ pip wheel -r src/python/multi-platform/requirements.txt  \     --wheel-dir=~/src/cookbook/wheelhouse 

this create platform specific wheel file.

linux $ ls ~/src/cookbook/wheelhouse/ psutil-3.1.1-cp27-none-linux_x86_64.whl 

now, need copy platform specific wheel on machine want build multi-platform pex (in case, mac laptop). if use recipe on regular basis, want configure python respository store pre-built libraries.

we’ll use same build file setup in above, modify python_binary specify platforms= parameter.

# src/python/ps_example/build python_binary(name='ps_example',   source = 'main.py',   dependencies = [     ':psutil',  # defined in requirements.txt   ],   platforms=[     'linux-x86_64',     'macosx-10.7-x86_64',   ], )  # defines targets specifications in requirements.txt python_requirements() 

you need tell pants find pre-built python packages. edit pants.ini , add:

[python-repos] repos: [     "%(buildroot)s/wheelhouse/"   ] 

now, copy file psutil-3.1.1-cp27-none-linux_x86_64.whl on mac workstation , place in directory named wheelhouse/ under root of repo.

once done can build multi-platform pex with

mac $ ./pants binary src/python/py_example 

you can verify libraries both mac , linux included in pex unzipping it:

mac $ unzip -l dist/ps_example.pex | grep psutil     17290  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/description.rst     19671  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/metadata      1340  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/record       103  12-21-15 22:09   ...   .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/description.rst     19671  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/metadata      1338  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/record       109  12-21-15 22:09    ...