Bidding Starts: Wednesday September 25, 2024

Bidding Ends: Sunday November 10, 2024

Summary:

RISE is seeking help to expand the development, testing, deployment, and maintenance of pre-built Python binary packages for the riscv64 architecture. Currently, many critical Python packages like NumPy, pandas, scikit-learn and PyTorch, rely on C/C++ code and must be compiled before use. While pre-built binaries exist for other architectures, there is limited support for riscv64, making Python less accessible for AI/ML and data analytics on this platform.

The project includes building, testing, deploying, and maintaining Linux riscv64 versions of key Python packages on RISE’s GitLab. Expanding beyond the current 18 packages will make Python more usable on riscv64 with additional resources.

Structure: 

RP011 (Python Packaging Support) requires an estimated 2-3 days of developer work per week over 12 months, with the workload fluctuating between 0 to 5 days depending on project needs. The developer will collaborate with a RISE project lead to stay aligned with Python packaging requirements. The project lead will review and approve both the work completed and the time spent. RISE lead will oversee the developer’s work. The total project budget is up to $200k over the 12-month period.

Developer Expectations: 

Your work will be essential in ensuring efficiently providing riscv64 Python support.

This project involves several key work streams aimed at building, maintaining, and distributing Python packages for riscv64. Here's what you'll need to do:

1. Maintain Patched Projects (manylinux, cibuildwheel)

   • Ensure the patches we use for manylinux and cibuildwheel are up to date.

   • Rebase patches as needed when new versions of packages require updated versions of manylinux for successful builds.

   • Identify and add any missing infrastructure necessary for riscv64 package support.

2. Build, Test, and Distribute Updated Packages

   • For simpler packages (e.g., ninja), update the fork and trigger the build process.

   • For complex packages (e.g., NumPy, SciPy) with extensive dependencies, the process will be more involved. Be prepared for significant engineering effort, as seen in the case of NumPy 2.0, which required a full week to make available, even though a prior version had already been distributed.

3. Build, Test, and Distribute New Packages

   • Create GitLab CI scripts for building new packages not currently available on RISE’s GitLab.

   • Patch projects as necessary to ensure builds succeed and all tests pass on riscv64.

   • For each new package, ensure smooth integration into the GitLab pipeline.

4. Upstream Patches

   • Where feasible, submit patches upstream to enable future compatibility.

   • Ensure that projects build and run tests cleanly on riscv64, aiming for long-term maintainability.

5. Open Source License Compliance

   • Take reasonable steps to ensure compliance with applicable open source licenses.

   • Review all components included in a wheel to ensure compliance with licenses applicable to the components, including ensuring all applicable licenses are included in the wheel and that, when required, including pointers to source code.

   • Not all packages on pypl necessarily fully comply with the terms of the applicable licenses. When such non-compliance is identified, open bugs on upstream projects.

6. Rebuild for New Python Versions

   • Rebuild existing packages and the manylinux container whenever new Python versions (e.g., Python 3.13) are released.

The following new packages, infrastructure projects, and their dependencies must be built for riscv64. Please note there might be new packages added. 

• charset_normalizer

• frozenlist

• httptools

• opencv-python

• orjson

• protobuf

• requirements for pydantic

• ray

• regex

• rpds_py

• ruamel.yaml.clib

• safetensors

• sentencepiece

• tiktoken

• tokenizers

• torchaudio

• torchvision

• transformers

• ujson

• uv

• requirements for  uvicorn[standard]

• uvloop

• watchfiles

• websockets

• psutils (Already available but there are some test failures to investigate)

• pytorch (CPU) (some porting work already done)

• scikit-learn 

• aiohttp

• msgspec

• pyzmq

• jiter

• multidict

• yarl

• numba

• llvmlite

• xxhash

Note These packages are already done: 

• cffi 

• cryptography (done)

• fastapi

• lxml (done)

• maturin (done)

• msgpack (done)

• nh3 (done)

• pandas 

Please Submit: 

Are you interested in improving Python's compatibility with the riscv64 architecture? RISE is seeking a Python developer to help expand and maintain key Python packages on RISC-V. 

If you're a Python developer eager to tackle challenges and contribute to meaningful projects, we encourage you to apply and join us in this important work!

As you submit your bid for the project, please ensure you demonstrate the following required skills:

• Experience with Python packaging and a solid understanding of tools used to build Python packages (e.g., manylinux, cibuildwheel, auditwheel).

• Familiarity with writing CI/CD scripts for GitLab or GitHub.

• Proficiency in Git for version control.

• Experience with Docker for containerization.

• Strong programming skills in Python and C/C++.

• Knowledge of Rust is desirable and would be an added advantage.

These qualifications will be key in evaluating your suitability for the project. 

Bidding Closed