The OptimizeLab Overlay software repository provides high-performance pre-compiled key applications and function libraries for the AArch64 architecture. It is provided as Launchpad personal package archives (PPAs) and built against Ubuntu 18.04 LTS and Ubuntu 20.04 LTS. This guide describes techniques and methods required to participate in the repository development so that it can be quickly started.
Currently, the repository consists of four components:
science. As development continues, we will also add more components based on user feedback and development plans.
You can install repositories as required. However, most of them depend on the
base component, which bears the modification of basic system development tools such as the compilation toolchain. The
database component is mainly used for database management systems such as MySQL and PostgreSQL. The
media is audio and video encoding and decoding software. The
science is targeted at the core software package of scientific computing and is mainly used in scenarios such as scientific computing and high-performance computing (HPC).
Currently, the following sub-repositories are planned: general library sub-repository
common, deep learning software sub-repository
deeplearning, and robot software sub-repository
The number of applications in a repository should not be as large as possible. An increasing number of software packages means the increase of maintenance workload. When the number of software packages reaches a certain value, the overall maintenance quality deteriorates. The OptimizeLab Overlay repository is oriented to performance optimization and aims to provide software versions with high-quality maintenance. Therefore, the software packages in the repository need to be carefully selected for the following purposes:
upgrade pathfor users to safely and smoothly upgrade the Ubuntu LTS version.
During the development, comply with the following principles:
build-essentialand software package dependencies are installed.
common, no dependency exists between sub-repositories.
The overall development and release process consists of seven steps. To make the process as lightweight as possible, only a node description is provided if necessary.
Functions that have been automated are as follows:
The following functions are planned to be automated:
The following functions are not planned for automation:
The maintenance and upgrade policies are closely related to the software. The software package in the software repository needs to continuously track the update of security issues and provide quality update for the software package if possible. 100% of the security and quality issues that have been fixed in the Ubuntu official release need to be identified and fixed in the repository.
The release pipeline consists of two phases: Build and Test. Details are as follows:
In addition to the preceding steps, manual inspection is required before entering the formal repository to ensure that there is no back line feed in the verification of each step and no new issue is introduced.
The Ubuntu system is developed based on Debian and uses the
.deb software package format. Therefore, mastering Debian software packaging is a basic skill for participating in OptimizeLab Overlay. To master the basic knowledge of Debian software packaging, read the following documents:
In addition to the preceding tutorials, you are also advised to read the following documents, which describe Debian's technical specifications, values for free software, and best practices summarized by many developers in the past. It is a must-read document for developers.
You can run the
apt rdepends <package> command to list the reverse dependencies of a software package. For example, to query all software packages that depend on the software package julia, run the following command:
$ apt rdepends julia julia Reverse Depends: Breaks: julia-common (<< 0.4.1-1~) Replaces: libjulia1 (<< 0.5.0~) Breaks: libjulia1 (<< 0.5.0~) Suggests: julia-doc Replaces: julia-common (<< 0.4.1-1~) Recommends: julia-common Recommends: science-mathematics Suggests: elpa-ess Recommends: science-numericalcomputation
API is short for application programming interface, and ABI is short for application binary interface. API compatibility indicates that a code segment can invoke a function library of any version for compilation to obtain an available program when the API remains unchanged. ABI compatibility indicates that a code segment can dynamically invoke the binary function library files (such as .so and .dll) of other versions that are compatible with the ABI without recompilation when the binary file is compiled using any compatible version.
Typically, an application program ABI includes the following aspects:
On the Linux platform, except for bytecode applications, programs (such as target file format and function call procedure) are compatible by default under most conditions, mainly considering the compatibility of function data types, parameter passing method, and returns in the symbol table of the program.
You can use the
objdump tool, which is stored in the
binutils software package. In this example, the symbol table of libzstd.so.1.3.8 is analyzed. To clearly see symbols provided by the function library, the part with the GLIBC tag is excluded and only the first 30 lines are used to keep concise.
$ objdump -T libzstd.so.1.3.8 | grep -v GLIBC | head -30 libzstd.so.1.3.8: file format elf64-x86-64 DYNAMIC SYMBOL TABLE: 0000000000000000 w D *UND* 0000000000000000 _ITM_deregisterTMCloneTable 0000000000000000 w D *UND* 0000000000000000 __gmon_start__ 0000000000000000 w D *UND* 0000000000000000 _ITM_registerTMCloneTable 000000000006c990 g DF .text 0000000000000005 Base ZBUFF_decompressInit 000000000006c740 g DF .text 000000000000000a Base ZBUFF_isError 000000000000d890 g DF .text 00000000000001f8 Base ZSTD_CCtxParam_getParameter 00000000000670d0 g DF .text 000000000000006e Base ZSTD_DCtx_setMaxWindowSize 000000000000db30 g DF .text 0000000000000031 Base ZSTD_CCtx_refCDict 0000000000067090 g DF .text 000000000000003d Base ZSTD_dParam_getBounds 0000000000013850 g DF .text 000000000000007f Base ZSTD_compressStream2_simpleArgs 0000000000065860 g DF .text 0000000000000007 Base ZSTD_getFrameHeader 0000000000065550 g DF .text 000000000000026e Base ZSTD_getFrameHeader_advanced 0000000000012a90 g DF .text 000000000000013b Base ZSTD_CCtx_loadDictionary_advanced 0000000000077be0 g DF .text 000000000000000a Base ZDICT_isError 000000000005bcb0 g DF .text 0000000000000025 Base ZSTDMT_endStream 00000000000113a0 g DF .text 0000000000000006 Base ZSTD_CStreamInSize 0000000000010cf0 g DF .text 000000000000007b Base ZSTD_freeCDict 0000000000013600 g DF .text 0000000000000219 Base ZSTD_compressStream2 0000000000013be0 g DF .text 000000000000006e Base ZSTD_estimateCDictSize 000000000000dbb0 g DF .text 000000000000000a Base ZSTD_CCtx_refPrefix 0000000000010fe0 g DF .text 000000000000011c Base ZSTD_initStaticCDict 00000000000672a0 g DF .text 000000000000001d Base ZSTD_decodingBufferSize_min 000000000000dbc0 g DF .text 0000000000000064 Base ZSTD_CCtx_reset 00000000000118c0 g DF .text 0000000000000006 Base ZSTD_minCLevel 0000000000067370 g DF .text 0000000000000cec Base ZSTD_decompressStream 000000000006c770 g DF .text 0000000000000005 Base ZBUFF_createCCtx ......
The compiler encodes more information in the symbol table of the C++ program, which affects manual analysis. In this case, you can run the
c++filt command to decode the information. For example, you can parse libboost system.so.1.67.0, use the first 30 lines to exclude the impact of the GLIBC and C++ standard libraries.
$ objdump -T libboost_system.so.1.67.0 | grep -E -v "(GLIBC|CXX)" | head -30 | c++filt libboost_system.so.1.67.0: file format elf64-x86-64 DYNAMIC SYMBOL TABLE: 0000000000000000 w D *UND* 0000000000000000 _ITM_deregisterTMCloneTable 0000000000000000 w D *UND* 0000000000000000 __gmon_start__ 0000000000000000 w D *UND* 0000000000000000 _ITM_registerTMCloneTable 0000000000002430 g DF .text 00000000000000af Base boost::system::system_category() 0000000000002cd0 g DF .text 000000000000004d Base boost::system::detail::system_error_category::message[abi:cxx11](int) const 0000000000002e70 w DF .text 0000000000000052 Base boost::system::detail::system_error_category::~system_error_category() 0000000000005b28 w DO .data.rel.ro 0000000000000018 Base typeinfo for boost::system::error_category::std_category 0000000000005b18 w DO .data.rel.ro 0000000000000010 Base typeinfo for boost::noncopyable_::noncopyable 0000000000002dc0 w DF .text 0000000000000019 Base boost::system::error_category::std_category::default_error_condition(int) const 0000000000002590 g DF .text 0000000000000737 Base boost::system::detail::system_error_category::default_error_condition(int) const 0000000000005b70 w DO .data.rel.ro 0000000000000018 Base typeinfo for boost::system::detail::system_error_category 0000000000004260 w DO .rodata 0000000000000024 Base typeinfo name for boost::noncopyable_::noncopyable 0000000000002d20 w DF .text 000000000000000a Base boost::system::error_category::std_category::name() const 0000000000002de0 w DF .text 0000000000000013 Base boost::system::error_category::std_category::~std_category() 0000000000002ed0 w DF .text 0000000000000038 Base boost::system::detail::generic_error_category::~generic_error_category() 0000000000002f70 w DF .text 000000000000016a Base boost::system::error_category::std_category::equivalent(std::error_code const&, int) const 00000000000030e0 w DF .text 0000000000000179 Base boost::system::error_category::std_category::equivalent(int, std::error_condition const&) const 0000000000002210 g DF .text 000000000000021a Base boost::system::detail::generic_error_category::message[abi:cxx11](int) const 0000000000002e00 w DF .text 0000000000000025 Base boost::system::error_category::std_category::~std_category() 0000000000002d70 w DF .text 0000000000000006 Base boost::system::error_category::default_error_condition(int) const 00000000000024e0 g DF .text 00000000000000af Base boost::system::generic_category() 0000000000002e30 w DF .text 0000000000000038 Base boost::system::detail::system_error_category::~system_error_category() 00000000000021f0 g DF .text 0000000000000008 Base boost::system::detail::generic_error_category::name() const 0000000000002200 g DF .text 0000000000000008 Base boost::system::detail::system_error_category::name() const 0000000000005b40 w DO .data.rel.ro 0000000000000018 Base typeinfo for boost::system::error_category 0000000000002d80 w DF .text 000000000000001a Base boost::system::error_category::equivalent(int, boost::system::error_condition const&) const ......
In some cases, different encode results are generated when the same symbol table is decoded. The possible causes are that the compiler versions are different or different C++ specification versions are used. In this case, the symbol tables are different and need to be distinguished.
You can commit the patch as an issue attachment. Currently, all supported software packages are planned to be managed by Git. In this case, you can commit the patch to a pull request.
For more information about performance optimization, see OptimizeLab Documents.
Do not connect to the Internet during software compilation to ensure that the compilation result is deterministic. Technical measures have been taken on the formal compilation platform to prevent network connection. During local development and test, check whether the network is connected to ensure the consistency of compilation results.
autopkgtest is a test case design based on the DEP-8 standard defined by the Debian community. Different from the unit test provided by software, autopkgtest is mainly used to test the integration result after the release is integrated, the automatic CI test system is allowed to perform automatic cross-combination tests on software of different versions, so as to detect an impact that may be caused to other software when the software is updated. In the Debian project, the debci platform is used to execute all actions.
The repository itself is distributed under the Apache-2 license. For details, see the ``LICENSE` file. All software in the repository retains its original license agreement.
If you have any questions or requests, please commit an issue to us.