Huawei PSIRT: Technical Analysis Report Regarding Finite State Supply Chain Assessment

• Last Release Date: 2019-07-03

At Huawei, we welcome collaboration with cyber security researchers and independent testing of our products and solutions. We have a long-established Product Security Incident Response Team (PSIRT) that manages the collection, investigation, internal coordination and responsible disclosure of security vulnerability information related to Huawei products. Once a vulnerability has been confirmed, PSIRT promptly conveys the information to the teams responsible for the affected products, and then actively tracks the progress to resolution.

Huawei has built and implemented a multi-tiered end-to-end cyber security evaluation process to ensure that our products are reviewed for potential security issues from product concept, design, development, and right throughout to deployment and maintenance in our customers' networks around the world.

On June 26, 2019, U.S.-based Finite State publicly disclosed the Supply Chain Assessment report about Huawei on its official website. In this report, Finite State describes its use of a static analysis tool for firmware images (binary software packages) to analyze more than 500 Huawei enterprise network products and the comparison analysis between Huawei CE12800, Juniper EX4650, and Arista 7280R, with conclusions that Huawei products have poorer security and potential backdoors.

We were surprised and disappointed by the unconventional approach of Finite State. We cannot determine whether Finite State obtained the software from legitimate channels or guarantee its integrity, nor has Huawei ever received any communication requests from Finite State. They made no contact with Huawei to assist them in their understanding and refused to provide a copy of their analysis before it was published. Sadly, this means what has been published lacks the insight, integrity and accuracy we would normally expect from a professional, serious and capable organization.

Due to the approach Finite State has taken and the weakness of their tools and methodology, the results are at best suspect and at worst just inaccurate. This could have been avoided by collaborating rather than taking a political stance on security.

We are unsure of the objectives of the CEO Matt Wyckhouse and Finite State overall and why they did not select the market leader Cisco for comparison, or indeed why they evaluated old versions of Huawei products and identified issues that had been fixed in new versions.

Whilst Finite State has had many months to undertake their flawed analysis, over the last few days Huawei PSIRT has investigated the issues mentioned in the report immediately after the report was published. We believe that the approach used by Finite State has serious operational and technical defects, the tests lack neutrality, and the report contains material inaccuracies.

1   Finite State's Test Process and Report Development Approach Are Contrary to General Practices of Responsible Security Testing Companies

Responsible disclosure of security issues or vulnerabilities is widely recognized and practiced by the industry. Typically, security research organizations or researchers deliver identified issues and potential vulnerabilities to vendors, and vendors then confirm whether they are defects or vulnerabilities and carry out coordinated handling. Finite State simply used a tool to scan raw binaries and then conducted simple partial reverse analysis of some potential issues to reach their conclusions. Finite State has not used vulnerability exploitation in real-world products to verify the analysis, nor has the analysis been confirmed by Huawei product R&D. Binary vulnerability scanning tools are generally used for auxiliary analysis because their error rate can reach up to over 90%. Thus, Finite State's conclusions are drawn in a hasty manner and are inaccurate. Although Finite State mentions the limitations of the tool it has developed and used, for example, the tool does not support analysis in context, and vulnerabilities are based on file names and version information, Finite State has overestimated the sophistication and accuracy of its tool. As we demonstrate in Appendix, independent analysts do not rank Finite State tools as market leaders in any dimension.

Finite State made a hasty and unprofessional decision to deliver the assessment report to the media and government authorities, without providing it to Huawei beforehand, nor have the issues been confirmed by Huawei. This practice is contrary to best practice or even basic common sense in terms of responsible security organizations in the industry. A fair security technology organization shall remain neutral and express opinions from the perspective of technical security.

Finite State's assessment report repeatedly mentions potential backdoors in Huawei products in an emotional and overstated way. Any security company that claimed it has discovered many backdoors and unfixed serious vulnerabilities by tool-based scanning and without verifying the products or even having any context or knowledge of the products, their architecture and environment, cannot be taken seriously.

2   Assessment Report Gives No Explanation About the Selection of Vendors, Products and Versions for Comparison, and Selective Tests Have Been Conducted

The assessment report does not explain why products of Huawei, Juniper, and Arista were used as test samples but Cisco, another company who holds the largest market share of the global enterprise network. Why weren't Cisco products evaluated? Finite State tested almost all of the hundreds of Huawei enterprise network products, but only one product of Juniper and Arista without disclosing their versions. According to the report, it states that the latest versions of Huawei products are used, however, all versions mentioned in the report are actually old versions. For example, AR1200 V200R007C00SPCc00 released in 2017 was used. However, the updated versions released in 2018 and 2019, such as V200R009 and V200R010, are available on Huawei's technical support website. Moreover, AR3600 V200R007C00SPCb00 released in 2016 was used, but the updated versions released in 2018 and 2019, such as V200R008 and V200R009, are also available on Huawei's technical support website.

We believe selective tests have been conducted, with intentionally selected versions and comparison objects to achieve the "expected" results for Finite State or those that funded this "research".

3   Finite State's Conclusions Are False Through Our Investigation and Analysis

Regarding Finite State's conclusions, Huawei products have backdoors and many vulnerabilities or even serious vulnerabilities left unfixed, Huawei PSIRT and R&D have undertaken a detailed analysis and reached the following conclusions after verification.

3.1  Analysis of Suspected Backdoors

Many suspected backdoor conclusions drawn by Finite State are based on the prerequisite that Huawei is using standard Linux-based authentication. However, this prerequisite is incorrect and thereby the stated conclusions are wrong.

3.1.1  Analysis of Undocumented and Hard-coded Credentials

The report shows that huawei, python, and root accounts are potential privilege escalation backdoors. In fact, the three accounts identified cannot be used for unauthorized privilege escalation. The analysis is as follows:

Huawei AR products use only basic functions of Linux, such as task scheduling. Other functions, namely, user management, remote access control, and TCP/IP protocol stack, are taken over by Huawei Versatile Routing Platform (VRP). This design can better meet application requirements of products. Many telecommunications companies in the industry also use the similar design pattern, as shown in the following figure.

VRP taking over remote user access

The root account is used to start the VRP process. It is used internally and invisible externally. The python and huawei accounts are used by VRP users with the highest privileges to create VMs and install third-party applications. These accounts are invisible externally. The three accounts are protected from being exploited for remote device access and do not compromise system security.

It is true that the sudo configuration and sbin/insmod commands mentioned in the report may be exploited for privilege escalation. Huawei PSIRT has confirmed that this is a known and fixed vulnerability. The device administrator shall be assigned the least privilege to reduce risks. Huawei has eliminated the risk in V300R003C00SPC500 (released in August 2018) by using the program code to implement related management functions.

In addition, the huawei, python, and root accounts are documented in Command Reference (2018)

3.1.2  Analysis of Default Hard-coded Cryptographic Keys

As described in the report, the authorized_keys and similar files are engineering tools used during the development process. The E9000 and CE12800 R&D engineers use SSH of the Linux OS to facilitate debugging, and leave the key files in the firmware.

As illustrated in section 3.1.1"Analysis of Undocumented and Hard-coded Credentials", Huawei datacom products use the basic functions of Linux. Remote access control and TCP/IP protocol stack are taken over by the VRP. In official versions, the debugging function is disabled, and external users cannot access SSH of the Linux OS. Therefore, these key files do not cause any potential unauthorized access. These key files will be deleted in the version to be released in September 2019.

The report shows the presence of an authorized_keys file for the superuser account on the firmware image of SmartAX MA5800, but the SSH code has been deleted from the released versions, and therefore no security risks exist.

3.2  Analysis of Known Vulnerabilities Not Fixed

The report describes the use of outdated components and we agree with this analysis and have already announced substantial upgrades to enhance our products in this regard... However, the presence of outdated components does not necessarily mean the presence of security issues.

The known vulnerability analysis method SCA mentioned in the report is used to assess known vulnerabilities by open-source software name and version number. This method is defective for embedded devices because of the following causes:

(1) Code related to open-source component vulnerabilities is not compiled into the firmware.

(2) For some open-source software, after a vulnerability is detected, the source code patch will be preferentially released to fix the vulnerability. Then a formal fix version is planned. This process may take a long time depending on the open source community approach. To fix the vulnerability as soon as possible, telecom vendors usually incorporate the fixed source code. However, the version number of the open-source software used in the product firmware is still the old version number.

(3) The method of fixing vulnerabilities using binaries is similar to (2). The version number of the open-source software remains unchanged.

(4) The vulnerable code in the open-source component is included in the firmware, but the corresponding functional module is not used.

After analyzing the 10 well-known vulnerabilities reported in AR3600 V200R007C00SPCb00, we find that 6 vulnerabilities cause no impact, 2 are fixed, and 2 are of low risks. The details are as follows:

3.3  Analysis of Conclusion that Huawei Situation Is Getting Worse Drawn on An Increased Number of Publicly Known CVE Vulnerabilities

Finite State concluded on page 23 of the report that the situation of Huawei is getting worse based on an increased number of CVEs, which is unscientific.

It is a basic requirement of ISO/IEC 29147:2018 Vulnerability Disclosure to disclose a vulnerability to customers and notify them of risks and mitigations after fixing the vulnerability. Huawei PSIRT is a dedicated global vulnerability response team which established Huawei's vulnerability response process based on related standards. In 2012, Huawei PSIRT established a public channel (www.huawei.com/psirt) for vulnerability disclosure.

According to the number of vulnerabilities disclosed by the NVD, the Top 5 vendors are Microsoft, Oracle, Apple, IBM, and Google.

Vulnerability trends of Microsoft show that its number of vulnerabilities remains at a certain level. This shows on one hand Microsoft's continuous investment in security and on the other hand Microsoft's responsible disclosure of vulnerabilities. In addition, Microsoft uses the bug bounty program to encourage people to discover vulnerabilities.

Link: https://www.cvedetails.com/vendor/26/Microsoft.html

Cisco also has its own vulnerability disclosure channel. The number of vulnerabilities disclosed by Cisco also remains at a certain level.

Link: https://www.cvedetails.com/vendor/16/Cisco.html

3.4  Secure Coding Practices

3.4.1  Safe Function Analysis

The method used in the report to analyze safe functions has the following problems, which leads to inaccurate results:

1)  The assessment does not cover a large number of safe functions in Huawei products, such as VOS_MemCpy_Safe and VOS_nsprintf_Safe, which causes serious deviation in the security assessment results.

2)  Inaccurate understanding of unsafe functions

(1) The report lists on page 33 some unsafe functions, including puts, memcmp, and asprintf.

(2) As a memory clearing function, memset is used within Huawei to clear the newly applied memory and arrays with a fixed length. It has a very low risk. Even Microsoft that promotes safe functions does not have the corresponding safe function.

(3) Functions such as fopen, access, system, remove, and execl must be used in code to meet service requirements. Using these functions does not necessarily lead to vulnerabilities.

3)  Some functions in the report are regarded as both safe and unsafe.

(1) asprintf is listed as an unsafe function in "Top 20 Most Commonly Used Unsafe Functions" on page 34, but is listed as a safe function later in "Safe and Unsafe Function Collections" on page 36. drv_cvb_memcpy_s_impl is listed as both a safe and unsafe function in "Safe and Unsafe Function Collections" on page 36 of the report.

(2) According to "Top 20 Most Commonly Used Unsafe Functions", the author regards the execl function as unsafe but execlp, execv, execve, execvp, execle, and execvpe as safe, which is incorrect.

3.4.2 Compiler Security Option Analysis

In addition to RELRO, ASLR, DEP, and StackGuard mentioned in the report, at least three other compiler security options are important. In an embedded communications device, enabling compiler security options generally deteriorates product performance, even prevents product functions from running properly in some cases. It demonstrates the lack of maturity and competence of Finite State to comment on the enabling of compiler security options in embedded communications software from the perspective of general software only. Huawei would be happy to teach Finite State the basics of imbedded systems and global telecommunications operations that cover the globe.

Huawei has been carrying out in-depth researches on compiler security options for many years and attaches great importance to security. We will enable compiler security options as much as possible when conditions allow. As far as we know, Huawei leads the communications industry in terms of implementation in this regard.

Appendix: Introduction to the SCA Method of Finite State

Finite State uses the Software Composition Analysis (SCA) method in the assessment report, which is consistent with the industry. Many companies in the industry provide such an analysis service. The report of the research firm Forrester shows their SCA vendor evaluation, in which Finite State is not found. The Forrester Wave™: Software Composition Analysis, Q2 2019 link is :https://reprints.forrester.com/#/assets/2/230/RES146435/reports.

SCA Principles

Currently, the commonly used open-source software and vulnerability analysis technology SCA have two major purposes:

1)  Identify the version and license information of the open-source software used to ensure compliant use.

2)  Search the vulnerability library by open-source software version to obtain all vulnerabilities in the open-source software.

Source: blog of WhiteSource, a leading SCA solution provider according to Forrester

First and foremost, SCA tools generate an inventory report of all open source components in your products, including all direct and transitive dependencies. Taking inventory of open source usage is critical as it is the basis for properly managing your open source usage. After all, how can you secure or ensure compliance of something you do not know you're using?

Once all open source components have been identified, SCA tools provide information on each component. Basic information includes the open source license and whether there's a security vulnerability associated with that component.

The SCA method analyzes the firmware in the following steps:

1)  Extract the complete hash value, partial hash value, function symbol name, file name, etc. of each binary file in the firmware as features. Identify the name and version of the open-source software referenced in the firmware based on these features.

2)  On the basis of open-source software name and version information obtained in step 1, search the vulnerability library and obtain all vulnerabilities in the open-source software.

The SCA result is only an intermediate result and generally needs to be further confirmed with the firmware developer.

Revision History: V1.1 UPDATED Add the link of the Software Composition Analysis from Forrester