Storage is getting faster, more efficient, and – coupled with data analysis – more useful. Based on applications, Huawei's intelligent storage platform offers data analysis, processing, and storage for different big data scenarios.
Pre-buffered movie recommendations, instant-searches for photos, quick route planning for complicated trips – sounds good, right? Well, we can in fact do all of this today.
The two enablers are big data storage and intelligent analysis. Massive amounts of data are needed to produce results from analysis that are relevant to users, and that data needs to be stored.
The massive amount of data produced by IoT and mobile Internet is increasing by 50 percent year on year, with Gartner predicting a total global data volume of 35 ZB by 2020 – equivalent to 8 billion 4TB hard disks. So, sourcing big data is evidently not the problem – the real issue is how to store and intelligently analyze it.
Big storage for big data
The three main types of big data storage applications are mass data storage, fast data read, and intelligent data analysis.
Mass data storage takes care of traditional features like storage capacity, efficiency, and security.
Fast data read manages, schedules, and processes data streams to quickly handle data, and is an area where technology advances are fast because the Internet and IoT need speed.
Intelligent data analysis provides customizable services under a very rapid update cycle for tools, including machine-learning algorithm libraries and software from companies like Hadoop and Spark.
Instead of integrating these three applications to deeply integrate storage and intelligent analysis, most vendors’ products miss the mark by only trying to improve the performance of big data storage technology.
Big data-driven companies like LinkedIn and Facebook focus more on high throughput and constructing efficient data analysis platforms on existing hardware, and overlook traditional big data storage.
This separation of technologies causes problems for customers that need big data analysis. First, data analysts don’t have efficient data storage technology, and moving data from traditional storage clouds to analysis clouds is inefficient. Second, storage technology researchers often lose sight of data application requirements and, by only focusing on access speed and storage capacity, produce technology that’s no good for data analysis teams. In effect, this divides big data storage and analysis into two separate teams and platforms.
Today's big data storage needs to go beyond massive storage capacity to integrate data storage, fast data reading, and intelligent analysis.
Let’s integrate
Huawei’s storage platform integrates mass storage, fast reading, and data analysis capabilities.
Just a 1 percent efficiency increase in the Industrial Internet by using big data analysis can generate up to US$1 trillion worth of value. For example, 80 percent of Facebook's revenue comes from mining analysis on social graphs, while EverString reveals hidden business trends and opportunities for users.
The public are generating huge amounts of data on IoT and social media in areas like entertainment, financial services, and healthcare. Rapid and precise analysis would be a great predictor of flashpoint technologies and events in particular sectors, which could in turn help shape policies.
Key technologies
Combining intelligent analysis and storage will integrate stored data and the information to be mined and extracted.
Huawei's efficient intelligent storage platform adopts the following key technologies:
Integrated intelligent analysis provides popular storage capabilities like HDFS and databases alongside data analysis capabilities such as Kafka and Spark, processing frameworks including Samza and Spark Streaming, and data analysis algorithm libraries such as machine learning and image processing.
Integrated storage system architecture integrates multiple storage system architectures at the resource and system level, and transfers data between them. The resource level includes unified management and name space, heterogeneous resource management for allocating cluster resources, and isolation guarantees. The system level supports multidimensional semantic data, and flexibly uses different-format storage systems, including KV, relationship, graph, file, and object. It also supports active data flow, and can convert different storage systems, including NAS, OSD, SAN, and Cloud.
Data access semantics and service capabilities includes multiple storage methods such as file, block, object, and cloud storage; content searches and document lifecycle management services; hierarchical SLO for performance, availability, and reliability; and methods for visualizing data value.
Application requirements
Huawei's intelligent storage platform meets the requirements of different applications with its application-driven Data Aware Engine (DAE).
In traditional storage systems, an application generates heterogeneous multi-source data; for example, medical records are processed into different formats and stored in different systems such as a picture database, key value system, relational database, and filing system. However, system barriers complicate data integration and analysis between different systems, so data can’t be extracted easily. Data-aware storage engines can solve this problem with the following functions:
Cross Storage System Access includes a unified storage engine that formalizes data when it’s written, chooses the best bottom-layer storage systems for different data types, and returns the most appropriate data during queries and processing.
Smart Layout Tier & Transfer supports longitudinal or transverse data flow between hot, warm, and cold layers to facilitate data processing.
Unstructured Data Analysis carries out intelligent analysis decisions and deeper mining.
Integrated Access History Logging records the access behaviour of multi-dimensional and multi-source data, so intelligent analysis decision modules can perform deep analysis, mining, and stream processing line analysis.
Huawei's application-driven intelligent storage platform meets the data analysis, processing, and storage requirements of diverse application content by integrating different types of big data analysis tools to provide different services. Using applications as enablers, Huawei delivers customized, efficient, and more secure storage architectures and models for different applications at lower cost.
By integrating intelligent analysis and storage, Huawei’s intelligent storage technology can transform economic activities and sectors such as transportation and communications – the potential to generate new business and increase market value is huge.
