By staying on our site, you agree to the use of cookies.
More detailed

The Digex Co. website uses Google Analytics and Yandex.Metrica services, which allows us to build the content of pages in accordance with your interests and requests, thereby improving the quality of the content offered to you. The cookies we use allow us to recognize those who visit our site repeatedly. These files also serve to track the traffic patterns of visitors on the site, which helps to improve its usability.
You can learn more about the types and files of cookies that we use in our cookies policy.

Digex Data Fabric

Background:

The lack of data engineering-based analytics for making real-time management decisions and improving business processes is a significant challenge.
There are multiple data sources and databases that need to be synchronized into a single dataset.
Usage of popular storage systems such as MS SQL, Oracle, PostgreSQL, MYSQL, MongoDB, and others.
Setting up the storage process and Change Data Capture (CDC) — monitoring changes in data sources and quickly delivering them to all IT systems in use — is necessary.
Introducing Data Governance to identify the most critical data for employee use, assigning them access rights, managing them for smooth business process operation, and protecting them from external threats is crucial.
There are narrow specialized datasets whose storage and processing are impossible in existing standard storage systems.
Transforming data into events and actions is often error-prone, negatively impacting information validity and its application in business processes.
Data collection and preparation problems for BI systems do not allow for timely analysis, affecting business processes.
The need to use open source to address critical IT tasks for businesses is evident.
The lack of Russian comprehensive data management solutions is a significant barrier to import substitution.
Background:
The lack of data engineering-based analytics for making real-time management decisions and improving business processes is a significant challenge.
There are multiple data sources and databases that need to be synchronized into a single dataset.
Usage of popular storage systems such as MS SQL, Oracle, PostgreSQL, MYSQL, MongoDB, and others.
Setting up the storage process and Change Data Capture (CDC) — monitoring changes in data sources and quickly delivering them to all IT systems in use — is necessary.
Introducing Data Governance to identify the most critical data for employee use, assigning them access rights, managing them for smooth business process operation, and protecting them from external threats is crucial.
There are narrow specialized datasets whose storage and processing are impossible in existing standard storage systems.
Transforming data into events and actions is often error-prone, negatively impacting information validity and its application in business processes.
Data collection and preparation problems for BI systems do not allow for timely analysis, affecting business processes.
The need to use open source to address critical IT tasks for businesses is evident.
The lack of Russian comprehensive data management solutions is a significant barrier to import substitution.
The modern organization’s business works with various accounting IT systems that handle customer data. Constant information streams are used in critical business processes daily. The organizations that can work with these data streams by extracting useful metrics, delivering information changes promptly to different departments of the organization, aggregating data from various sources, conducting objective-based analytics, and making real-time decisions gain commercial advantages.

Digex Co.'s Digex Data Fabric (DDF) system performs the entire cycle of tasks related to databases, information array processing, and data streams within a single interface. It collects data from all sources, including internal IT systems, databases, and storage systems.

  • Data collection from all sources: internal IT systems, databases, data warehouses, partner IT systems, and government agencies.
  • Ensuring data accuracy in real-time across the entire business IT infrastructure.
  • Implementation of CDC (Change Data Capture) process and management of Big Data change history.
  • Synchronization of information across different software, databases, APIs, and data centers in different locations.
  • Data migration between databases, data warehouses, and IT systems, and timely backup.
  • Ensuring 100% data security and preventing abuse.
  • Conducting analytics with BI reports based on historical data used in business processes.

Comprehensive approach to Big Data

Data Connectors, ETL, Data Lake / Data Warehouse, Data Processing System, and BI are the five key components of DDF.

  1. Data Connectors are extensions that allow the IT solution to abstract from the specific database level and standardize rules for working with all databases.
  2. ETL (Extract, Transform, Load) is a tool for synchronizing information that combines data from different accounting systems and databases.
  3. Data Lake / Data Warehouse (DWH) is a system that provides continuous data flow to business processes, unifying disparate databases into a single array with rules for each specific information source.
  4. Data Processing System is a tool for data transformation using artificial intelligence, basic algorithms, and machine learning, supporting Multi-Thread Mode for processing data from an infinite number of streams.
  5. Business Intelligence (BI) on Open Source is used for objective data analysis and charting.

Turnkey toolset

DDF can be delivered through four different options

On-Premise servers
Private Cloud
Public Cloud
Hybrid Cloud
On-Premise servers
Private Cloud
Public Cloud
Hybrid Cloud
These connectors allow abstraction from the specific database level and standardize rules for working with all databases. The connectors support the Open Database Connectivity (ODBC) protocol and simultaneous operation with 15 of the most popular storage systems, including MS SQL, Oracle, PostgreSQL, MYSQL, and MongoDB. ETL tools interact with the collected data according to standardized rules, which are part of Data Management Fabric.

Simple scenarios are supported Out of the Box, but implementing and using advanced scenarios requires configuration, knowledge, and additional components. Support for Kafka and other tools is also available. Quick data synchronization is organized, where any of the well-known databases can act as a data source.

Universal data connectors based on Open Source tools

The ETL solution from Digex Co. allows you to build a data processing scheme using visual tools and implement a Pipeline. The ETL module ensures data validity in all sources. It is presented as a Low-Code solution from Russian developers with a user-friendly interface for easy changes to the data processing process.

Previously, most ETL solution users purchased boxed products from vendors that solved all tasks with timely license payments. Many foreign vendors now do not officially provide licenses in Russia. Without official licensing, there is no technical support. The solution stops working when the key has a temporary character (SAAS, temporary key). We provide a direct analogue of foreign ETL tools without such licensing and Vendor Lock problems.

ETL solution without Vendor Lock

Data Warehouse is a storage for various data that has already been sorted and transformed. This approach makes it easier and faster to use data for decision-making. Unlike Data Warehouse, the Data Lake stores scattered data for analytics and requires refinement when adding new indicators. In its projects, Digex Co. combines the use of storage and data lake. Storage is effective in preparing strict reporting (financial, management, etc.), and the data lake is used for research.

Data Warehouse allows working with specialized information, the storage and processing of which is impossible in existing standard systems. The storage can provide aggregated data in a specific thematic and temporal section and in the form of a so-called data mart. A data mart can be used by business development departments in a company for developing sales and marketing strategies, and analyzing the audience. Production departments use the tool to analyze performance and improve the production process.

Reliable Data Lake and Data Warehouse storage

The Data Processing module collects and transforms data using sophisticated algorithms, turning it into useful information for businesses. Raw data is converted into more convenient formats (such as graphs, documents, etc.) and given the form and context necessary for interpretation and use by employees within the organization’s current processes.

First, data is extracted from available sources, including data lakes and data warehouses. After data collection, it undergoes cleaning and error checking. Incorrect, redundant, and incomplete data is corrected and eliminated. Valid, cleaned information is then entered into the destination (IT system, data warehouse, etc.) and translated into understandable language.

At each stage of the Data Processing module, data processing is performed using high-performance computing (HPC). The system supports Multi-Thread Mode for processing data from an infinite number of threads. Digex Co.'s solution utilizes artificial intelligence and machine learning algorithms (AI/ML), applying popular machine learning methods such as OpenCV, TensorFlow, and others.

High-performance AI/ML-based Data Processing

Business Intelligence (BI) uses the obtained data to analyze business metrics. Convenient graphs and diagrams can be used to create performance indicators, identify market trends, and improve virtually all aspects of the organization’s work.

Digex Co. applies its own BI class development based on Open Source in its projects. This solution is popular in projects for large corporations and solves import substitution tasks. Our engineers have extensive experience in painless migration from well-known foreign IT products from other vendors, such as Tableau, Microsoft Power BI, QlikView, TIBCO Spotfire, and others. Data integrity and user convenience are our top priorities in the implementation of domestic BI solutions.

Data visualization for real-time management

Examples of industry application

Banks and finance
  • Systematizing bank partners' data to optimize routine processes, speed up information retrieval from partners, and eliminate errors in communication with clients.
  • Maintaining blacklists of clients to minimize risks and prevent client fraud, automate work with problematic clients and non-payers.
  • Unified storage for data migration to new applications, information exchange with counterparties, and management of the bank’s information assets.
Healthcare and pharmaceuticals
  • Replacing foreign CDC backup systems for storing and monitoring data changes, such as Keboola, Oracle GoldenGate, Qlik Replicate, IBM InfoSphere, Change Data Capture, Fivetran (HVR), Hevo Data, and Talend.
  • Integration system with robotic medical equipment to support the entire drug development or laboratory research process.
  • Storage and processing of specialized medical data for use in IT systems for patient records and medical research.
Government Sector
  • Accelerating and improving the quality of public services by synchronizing and monitoring citizen data across government platforms for citizens.
  • Secure collection and timely delivery of financial transaction data from citizens, received from government agencies and departments, including information on traffic fines, social benefits, excise and duty payments, as well as taxes and other fees.
  • Preventing terrorist activities through the collection of public data on citizens and monitoring online publications.
Retail and E-commerce
  • Forecasting commercial processes in a BI system and enabling seamless automation of operational activities in trading enterprises using Big Data and synchronization with existing CRM systems.
  • Optimizing logistics using mathematical models built on the basis of historical data on logistics services and forecasted demand for goods or services.
  • Preparing data for creating a chatbot based on Telegram or WhatsApp, with synchronization of customer data from IT systems for effective communication with buyers.
Telecommunications
  • Standardizing business processes and conducting business analysis based on valid performance indicators by synchronizing data sets from different business directions within a single telecommunications corporation.
  • Improving the quality of customer service and generating additional sales of related services through marketing communications and personalized advertising campaigns.
  • Increasing efficiency of maintenance and repair processes thanks to predictive analysis based on data sets on the functioning of multiple telecommunications equipment.
Banks and finance
  • Systematizing bank partners' data to optimize routine processes, speed up information retrieval from partners, and eliminate errors in communication with clients.
  • Maintaining blacklists of clients to minimize risks and prevent client fraud, automate work with problematic clients and non-payers.
  • Unified storage for data migration to new applications, information exchange with counterparties, and management of the bank’s information assets.
Healthcare and pharmaceuticals
  • Replacing foreign CDC backup systems for storing and monitoring data changes, such as Keboola, Oracle GoldenGate, Qlik Replicate, IBM InfoSphere, Change Data Capture, Fivetran (HVR), Hevo Data, and Talend.
  • Integration system with robotic medical equipment to support the entire drug development or laboratory research process.
  • Storage and processing of specialized medical data for use in IT systems for patient records and medical research.
Government Sector
  • Accelerating and improving the quality of public services by synchronizing and monitoring citizen data across government platforms for citizens.
  • Secure collection and timely delivery of financial transaction data from citizens, received from government agencies and departments, including information on traffic fines, social benefits, excise and duty payments, as well as taxes and other fees.
  • Preventing terrorist activities through the collection of public data on citizens and monitoring online publications.
Retail and E-commerce
  • Forecasting commercial processes in a BI system and enabling seamless automation of operational activities in trading enterprises using Big Data and synchronization with existing CRM systems.
  • Optimizing logistics using mathematical models built on the basis of historical data on logistics services and forecasted demand for goods or services.
  • Preparing data for creating a chatbot based on Telegram or WhatsApp, with synchronization of customer data from IT systems for effective communication with buyers.
Telecommunications
  • Standardizing business processes and conducting business analysis based on valid performance indicators by synchronizing data sets from different business directions within a single telecommunications corporation.
  • Improving the quality of customer service and generating additional sales of related services through marketing communications and personalized advertising campaigns.
  • Increasing efficiency of maintenance and repair processes thanks to predictive analysis based on data sets on the functioning of multiple telecommunications equipment.
Do you need a consultation or willing to work with us?

Have a question?