Technical Architects Blog
In Memory Database
BUSINESSES NEED TO RESPOND FASTER AND PROCESS MORE
For businesses today, every millisecond counts. With the prevalence of the Internet, many aspects of modern business as well as our personal lives are conducted online. This has translated into an exponential increase in business transactions.
Whether trading systems on Wall street processing streams of market data; online news agencies providing real time news feeds to their subscribers; e-commerce businesses providing a consumer-friendly experience while browsing online inventory; telecommunications systems that need to meet extremely demanding quality of service standards; or, defence monitoring systems that process sensor data in real time—all of these systems need to process large data volumes while being very responsive to end users.
The need to process more data faster has brought new requirements for existing data management systems. These challenges have to be met cost-effectively, and without introducing further complexity into already complex IT departments and data centres. In short, these solutions need to be elegant, simple and readily integrated into existing environments.
Sybase Adaptive Server® Enterprise (ASE) meets these challenges with the introduction of in-memory databases. Sybase’s new ASE in-memory databases bring a truly integrated in-memory data processing capability to ASE.
Sybase in-memory databases will be of immense business value where traditional paradigms of transaction processing break down, and where fast response times are essential to our customers’ competitive edge.
WHY TRADITIONAL PARADIGMS FAIL
The traditional outlook towards management of transactional business data is based on a persistent transaction data processing model: a debit from one account and a credit to another account once acknowledged to an end user must necessarily be reflected in the underlying accounts, even if the underlying computing system suffers an immediate outage at the point of transaction completion. In technical terms, these changes need to be durable. When the system comes up, the debit account as well as the credit account must correctly reflect their state as they existed at the precise moment the transaction was successfully acknowledged to the end user.
This makes perfect sense, especially if you are the owner of the credited account. Transaction processing data management systems are built around providing such guarantees. These OLTP (online transaction processing system) systems are designed around ACID properties: Atomic, Consistent, Isolated and Durable. These properties ensure that business transactions are fully reliable and accurately reflected in the underlying system, even when hundreds of transactions are processed simultaneously and under unpredictable system failure scenarios.
To ensure durability of transactions, database systems have complex algorithms to guarantee that committed transactions indeed make it reliably to disk. This includes pushing log records to disk before acknowledging to the end user that the transaction was complete.
Furthermore, these database systems were built at a time when memory was very expensive and data sizes were always assumed to be far larger than the amount of available memory in a computing system. As a result, today’s sophisticated systems have complex logic for managing data movement between disk and memory, built with the assumptions that data may not always be in memory. The worst case is always accounted for.
However, many data management scenarios and business-use cases need not adhere to such constraints. The old paradigms are a hindrance in these cases. Response times and throughput suffers, and businesses lose their competitive edge.
USE CASES WHERE OLD APPROACHES FAIL
Capital Markets
The financial services industry is all about speeds and feeds. Every millisecond counts and can add to the bottom line.
Processing Market Data
Trading systems need to look at large volumes of trade data and then process it as it arrives over the wire. This may be as part of risk management systems, automated trading systems, live trader desk monitoring and so on. These systems need to be able to cache large data volumes as they arrive; rapidly process them into meaningful and actionable information; and, service user requests against this data in near-real time.
This market data is not stored in the operational database, but is usually stored in non-transactional historical data stores. The durability of this market data is not important for these live trading systems. However, being able to process this data rapidly, manage derived data sets, and rapidly responding to queries, is.
Trading System
Trading systems have to comply with many regulations. Client trades are checked against various compliance data and rules before these are allowed to go through the system. Furthermore, reference data (that is either not changing or changes infrequently) is read repeatedly when processing trades.
Compliance and reference data often need to be pulled from different systems for locality of reference and lookup speeds. This data is read-mostly. Trading systems cache this data in a global cache and are primarily concerned with look-up speeds not durability. How fast this data is queried determines how quickly a trade is executed.
Telecommunications
Telecommunication providers are paranoid about quality of service issues and do everything possible to avoid latency in their systems. Service providers need to provide many operations in near real time. These may include account authentication, account management for pre-paid calling services, web accessible records of calls made, minutes charged and funds remaining, real time checks for discounts and special offers and so on. Many of these operations take place even before a call goes through.
Often service providers are less concerned about losing a few transactions and more concerned that end users do not experience noticeable delays in their call connections. In order to meet such high levels of service, lots of data, such as subscriber information and currently available special offers, is cached. Even though this data might not be changing, in some cases providers are willing to lose some data to ensure low response times.
E-Commerce Systems
On-line retailers provide services and products over the web. Consumers browse and search online-inventory, compare products and add and drop items from their shopping carts often discarding the shopping cart altogether. Very small portions of this online activity actually translate into an actual business transaction.
Retailers also monitor vendor behaviour, site surfing patterns and manage real-time marketing and product offers. The majority of this activity does not change any underlying data and the underlying changes need not persist across a back-end system outage. The inventory does not change frequently. Instead, vendors are most concerned with handling large numbers of customers without noticeable degradation in user experience.
Real Time Monitoring
Many monitoring systems need to process incoming data to provide near-real time dashboard views for business executives. Data could come from a variety of sources such as RFIDs in a warehouse, sensor data from defence systems, surveillance data for intelligence networks, telecommunication networks and so on.
This data is generally transient and need not be durably stored in a transactional system. However, it does need to be processed for real-time decision making.
IN-MEMORY DATABASES MEET THE CHALLENGE
In the above use cases, the most important data challenges are to:
- Consume large data volumes and potentially discard data once it has been processed
- Respond to user queries on this data
To meet these data management challenges, a system needs to be able to insert, delete and update data, while maintaining very high throughout. At the same time, response times need to be very low. Data durability for such scenarios is not the priority. High throughput and low response times are.
In-memory databases meet these challenges. In-memory databases have become a viable technology with the reduction in memory prices and the shift towards 64-bit computing systems. Having 64GB and above memory configurations in commodity class servers is already common. This trend will continue as memory prices drop further.
In-memory databases keep all the data in-memory and do not write to disk. Data can be moved to disk if desired by the user but this is not a regular part of processing a transaction. By keeping the data in memory, these databases always ensure that reads and writes to disk never happen thus providing fast data updates. Since data is always known to be in memory a lot of complex logic in the server can be bypassed and data access paths can be optimized for fast access.
Whereas traditional disk databases enforce durability even if not needed by the application, in-memory databases remove this constraint. The result is that in-memory databases give superior performance when compared to a traditional disk oriented database. In-memory databases are fully transactional in all other aspects.
Sybase ASE has introduced in-memory databases as a fully integrated technology in ASE. ASE now provides both traditional disk databases that guarantee durability, as well as in-memory databases that remove this constraint.
Back to Blog Homepage
