01 August 2014

The Big Data Blog

Amir Halfon - MarkLogic

9 | posts 47,680 | views 1 | comments

Banking Architecture

A community for discussing the latest happenings in banking IT. Credit Crunch impacting Risk Systems overall, revamp of mortgage backed securities, payment transformations, include business, technology, data and systems architecture capturing IT trends, 'what to dos?' concerning design of systems.

ACID, BASE and NoSQL

09 May 2013  |  2727 views  |  0

My last post talked about Enterprise NoSQL and ACID vs. BASE in the context of handling data variety. In this one I'd like to delve deeper into transactional, Enterprise NoSQL. 

Let's start by focusing on the main question: How can one guarantee cross-record ACID transactions in a horizontally-scalable, schema-agnostic database?

The short answer is an architectural pattern called Multi Version Concurrency Control or MVCC.

The basic notion behind MVCC is that records are never modified, but instead a new version is created every time a record changes. The system eventually deletes these old versions after a configureable period of time, but within that time window it's simple to roll back a transaction. More over, it's also straight forward to roll back the entire database to an earlier point in time - A.K.A. point-in-time recovery - a key requirement of enterprise databases. 

Interestingly enough, the availability of Enterprise NoSQL - a schema-agnostic technology that satisfies these requirements - is now starting to blur the boundaries between the traditional Data Warehouse, Operational Data Store and DataMart, and converge them into a single store. The enabler for this is the notion of schema-on-read (vs. the traditional schema-on-write), which refers to the ability to enter data without requiring a pre-defined schema, while supporting multiple schemas when the data is read. This means that the categories mentioned above can be merged into a single platform that satisfies many data consumers without requiring intense modeling and transformation ahead of time.

In addition to schema-on-read, it is also the unification of data management and search that is key to handling data diversity. In fact it was the immense success of search engines that paved the way to this new data management paradigm. Search technologies have established the use of a rich set of indexes as a means for querying non-relational data. From there it was a small leap to apply this notion to a database, converging it with database indexing. But unlike traditional RDBMS, indexes in the NoSQL world do not have to be pre-defined, nor rebuilt as the data changes.

So we're witnessing some related convergence trends - the convergence of structured and unstructured data, that of database and search technologies, and of traditional data management tiers into a single platform.

My next post will tie these concepts back to the related industry use-cases that benefit from them.

 

 

TagsPost-trade & opsInnovation

Comments: (0)

Comment on this story (membership required)
Log in to receive notifications when someone posts a comment

Latest posts from Amir

The Case for Semantic Technology in Financial Services

14 April 2014  |  3240 views  |  0  |  Recommends 0 TagsTrade executionInnovationGroupData Management 101

NoSQL Use Cases

04 January 2014  |  2569 views  |  0  |  Recommends 0 TagsRisk & regulationInnovationGroupInnovation in Financial Services

ACID, BASE and NoSQL

09 May 2013  |  2727 views  |  0  |  Recommends 0 TagsPost-trade & opsInnovationGroupBanking Architecture

Enterprise Big Data: It's Not About Size

16 April 2013  |  3247 views  |  0  |  Recommends 0 TagsPost-trade & opsInnovationGroupInnovation in Financial Services

Big Data Use Cases

24 February 2012  |  12393 views  |  0  |  Recommends 0
name

Amir Halfon

job title

CTO

company name

MarkLogic

member since

2011

location

New York

Summary profile See full profile »
Amir Halfon is Chief Technologist for Financial Services at Marklogic, where he oversees the deve...

Amir's expertise

What Amir reads
Amir writes about

Who is commenting on Amir's posts