Big data – do you really need it in your production database?

Rate this item
(0 votes)
Big data – do you really need it in your production database?

Big data is all the buzz, as organisations scramble to leverage their mountains of data to drive business insight. The question that begs, however, is how much data does a business actually need? Organisations sit with terabytes of data, many several years old, and all of it kept in the production database for instant access. The reality is that this is often not required for everyday business operations. While governance and regulations may require that certain data is kept for legal purposes, it is not necessary to store this data in expensive, 'instant access' databases. Historical data can be archived, saving money and time and helping organisations to use the right big data to make better business decisions.

As content generation continues to explode, data storage strategies have become increasingly important for businesses. Effectively managing this data should be a top priority, for greater cost effectiveness and efficiency. When it comes to big data, not only is it not cost effective, it is also impractical to store all data up front in the production database, and can in fact decrease everyday server performance. Organisations need to have a strategy in place to reduce storage costs in the face of exponential data growth, optimise performance according to the needs of the business and mitigate the risk of lost data and information.

The production database should contain only the current data that is needed for the day-to-day business and operations of the organisation. This database should feature high performance capabilities to deliver this data to users quickly and efficiently. However, if it is being used to store data that is not needed for everyday use, and becomes 'heavy' or bogged down with data, performance will inevitably be compromised.

Data cleansing and consolidation can assist with reducing data volumes in the production database, but this is often not enough to deliver the required performance gains. Strategy needs to be put into place to ensure that data is archived, removed from production and stored in more cost effective options. This strategy, however, must be linked into the business and its needs, including its daily operations. If data storage, retention and archiving strategies are not in line with the needs of the business, users will not be able to access the data they need when they need it and as fast as they need to. It is vital to first understand the needs of business and then put rules into place around archiving. This means that archiving is not simply an IT decision, but a business decision as well, and database administrators need to understand the business in order to provide advice for a better strategy.

With data maintenance plans and archiving strategies in place, data can be moved out of the production environment onto archive servers, which will still enable the data to be easily accessed by users but will not affect the performance of the production server. This will make searching faster and increase performance when accessing or creating data. Historical data will take longer to access, but since this is not needed as often the performance gains on daily data outweigh the minor inconvenience.

Partitioning data in this way will bring down the costs of hardware, software and licensing as well, saving organisations money. Production databases must deliver high performance, which means higher cost. Database size, server memory or CPUs and licensing are interlinked. Those companies who have historical data residing on the production server will need to spend more on ensuring high performance and licensing that is based on CPU's. Archive servers do not need to provide the same levels of performance, so lower cost and lower specified servers can be used for this purpose. This approach also means that maintenance on the production environment is easier, rebuilding indexes is quicker and backups will run faster. In general, performance and uptime will be maximised. The archive server can also be used as a quality control environment to validate data integrity in a safer manner, since doing this on the production server can have a negative impact on business performance.

Ultimately the rules of data storage strategy are simple. The production database should contain only the data needed for day-to-day operations, and all historical data should be moved onto an archive server. This will allow for the production server to be streamlined and deliver the best possible performance, and will optimise the cost of maintenance and running of storage. This in turn will allow organisations to deal with big data in a more intelligent fashion, comply with regulations around data retention, and make more agile decisions based on current data thanks to optimised system performance.

Last modified on Monday, 03 September 2012 13:02
Jaroslav Cerny

Jaroslav Cerny

Jaroslav Cerny founded RDB consulting in 1995, an outsource and consulting company that specialises in five areas: Relational Databases, Operating Systems, Database Security, Monitoring and Enterprise Resource Planning. With over 25 years experience in this field, Jaroslav embarked on his career after he graduated with his Honors in IT at the University of Belgrade in Serbia.

He gained extensive experience in Oracle Database Administration and IT, working in South Africa, Asia and the Middle East (Indonesia, Singapore and Saudi Arabia) as an Oracle Database Consultant. His international experience gives him the edge in the industry and his exposure to different countries and cultures allows Jaroslav to have a deeper understand of the business. Jaroslav leads a team of 35 employees and his goals are to further grow the business and build on the success they have accomplished thus far.

RDB Consulting's clients include some in the Top 200 companies in South Africa.


Related items

More in this category: Share and protect - bringing your SMB’s data storage and protection up to speed »

Leave a comment

The SA Leader Magazine


In the June issue

Balancing of Business Requirements and Cultural Beliefs in the Workplace

Measuring your worth – productivity in the modern workplace

Turning Terrorists into Angels

Marketing needs to continually re-invent itself if it wants to retain the 'X' Factor


Copyright © 2014 gdmc (Geoffrey Dean Marketing Corporation cc). All rights reserved. Material may not be published or reproduced in any form without prior written permission. Use of this site constitutes acceptance of our Terms & Conditions and Privacy Policy. External links are provided for reference purposes. is not responsible for the content of external Internet sites.

Login or Subscribe