Miao’s Thoughts

Using Data Warehouse in Battle Damage Assessment? September 15, 2008

Today I read a paper about using data warehouse in Battle Damage Assessment by Ma Zhi-jun et al:

Ma zhi-jun; Chen li; Zhang yi-zhen, “BDA information management and decision support based on DW,” Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing, 2007. SNPD 2007. Eighth ACIS International Conference on , vol.2, no., pp.384-387, July 30 2007-Aug. 1 2007

DOI 10.1109/SNPD.2007.364

Since I have worked as an DW intern, this paper prompts some very interesting questions.

Since we have many systems in battle damage assessment, with the sharp increase of data, it will be very hard to analyze damage with so many systems. The traditional RDBMS can only provide superficial information. We cannot do in-depth analysis such as the subtle relationship between data. The result is that we have huge amount of data but most of them are useless. The problem is called “data rich, knowledge poor”. The second problem is that because our existing systems are developed by different people at different time, the data format in each system is also different. And also, there must be some incorrect data during system bugs, wrong operation by people or some other reasons. The quality of data will greatly influent the result of data analysis. By using data warehouse, one obvious pros is that we can integrate all the information from seperated systems together and provide a uniform view for the users, like the commander.

But there are some other problems of using data warehouse in battle damage assessment.

The first problem is that since data warehouse integrates all the information needed to do battle damage assessment, will it become a major attack target of our enemy? Nowadays, most commercial data warehouses have at least one backup system in case of earthquake, hurricane and some other disasters. But in military usage, even you have back systems, the enemy can attack both of them. Since our analysis is totally based on one node — the DW, the risk is very high.

Second, almost all data warehouses around world are doing analysis based on historical data. In the other word, today is Sep 14th, we will update the date generated on Sep 13th into DW today. Because it is costly to continiously update DW whenever new data is generated or old data is modified. As far as I know, only Walmart’s DW has the ability to analyze what products are out of stock dynamically. But it is only a small portion of the data warehouse not the whole. So, is it practical to update DW from time to time in BDA?

In Ma’s paper, they argue that storing sleeping data is meaningless and we should detect and delete sleeping data. I am not sure whether we need old data in BDA since I am new to this area. But according to my understanding of DW and the definition of DW, their argument is wrong. Data warehouse in non-volatile which means that there are a lot of query and upsert(update and insert) operation in DW, but we seldom delete date from DW. One of the purpose of data warehouse is to record all the changes of the data in the history. This is one significant difference between DW and RDBMS. For example, if we have a table recording the balance of a user, in RDBMS, we only record the current balance of the account, however, in DW, if the user have balace of $200 at first and then spends $100, we will not update 200 to 100, we will add one row store the new value, which is 100. The old row, represent the change of balance in the history.