“I have this one flat file and I need it in DWH” How often do you get this kind of request? Or maybe you’ve had this kind of request yourself? No matter what, this is a common issue connected to data warehousing, and it sometimes seems hard to tackle. So, the next time you encounter it, try the following approach. Ask yourself two questions: Is this new data or processed data from the data warehouse?

When building a data warehouse for a product, business people usually need some kind of daily log on a level of a single entity (so, you want to know what the customer data looked like on a day x). This is a rather regular task for a data engineer. It’s also not the most challenging one. But sometimes there’s a database imp upstream. It tells database developers not to put update timestamps in tables.