Question :
I have an ssis package which loads 1.2 million records of data from oracle into SQL SERVER 2008R2(with 32 GB RAM) dataware house table once every week. (truncating and loading)
This process is taking 5 hours to complete the load of 1.2 million records.
If any body did the same thing,i wanted to know their average time to complete the load.
I am new to ssis. Please advice me. Is 5 hours an optimum time ??
Note: the destination table is de-normalized and no indexes are there on it because it is in the dataware house.
Answer :
Time taken would depend on
- Row length
- Network Bandwidth
- Available resources on the source DB server
- And finally on data volume, as in number of rows
To put things in perspective – I recently did a similar migration. A table with 40 columns ( a mix on nvarchar and numeric) and 3 million rows of data took 35 minutes, where as another table with 7.5 million rows of data ( again a mix of nvarchar and numeric), but with 104 columns took more than 7 hours.
Conversely, a third table with 18 million rows of data, 6 columns, all numeric took 14 minutes. It is impossible to answer this question with Yes or No, with the information provided.
Raj
Can you please post the schema of the source and destination tables? How wide are the rows and are the clustered index keys identical? 5 hours could or could not be optimal, it really depends on:
How fast it can read the data from the source
Some factors that affect this are read performance hits on the source:
- logical and physical fragmentation
- locking/blocking
- disk architecture
- How much data is being pushed through.
Then on the transport layer of the data/package you need to see:
- network bandwidth/quality
The SSIS package then would try to use bulk inserts assuming your table is properly setup for that.
http://henkvandervalk.com/speeding-up-ssis-bulk-inserts-into-sql-server
Then you’d be confined to the weakest link on the destination:
-
DiskIO
-
blocking/locking
-
bad clustering key
You should clear and analyze your waitstats, view the wait stats associated with the load queries, and also read into how to speed up SSIS packages from some blogs such as:
http://www.simple-talk.com/sql/reporting-services/using-sql-server-integration-services-to-bulk-load-data/